Putting Scientometrics 2.0 in its Place [v0]

This is version 0 of an abstract to be presented at altmetrics11.

Ralph Schroeder
Lucy Power
Eric T. Meyer
Oxford Internet Institute

Measuring science, scientometrics, has provided an enhanced understanding of how science is changing for some time, and is thus useful for (among other things) research policy. With the rapid spread of Web 2.0 tools in scholarly communication, it thus seems appropriate to apply scientometrics to Web 2.0 uses in science. Web 2.0 can be defined here as ‘participatory information sharing’, taken from Wikipedia (en.wikipedia.org/wiki/Web_2.0, accessed 3.3.2011), which in this case is enabled by ‘off‐the‐shelf’ tools such that non‐ computer experts can contribute content onto the Web in a way that is accessible to others. The popularity of Web 2.0 among scholars is the starting point for the argument by Priem and Hemminger (2010) for a ‘Scientometrics 2.0’. They argue that, on the one hand, researchers are nowadays overloaded with content, and on the other that Web 2.0 offers powerful new measures of scientific outputs. A related argument has recently been made by Evans and Foster (2011), who also start with the flood of digital knowledge that researchers are nowadays confronted with. They go on to sketch what they call ‘metaknowledge’, by which they mean the use of computational tools to identify patterns in how knowledge is created in order to better master this process. This goal, of improving the evaluation, filtering, and mapping of knowledge, is also the impetus behind the new programme outlined by Priem and Hemminger.

This essay argues that both of these new tool‐based ideas about science make important points about how science is changing, but they overlook several features about the context of knowledge production. Briefly, the overload that researchers are experiencing is something that they are bound to adapt to, following the law of small numbers and the limits of the attention space (Collins 1998; Meyer and Schroeder 2009), features that have governed and continue to govern science, including science 2.0. Secondly, it is true that Web 2.0 technologies lend themselves to powerful new ways of understanding changes in research, but these must always be put in the context that web 2.0 tools are complementary to and add to existing modes of scholarly communication. In short, scientometrics, including the 2.0 version, and like any account of knowledge, needs to be put into a social context.

These points can be expanded briefly: From a sociological perspective, the structure of knowledge is controlled by the limits of the attention space and the law of small numbers. Applied to online material, a researcher may pay attention to the complementary material that is available on the user‐generated (or ‘participatory information sharing’) Web. However, the stream of research, while constantly growing, is also constantly being winnowed – if only because much research is never cited and only a small amount of research contributes to the advance of knowledge. Even if researchers may thus spend additional time and effort on new channels of scientific dissemination on an everyday basis, ultimately they will be paying attention to the same small number of researchers that are generating novel results or ideas. This also applies to the tools for ‘metaknowledge’, which need to be judged in the end by how they contribute to these new results.

Put the other way around, what scientometrics 2.0 tools will be able to tell us is how the new channels of scientific dissemination are supporting this focus on those dominating the attention space. A useful question here will be: do those dominating the online attention space differ from those who dominate the attention space in the channels of traditional scholarly dissemination? We would expect that although in certain respects, new ‘stars’ emerge online, the main impact of Web 2.0 will be to make their traditionally disseminated work also garner more attention. Further, only work that is assessed by peer review will enter into ‘winnowed’ knowledge, thus making Web 2.0 dissemination ancillary. The exceptions are new online channels that may also have new mechanisms for evaluation, though these channels – like individual researchers – will compete with existing ones. A different way to make this point is that not only just as much, but more material will now receive no attention at all. There are also limits to ‘open’ peer review or online comments or recommendations of articles: these will be an important complement to existing anonymous reviews and anonymous or non‐ anonymous comments: yet ultimately, the constraints of science are that most scientific work needs to be evaluated anonymously (even if some open peer review journals provide an additional avenue for dissemination) while it is still knowledge‐in‐the‐making, and authorship needs to be asserted for knowledge‐ already‐made.

Thus it is inevitable that there will be an ever greater diversity – a proliferation ‐ of the means of scientific communication, as well as an increasing amount of knowledge content. Metaknowledge can be useful in filtering this flood, but these tools will also add to the cognitive workload (they cannot replace consulting sources of original scholarship, but are complementary) and they will also be subject to the laws of the limited attention space and small numbers (which could, in fact, be built into metaknowledge tools – producing an infinitely expandable capacity for meta‐ metaknowledges). It is also inescapable that the amount and diversity of Web 2.0 dissemination of research will add complexity to evaluating the impact of research, which will inevitably grow because of the increasing pressures on limited funding for research. This impact evaluation will make use of scientometrics 2.0, but there is a limit here too: the limit that only a certain amount of research can be said to have impact without making the very notion of impact meaningless.

Scientometrics 2.0, like other online content, will make scholars more reflexive about their visibility and impact. Collins argued some time ago that researchers are drowning in paper. Now we are drowning in even greater amounts of digital resources. Even so, the problem of this flood will be solved – not just at the individual level, but also at the level of the system of scholarly communication: only a few researchers and institutions concentrated in a global hierarchy will be visible. Even as the total number of researchers and outputs expands, this expansion is reduced by specialization and winnowing. Larger and more long‐term changes, such as the relative decline of the US and Europe and the rise of countries like China and India in terms of research rankings, will affect this overall picture. Here, too, scientometrics 2.0 and metaknowledge could be useful: do certain disciplines or geographical areas dominate the attention space of online knowledge more than those of more traditional forms of disseminating knowledge? And where do these new means of dissemination become complementary to established forms of scientific communication in becoming accepted as ‘legitimate’? Scientometrics 2.0 and metaknowledge will thus complexify the research landscape rather than simplifying it. Yet this greater complexity also enriches our understanding of how knowledge works, even if it also puts additional demands on this understanding. The full version of this paper will illustrate these arguments by reference to various Web 2.0 tools, including scholarly Wikis, new multimedia life science journals, and data about Web 2.0 uses in the humanities and in physics.

References

Collins, R. 1998. The Sociology of Philosophies: A global theory of intellectual change, Cambridge: Harvard University Press. Evans, J. and Foster, J. 2011. Metaknowledge. Science, vol. 331, no. 6018: 721‐725

Meyer, E.T., Schroeder, R. (2009). Untangling the Web of e‐Research: Towards a Sociology of Online Knowledge. Journal of Informetrics 3:246‐260.

Priem, J. and Hemminger, B. 2010. Scientometrics 2.0: Toward new metrics of scholarly impact on the social Web. First Monday, vol.15, no.7, firstmonday.org/htbin/cgiwrap/bin/ojs/index.php/fm/article/view/2874/2570

4 Comments

Aalam Wassef

Posted May 9, 2011 at 1:20 pm | Permalink

Unless you did that already, It could be interesting that you establish and describe the concept of “personal strategies” of research, dissemination and reputation (as opposed to institutional/traditional). These personal strategies could be described as belonging to the non-traditional forms of knowledge dissemination, and as relying on the “off-the-shelf” Web 2.0 tools (Wikis, Twitter, Facebook, Blogs…). These strategies are often decided and managed at the level of the individual, within a given social context, in liaison and in full natural complementarity with other traditional forms of dissemination. The context of the individual ties it all together. You already coined this behavior when you wrote: “We would expect that although in certain respects, new ‘stars’ emerge online, the main impact of Web 2.0 will be to make their traditionally disseminated work also garner more attention.” I would be interested to discuss this further with you.
ralph.schroeder

Posted May 9, 2011 at 6:56 pm | Permalink

Aalam,
thanks for this – and agree that there are personal strategies, though the point you raise then leads to at least two further ones: one concerns when personal strategies must ‘bend to the will’ of instutional/traditional forces (in this, the means of research dissemination)? And secondly, when will the new Web 2.0 forms of dissemination themselves take their fully institutionalized form or become traditional?! This just expands on your point, but alsoasks how much scope individuals now have while the new system is still in flux. Thanks for the comment, and look further to discussing it further. See you in Koblenz, Ralph
Victoria Uren

Posted May 19, 2011 at 4:29 pm | Permalink

Ralph,

I am interested in your comment about “dominating the online attention space”. Are you aware of Huberman’s work on social attention online? there was an excellent keynote at EKAW last year
(ekaw2010.inesc-id.pt/invited-speakers.html) but no slides seem to be available. His work is quite commercially oriented (turning attention on websites into profit) but could possibly be relevant for measuring attention to science. regards Victoria
ralph.schroeder

Posted May 20, 2011 at 8:08 am | Permalink

Victoria,
this is a useful comment. For science, I borrow ‘attention space’ from Collins (as per the reference in our paper). I think you are right to point out that, for our argument, one would need to have an overall assessment of ‘scarcity’ (this is where economics perhaps comes in) of the total amount of attention online and offline. Huberman is useful here, but when I look at his publications, such as his article (with Wu) on ‘Novelty and Collection’ (2007), he has a psychologistic understanding of attention. I think we need a more sociological one, but I admit that this is still lacking. Thanks for pushing me to explore this further, and look forward to discussing it more, best wishes, Ralph

altmetrics.org