Validating Research Performance Metrics Against Peer Rankings

Download 4.15 Mb.
Size4.15 Mb.
1   2
Figure 2. Some of the metrics on which Citebase can rank search results.
For a given paper, Citebase can also generate growth curves for downloads and the growth of citations (see Figure 3). It turns out that early download growth is a predictor of later citation growth (Brody et al. 2006).

Figure 3. Citebase growth curves for citations (red) and downloads (green) for a particularly important author in physics (E. Witten).

The various different metrics according to which Citebase can rank papers or authors can only be applied individually, one at a time in the current implementation. There is a menu (Figure 4: ‘Rank matches by…’) that allows the user to pick the metric. But in principle it is possible to redesign Citebase so as to rank according to multiple metrics at once, and even to adjust the weight on each metric. Imagining several of the vertical metric ranking options in Figure 2 arranged instead horizontally, with an adjustable weight (from -1 to +1) on each, gives an idea of how a search engine like this could be used to calibrate the outcomes of the multiple regression analysis described earlier for validating metrics. Exploratory analysis as well as fine-tuning adjustments could then be done by tweaking the beta weights.

Figure 4. Citebase allows users to choose the metrics on which they wish to rank papers, as well as to allowing them to navigate on the basis of of citation links.
Bradley, F.H. (1897/2002) Appearance and Reality: A Metaphysical Essay. Adament Media Corporation.
Brody, T., Carr, L., Gingras, Y., Hajjem, C., Harnad, S. and Swan, A. (2007) Incentivizing the Open Access Research Web: Publication-Archiving, Data-Archiving and Scientometrics. CTWatch Quarterly 3(3).
Brody, T., Harnad, S. and Carr, L. (2006) Earlier Web Usage Statistics as Predictors of Later Citation Impact. Journal of the American Association for Information Science and Technology (JASIST) 57(8) pp. 1060-1072.
Butler L (2008) Using a balanced approach to bibliometrics: quantitative performance measures in the Australian Research Quality Framework (this issue ESEP)
Campbell P (2008) Escape from the impact factor (this issue ESEP)
Hajjem, C., Harnad, S. and Gingras, Y. (2005) Ten-Year Cross-Disciplinary Comparison of the Growth of Open Access and How it Increases Research Citation Impact. IEEE Data Engineering Bulletin 28(4) pp. 39-47.
Harnad, S (1985) Rational disagreement in peer review. Science, Technology and Human Values. 10 p.55-62.
Harnad, S. (2004a) The invisible hand of peer review. In Shatz, B. (ed.) Peer Review: A Critical Inquiry. Rowland & Littlefield. Pp. 235-242.
Harnad, S. (2004b) Enrich Impact Measures Through Open Access Analysis. British Medical Journal 2004; 329:
Harnad, S. & Brody, T. (2004) Comparing the Impact of Open Access (OA) vs. Non-OA Articles in the Same Journals, D-Lib Magazine 10 (6)
Harnad, S. (2007) Open Access Scientometrics and the UK Research Assessment Exercise. In Proceedings of 11th Annual Meeting of the International Society for Scientometrics and Informetrics 11(1), pp. 27-33, Madrid, Spain. Torres-Salinas, D. and Moed, H. F., Eds.
Harzing AWK, van der Wal R (2008) Google Scholar as a new source for citation analysis (this issue ESEP)
Hirsch, Jorge E., (2005), "An index to quantify an individual's scientific research output" Proceedings of the National Academy of Sciences 102(46) 16569-16572
Hitchcock, Steve; Woukeu, Arouna; Brody, Tim; Carr, Les; Hall, Wendy and Harnad, Stevan. (2003) Evaluating Citebase, an open access Web-based citation-ranked search and impact discovery service
Kleinberg, Jon, M. (1999) Hubs, Authorities, and Communities. ACM Computing Surveys 31(4)
Kline, Paul (2000) The New Psychometrics: Science, Psychology and Measurement. Routledge
Lawrence, S. (2001) Online or Invisible? Nature 411 (6837): 521
Moed, H. F. (2005) Citation Analysis in Research Evaluation. NY Springer.
Oppenheim, Charles (1996) Do citations count? Citation indexing and the research assessment exercise, Serials, 9:155-61, 1996.
Shadbolt, N., Brody, T., Carr, L. and Harnad, S. (2006) The Open Research Web: A Preview of the Optimal and the Inevitable, in Jacobs, N., Eds. Open Access: Key Strategic, Technical and Economic Aspects, chapter 21. Chandos.
Page, L., Brin, S., Motwani, R., Winograd, T. (1999) The PageRank Citation Ranking: Bringing Order to the Web.

1 In "Appearance and Reality," Bradley (1897/2002) wrote (of Ayer) that 'the man who is ready to prove that metaphysics is wholly impossible ... is a brother metaphysician with a rival theory"

Download 4.15 Mb.

Share with your friends:
1   2

The database is protected by copyright © 2023
send message

    Main page