Beyond Evaluative Bibliometrics?

Peter Healey (SPSG)
Harry Rothman (Nottingham Business School)

ISSC Workshop, BBAW. Berlin, 14-15 March 2002

Why try to go beyond evaluative bibliometrics?

  • a mode two/knowledge society agenda of new actors and new linkages - institutional and epistemic - in knowledge production and use
  • the impact of the IT revolution on the communication of science, combined with the acknowledgement of traditional deficiencies in bibliometric measurement, particularly in relation to the social sciences
  • the increased relevance of science and technology to social and economic choices of high political salience
  • an increased social recognition of the contingent uncertainties of scientific knowledge and of the risks associated with its deployment
  • in association with this, an increased attention to two dimensions of public interest in assessing research - that of the consumer and that of the citizen - in addition to traditional policy concerns with economic costs and benefits, and the need for research which is 'socially robust' (Gibbons) or which commands 'informed public consent'
  • a continued policy interest in decision aids and a shift of focus in the use of these from local, contextual and evaluative to global, multidisciplinary and strategic

The UK policy context - some signs of changing policy need

UK Sustainability Indicators -
Latest indicators published 13/03/02

  • Quinquennial Review of the Grant-Awarding Research Councils, OST (stage 2 report, November 2001) - my emphases:
  • a 10-15 year road map of opportunities for science to underpin the consideration of future priorities, including the opportunities for international partnerships
  • a shared knowledge-base of opportunities, competancies and needs in order to serve the basis for decisions about science funding
  • a systematic long-range monitoring exercise...every five or six years to identify traceable outcomes arising from earlier science budget expenditure - whether from the immediately preceding five year period or from earlier periods
  • recognising the limitations of bibliometric analysis as a measure of the performance of the UK science base, a variety of assessment parameters... that can be used in a convergent manner to provide a more effective assessment of the UK science base

Measuring Excellence in Engineering Research, Royal Academy of Engineering, January 2000 - original emphases:

Excellence in engineering research demands a strategy to develop the appropriate mode 1 research, mode 2 research, scholarship and vitality & sustainability... engineering research groups produce knowledge which cannot be separated from its context of application, requiring researchers to develop links with those communities wishing to exploit the results. The research needs of these communities will need to be recognised in any assessment process...

Health warnings for application of evaluative bibliometrics to the social sciences

Drawing largely on Diana Hicks (1999) 'The difficulty of achieving full coverage of international social science literature and its bibliometric consequences' Scientometrics, 44,2, 193-215

The current European Commission exercise
Mapping excellence in RTD in Europe: the case of economics

Royal Academy of Engineering: Possible Components of Measures of Research Excellence (adapted from report cited, appendix 3)

Contrasting models for analysis of scientific productivity Adapted from Bozeman et al. (2001)

A four-step procedure for visualising intellectual structures

Chen C and Paul R J (2001) Visualizing a knowledge domain's intellectual structure.
IEEE Computer, 34,3,65-71.

VxInsightTM Software capabilities (1) Sandia National Laboratories

VxInsightTM Software capabilities (2) multi-resolution exploration with detail on demand

Indicators as lagged representations of STI theory

A framework for mode 2/knowledge society mapping and indicator construction
(click the image to see enlarged figure)


Katy Börner, Chaomei Chen, Kevin W. Boyack 'Visualizing Knowledge Domains' ARIST vol. 37, forthcoming 2002

Paul Bourke, Linda Butler and Beverley Biglia (1996) 'Monitoring Research in the Periphery: Australia and the ISI Indices' ANU Research Evaluation and Policy project Monograph series no 3

Barry Bozeman, James S Dietz and Monica Gaugin (2001) 'Scientific and technical human capital: an alternative model for research evaluation. Int. J. Technology Management, 22, 7/8. 716-740

Kevin Bryant (2000) 'Progress in S&T; Indicators: from R&D; to the Knowledge-based economy'. Australian Bureau of Statistics Workshop on Indicators for the Knowledge based Economy/Society, Canberra, November 2000

Kevin Bryant (2001) 'Promoting Innovation: an overview of the application of evolutionary economics and systems approaches to policy issues' in John Foster and J Stan Metcalfe, eds, Frontiers of Evolutionary Economics: Competition, Self-Organization and Innovation Policy. UK and Brookfield US: Edward Elgar

Kevin Bryant: 'Is the definition of research the real policy issue?' Australian contribution to the OECD workshop on basic research, Oslo, October 2001

P M Burnhill and M E Tubby-Hille (1994) 'On measuring the relation between social science research activity and research publication', Research Evaluation, 4,3, 130-152

Commission of the European Communities, staff working paper (2001): How to map excellence in research and technological development in Europe. Brussels, 12/03/2001; SEC (2001)434

David H Guston and Daniel Sarewitz (2002) 'Real-Time Technology Assessment' Technology in Society, 23,4. forthcoming

D.Hicks (1999) 'The difficulty of achieving full coverage of international social science literature and the bibliometric consequences' Scientometrics, 44/2, 193-215

Diana Hicks and Jonathan Potter (1991) 'Sociology of Scientific Knowledge: A Reflexive Citation Analysis or Science Disciplines and Disciplining Science' Social Studies of Science (SAGE, London, Newbury Park and New Delhi) vol21 (1991) 459-501

J Sylvan Katz (1999) Bibliometric Indicators and the Social Sciences prepared for the ESRC

Kyvik, S (1988) 'Internationality of the social sciences: the Norwegian case' International Social Science Journal, 163-172

F Narin, M Albert, P Kroll and D Hicks (2000) Inventing our future: the link between Australian patenting and basic research. Canberra: ARC and CSIRO

S Pierce (1987) 'Characteristics of professional knowledge structures: some theoretical implications of citation studies' LISR, 9, 143-171

U Shoepflin (1990) 'Problems of representativity in the Social Sciences Citation Index' in P Weingart, R Sehringer and M Winterhager (eds) Representations of Science and Technology, DSWO Press. 177-188.

Henry Small and Diana Crane (1979) 'Specialities and Disciplines in Science and Social Science' Scientometrics, vol 1, 445-61.

Helen Szigeti (2001) The ISI Web of Knowledge Platform: Current and Future Directions. Published online by ISI Inc, Philadelphia, PA

Robert Tijssen 'Citation impact measures: a metrics for benchmarking scientific performance of European economics research?' Presentation at the EC DG Research Workshop In search of scientific excellence: research performance by disciplines, Brussels, 13 November 2021

Quinquennial Review of the Grant-Awarding Research Councils, stage two: Report by the Review Team. UK DTI/OST, November 2001

M Yitzhaki, (1988) 'The language preference in sociology' Scientometrics, 41, 1-2, 243-254

Back to ISSC Workshops