Peer review and quality control
EC-JRC/ISIS, I-21020 Ispra (VA)
ISSC Workshop, BBAW. Berlin, 14-15 March 2002
Peer review is a mechanism for quality control in science, including the assessment of proposed projects and also of completed work. Its roots go back to the emerging science of seventeenth century, when novelty in natural knowledge became distinguished from technical invention, and the ideals of reproducibility and publicity became established. Peer review was acknowledged by the mid-twentieth century as the unquestioned norm in 'academic science'. Recent experience has shown the need for some modifications in the process of quality-assurance, such as changes to traditional procedures of collegial peer review and the inclusion of new stakeholders.
Among contemporary forms of scientific practice, curiosity-driven research with no regard for applications has been increasingly marginalized. A diversification has occurred, so that quality-assurance must also be consider in such areas as mission-oriented and issue-driven research, forensic science, and the provision of scientific advice for policy. Collegial peer review is being transformed to review by an 'extended peer communityš.
Peer review is a mechanism for quality control in science, including the assessment of proposed projects and also of completed work. Its roots go back to the emerging science of seventeenth century, when novelty in natural knowledge became distinguished from technical invention, and the ideals of reproducibility and publicity became established. Peer review was acknowledged by the mid-twentieth century as the unquestioned norm in 'academic science'. By then it was assumed that quality was unproblematic, owing to some special ethical disposition of scientists and their methods. With the recent transformations in the social practice and context of science, which now extends into technology and policy, corresponding changes in quality assurance are needed. The crucial requirement is for a more encompassing community of those evaluating the products and processes of science. (See, for example, Stampa III 1997). With this development goes a parallel enhancement of the methods and objects of scientific inquiry.
In most other spheres of institutional activity, the formalization of quality-assurance has become the norm, as for example through the wide-ranging standard-setting procedures of the ISO (International Organization for Standardization). In academic science however, along with cultural pursuits like the arts, the methods are still largely informal. Science has been almost unique in having self-assessment performed by practitioners rather than by external 'critics'. To what extent and in what ways this must change, to keep pace with science's expanding role in public life, is becoming an urgent question in the governance of science.
The assurance of quality is not a straightforward task. This has been known since the time of the Romans, as indicated by the Latin motto 'quis custodiet ipsos custodes?' ('Who will guard the guardians themselves?š). This motto implies an indefinite iteration. It is a reminder that however routine may be the tasks of quality control, full quality-assurance demands yet higher levels of supervision at which informality and explicit value judgments are necessary.
As long as science remained mainly academic, problems of quality were assumed to be resolved by the very nature of the scientific endeavor. The informal systems of checking by peers seemed a rational response to the problem, rather than a culturally contingent mechanism characteristic of a particular epoch. Scientific facts were believed to be discovered by some infallible Method, and scientists were themselves viewed as being endowed with certain superior moral qualities that protected them from ordinary human frailties. This latter could be explained in sociological terms, as in the 'four norms' of scientific practice expounded by Robert K. Merton in 1942 (Merton 1973), or philosophically, as in the committed attempts at self-refutation supposed by Karl Popper to be normal scientific practice (Popper 1959).
With the onset of the industrialization of science after World War II, the self-conscious study of science as a social activity, including the methods of quality-assurance, became inevitable. Growth in size, capital investment, scale, and social differentiation within science, created divisions between managers and research workers, as well as between researchers and teachers in universities. A 'Gemeinschaft' (community) of scholars could no longer realistically be assumed. The earliest disciplined analyses of the quality of production in science were quantitative. Derek J. de Solla Price, who devised some measures of quality and provided analyses of its distribution, did the pioneering work. He noticed that at the leading British science reference library, only a minority of journals was ever requested. The contents of the others could be inferred to have no interest, and hence to be of very low scientific quality (Price 1963). This phenomenon is a reminder that 'quality' is a relational attribute. 'Fitness for purpose' depends on whose purposes are dominant; not always perhaps those of a community devoted to the advancement of learning, but possibly those scientists working under constraints of 'publish or perish'.
Price's studies were continued in two directions. At the Institute for Scientific Information, Eugene Garfield produced more searching and sophisticated measures of quality, using citations rather than mere number of publications. Such attempts at quantification were bound to become controversial (Brooks 1982; Garfield 1970; Garfield 1972). It was impossible to avoid bias in the selection of the relatively small set of journals used for citations; those in conventional mainstream English-language research science were inevitably privileged at the expense of all others. Further, when quantitative measures of citations came to be used as indicators of academic merit, manipulative practices, including reciprocal citations, inevitably developed. The deep problems of a quantitative proxy for quality suddenly became acute.
In a more reflective vein, Jerome R. Ravetz applied the "quis custodiet" principle to analyze the vulnerability of the quality assurance system in science. He observes that the processes of peer review are conducted largely informally and (unlike research) are not themselves normally submitted to open scrutiny and review. They require a different sort of competence, which is not part of the formal training of scientists; and they are also more open to a variety of abuses, ranging from bias to plagiarism. In such terms one can understand the phenomena of low quality, both in scientific research and in technological development. Thus while denying that the practice of science automatically produces a higher morality, he agrees that it is necessary for the successful practice of science. On this basis he stresses the importance of morale and morality (and ultimately idealism and leadership) in science (Ravetz 1996).
This analysis provides a background for the current interest in 'trust' as an essential element of practice in science, in society and in their interactions. The broader society has provided resources to the esoteric activities of science because it trusts the scientific community to make good use of them. There has always been an undercurrent of distrust, based on evidence either of pointless research or of malign applications. Now that science is so deeply involved in technology and related policy problems that crucially affect public health and welfare, the traditional trust can no longer be assumed. It would appear to be necessary for the principles and practices of accountability to be extended from the institutions of political governance (as by representative democracy) to those institutions, which govern science and its applications.
Quality control in research science has become more difficult as the relatively inflexible technical requirements of the traditional printing process have been relaxed. There is no longer a well-defined 'gateway' to publication through the institutions that control reproduction of, and hence access to documents. First through inexpensive photocopying and now through the Internet, it has become easy for anyone to distribute scientific wares to an unrestricted audience. In addition, the presence of the global media tends to bypass the traditional processes of evaluation, which were conducted personally among colleagues. Isolated scientific results become media events. All those with an interest in the report, as consumers, politicians, regulators and the stock markets, become potential stakeholders in the evaluation of the result. Thus science arguably has become accountable to a drastically extended peer community in the quality-assurance process. As we saw above, the criteria of quality applied by these heterogeneous actors need not be identical to those of 'public knowledge' generated within tightly knit scientific networks.
These developments are ambivalent; while they may seriously disrupt the procedures of quality-assurance in normal science they can, on the other hand, bring public scrutiny to bear on controversies and scandals. The demystification of scientific practice both enables such events to become news, and is fostered by their exposure. Top scientists become like celebrities - needing the media for advertising themselves yet simultaneously hating it for its unwanted intrusions. The 'Baltimore affair', centering on the U.S. Nobel laureate David Baltimore's laboratory at M.I.T., is perhaps the most notorious case in which a dispute about scientific misconduct was blown up into a lengthy, visible, political saga that damaged all the individuals and institutions involved (Kevles 1998). The episode was symptomatic of an increasingly recognized problem of 'deviance' in science, which carries the unspoken danger that, without timely correctives, it might become the norm.
All these developments affect the maintenance of trust, which is necessary for ordinary scientific practice and even more for quality-assurance. As in other professional domains, the normal tendency in science has been for misconduct to be covered up by the responsible institution (not necessarily by the community of scientists). In such situations, ultimate exposure does even more damage and further erodes the basis in mutual trust. Attempts to circumvent the need for trust by increasing bureaucratic surveillance are likely to be counterproductive in their own way, by erecting impediments to free inquiry and communication among colleagues.
The relations between social science and natural science have also been transformed during the last decades. Starting with the acceptance of natural science as the ideal of knowledge, essentially independent of social forces, there has been a gradual but accelerating shift toward recognizing all sciences as incorporating social constraints and biases. An early critical interaction was in connection with the astronomical community's management of the eccentric Velikovsky (de Grazia 1966). Later, the social-science community embraced Thomas Kuhn's disenchanted picture of 'normal' science (Kuhn 1970). Finally, post-Feyerabend studies of science re-examined the whole institution without presupposing any privileged status in relation to either virtue or natural knowledge (Bloor 1991; Bloor, Edge and Henry 1995; Collins and Pinch 1993; Fuller 1993).
When natural scientists, led by physicists, eventually confronted the emerging socialized picture of their discipline, the reaction was so strident, that 'science wars' became an appropriate label (The Economist 1997; Gross and Levitt 1997; Nelkin 1996; Ross 1996). Sociologists of science and post-modernists were indiscriminately blamed for all the ills of science, including decline of public trust, budget cuts, resurgent Creationism and even poor teaching of science. A physicist whose hoax article (Sokal 1996) was accepted by a leading cultural studies journal, Social Text, mounted a counter-attack (Bricmont and Sokal 1998). The implication was that the critics of science had no real quality control of their productions. The science warriors' assumption was that within real science, such problems are prevented from occurring because of the verifiable empirical content of scientific research. In the ensuing debate, there was little mention of the ease of publication of erroneous or vacuous research in the standard scientific literature. Historical episodes like Millikan's manipulation of his oil-drop results in the course of a controversy on the electronic charge were discounted as mere embarrassments (Segerstale 1995).
We have presupposed thus far that 'science' refers primarily to traditional basic research. But among contemporary forms of scientific practice, curiosity-driven research with no regard for applications has been increasingly marginalized. A diversification has occurred, so that quality-assurance must also be consider in such areas as mission-oriented and issue-driven research, forensic science (Foster and Huber 1997; Jasanoff 1996), and the provision of scientific advice for policy (Jasanoff 1990; Salter 1988). In addition, the products themselves and the media through which they are diffused are increasingly diversified. For example, patents are now a common outcome of a research process, and this form of intellectual property is radically different from traditional published papers (Myers 1995). Also, results are reported in unpublished consultancy advice, 'gray literature' or kept confidential within institutions or even totally sealed under 'lawyer-client confidentiality' and legal settlement agreements. With traditional peer-review as the norm, the challenges of quality assurance for these new products and processes are nearly unrecognizable. A genre of critical literature has developed, with some authors directing anger at the new contexts of scientific production (Huber 1991), and others more clearly appreciating the problems they present (Crossen 1994; Jasanoff 1990).
A parallel diversification is now occurring in the types of knowledge-production that are accepted as legitimate. The democratization of knowledge now extends beyond the juries who assess the quality of technical evidence in courts (Jasanoff 1998), to include those who master previously esoteric aspects of their predicament (e.g. illness, contamination, pollution, oppression, discrimination, exploitation) through special-interest groups or the Internet. In addition, claims of specialized or local knowledge are present in even more diverse contexts, as among indigenous peoples, and in systems of complementary or 'traditional' medicine. These claims are commanding increasing commercial and political support among various publics, as well as gaining explicit recognition in numerous international treaty regimes. As a result, a new philosophy of knowledge appears to be emerging, based on a new disciplined awareness of complexity, in which a plurality of legitimate perspectives is taken for granted (Funtowicz and Ravetz 1991). Modern science, with its characteristic methodology and social location, is part of this enriched whole, but not coextensive with it. The criteria and tasks of quality assurance now must explicitly involve additional values and interests, even extending to ontological commitments of groups other than scientists. This new configuration has been termed post-normal science.
Quality assurance can thus be seen as a core commitment of post-normal science, replacing 'truth' as science's ultimate regulative principle (Funtowicz and Ravetz 1992). Defined in terms of uncertainties and decision-stakes, quality assurance encompasses 'public interest', 'citizen', and 'vernacular' sciences. In a period of domination by globalized corporate science (Gibbons et al 1994), this effort to make scientists accountable to interested groups presents a coherent conceptual alternative for the survival of the 'public knowledge' tradition of science. Collegial peer review is thereby transformed into review by an 'extended peer communityš. This new form of quality-assurance will be given its formal structure and routines by those who put it into practice.
Bloor D 1991. Knowledge and Social Imagery. Chicago, University of Chicago Press.
Bloor D, Edge D, Henry J 1995 Scientific Knowledge. Chicago, Athlone.
Bricmont J, Sokal A D 1998 Fashionable Nonsense: Post-modern Intellectuals' Abuse of Science. New York, Picador.
Brooks H 1982 Science Indicators and Science Priorities. Quality in Science. La Follette M Ch (ed.). Cambridge, MASS, The MIT Press: 1-32.
Collins H, Pinch T 1993 The Golem: What Everyone Should Know about Science. Cambridge, Cambridge University Press.
Crossen C 1994 Tainted truth: the manipulation of fact in America. New York, Simon & Schuster.
The Economist 13 December 2021 77-79.
Foster K R, Huber P W 1997. Judging Science - Scientific Knowledge and the Federal Courts. Cambridge, MASS, The MIT Press. 13
Fuller S 1993 Philosophy, Rhetoric and the End of Knowledge: The Coming of Science and Technology Studies, Madison, University of Wisconsin Press.
Funtowicz S O, Ravetz J R 1991 A new scientific methodology for global environmental issues. In: Ecological Economics. Costanza R (ed.). New York, Columbia University Press: 137-152.
Funtowicz S O, Ravetz J R 1992 Three Types of Risk Assessment and the Emergence of Post-Normal Science. Social Theories of Risk. In: Krimsky S, Golding (eds),. Westport CT, Greenwood: 251-273.
Garfield E 1972 Citation Analysis as a Tool in Journal Evaluation. Science(178): 471-479.
Garfield E 1970 Citation Indexing for Studying Science. Nature( 227): 669-671.
Gibbons M C, Limoges C, Nowotny H, Schwartzman S, Scott P, Trow M 1994 The New Production of Knowledge. Beverly Hills CA, Sage.
de Grazia A (ed.) 1966 The Velikovsky Affair: The Warfare of Science and Scientism, New York, University Books Inc.
Gross P R, Levitt N, Lewis M W (eds.) 1997 The Flight from Science and Reason, Baltimore, Johns Hopkins University Press.
Huber P W 1991 Galileo's revenge: junk science in the courtroom. New York, Basic Books.
Jasanoff S 1998 The Eye of Everyman: Witnessing DNA in the Simpson Trial. Social Studies of Science 28(5-6): 713-740.
Jasanoff S 1990. The Fifth Branch: Science Advisors as Policy as Policymakers, Cambridge, Harvard University Press.
Jasanoff S 1996 Science at the Bar: Science and Technology in American Law. Cambridge, Harvard University Press.
Kevles D J 1998 The Baltimore Case: A Trial of Politics, Science and Character. New York, W.W Norton & Co.
Kuhn T S 1970 The Structure of Scientific Revolutions, 2d.ed. Chicago, University of Chicago Press.
Merton R K 1973 The Normative Structure of Science, rpt. in Merton, The Sociology of Science. Chicago, University of Chicago Press.
Myers G 1995 From Discovery to Invention; The Writing and Rewriting of Two Patents. Social Studies of Science 25(1): 57-105.
Nelkin D 1996 What are the Science Wars Really About. The Chronicle of Higher Education: A52.
Popper K 1959 The Logic of Scientific Discovery, New York, Basic Books. 14
Ravetz J R 1971 Scientific knowledge and its social problems, with a new introduction by the author. New Brunswick, N.J.: Transaction Publishers, 1996. Originally published: Oxford: Clarendon Press.
Ross A (ed.) 1996 Science Wars. Durham NC, Duke University Press.
Salter L 1988 Mandated Science. Dordrecht, Kluwer.
Segerstale U 1995 Good to the Last Drop? Millikan Stories as "Canned" Pedagogy. Science and Engineering Ethics(1): 197-214.
Sokal A D 1996 Transgresing the Boundaries. Social Text(14): 217-252.
de Solla Price D J 1963 Little Science, Big Science. Cambridge, Cambridge University Press.
Stampa A E III (ed.) 1997 Advances in Peer Review Research. Science and Engineering Ethics 3(1): 1-104.
Back to ISSC Workshops