On the Road to a Mode II/Knowledge Society - Problems and Challenges

Gotthard Bechmann
Rainer Hohlfeld
Michael Rader

ISSC Workshop, BBAW. Berlin, 14-15 March 2002

This workshop is concerned with the quest for measures for the evaluation of certain types of scientific research in the emerging knowledge society. If we look at the current situation in research, we can see two basically opposite trends:

• On the one hand, the classical scientific disciplines, such as sociology or economics, are being differentiated, resulting in ever more specialised sub-disciplines


• One the other hand, types of research are emerging which are beyond the scope of a single scientific discipline and thus involve contributions from several disciplines working together on a single subject or problem. This type of project is gaining importance for the researchers themselves but also for the "clients" and "customers", indeed for society as a whole

Both trends create problems for the evaluation of the results of research and of publications resulting from the research. In the first case, specialisation has sometimes reached a degree where it is hard, even for experts with education and training in the "mother" discipline concerned to asses the quality of output. A well-known problem emerging in this context is the formation of "old boy" or "buddy" networks, whose members cite each others� works and are the only people sufficiently specialised for conventional peer review. There is obviously a danger that such networks will become self-serving. On the other hand, review by scientists belonging to the same mother discipline, such as sociology or economics, but not to the same sub-community may produce results perceived as unjust by those at the receiving end. At another level, researchers located in the "wrong" country, or publishing in the "wrong" language might be disadvantaged by peer reviews or citation indices.

In the second case, the leading question is whether at all the same standards can be applied as for disciplinary research. The kind of research concerned is variously labelled as problem-oriented, mode II or transdisciplinary research (not to mention interdisciplinary, multi-disciplinary or cross-disciplinary), denoting that it involves researchers from several scientific disciplines. The composition of the working group depends on the problem in hand and on its context.

This involvement of researchers from several scientific disciplines already indicates that research of this kind is dealing with subject which are too complex to be dealt with satisfactorily by any single scientific discipline.

Without any claims for exhaustive scientific analysis, these are some of the other features of this type of research.

• The research itself is triggered by societal needs and problems rather than scientific curiosity. The results of the research are intended as input for decision-making. These problems are urgent and usually cannot wait for scientifically validated, "sound" results


• This means that the projects have to deal with uncertain, controversial or disputed knowledge, or in some cases with lack of knowledge or non-knowledge


• The "old" model of science has been challenged, whereby only such knowledge is viewed as objective and scientifically valid, which has been produced by experiments, and which can be repeated and validated by anyone, independent of time and place. Awareness of the context dependency and contingency of knowledge is acknowledged by the existence of this type of project


• The attitudes of decision-makers and of scientists towards non-scientific knowledge generated by "ordinary people" in real-life situations have recently also undergone change. In the past, attitudes of lay persons towards technology were frequently dismissed as being due to lack of information and knowledge, in particular is such attitudes were perceived as "negative" and as a barrier to the implementation and application of what was, in the opinion of most scientists in the area concerned, a beneficial technology. The increasing number of crises and catastrophes related to science and technology has led to awareness on the part of decision makers of the contingent nature of science. It has also become apparent that critical or cautious attitudes towards science and technology are frequently displayed by well-informed lay-persons and that the knowledge underlying such attitudes should be considered in decision-making as well as the various existing scientific opinions. The usefulness of results is a major criterion for the quality of research, and this is judged by lay persons rather than experts


• Research of this type is usually organised as a process. The aim of the process is to bring together the stakeholders, including the proponents of various scientific positions, and to feed their positions and opinions into the decision-making process on the problem forming the subject of the project

The significance of this kind of research for science as a whole is controversial. Weingart et al. based on empirical findings, claim that it is fairly marginal and that most science is still organised in the disciplinary structures which traditionally characterise universities. Even when research does cross the borders of the traditional disciplines, the contributions of the individual researchers can be measured largely in the traditional terms of scientometrics or bibliometrics. Where indicators are missing, in many cases it is possible to devise new, adequate measures for scientific quality, such as publications in journals belonging to the other disciplines active in the field. Materials research or nano-technology are outstanding examples for this kind of research.

The significance of transdisciplinary, mode II or problem-oriented research has, however, been recognised by decision makers, for example in the shape of the "precautionary principle" or in the debate on "science and governance" at the EU level.

The goal of commissioning this type of research is to use its results to guide decision-making in areas of societal importance. Its clients obviously have an interest in being able to assess the quality of the work leading to such results to decide on the one hand, on the ways in which these results may be considered in decision-making, and, on the other hand. whether to commission the same institutions with further work of a similar nature. At later stages a measure of quality could be the impact of the results on decision-making on the problem at the heart of the research. This could be in terms of the recommendations in the report actually implemented through the decisions taken, or the number of times a report is quoted by participants in the actual decision-making process. There have been several attempts in the past to evaluate studies from this viewpoint, for instance of technology assessment reports for decision-making by parliaments, but measurement of impact of this kind has proved notoriously difficult (Hoppe, Grin 19 , B�hle et al. 2001).

Apart from impact, what else is there specific to measure?

1. Scientific quality - it is simple enough to measure scientific quality in the standard terms of scientific disciplines, but is this adequate? The goal of projects of the type we are talking about is seldom to produce particularly new scientific results, but to combine knowledge already existing into solutions to pressing societal problems. How should such solutions be assessed against other possible alternatives?


2. The treatment of uncertain or lacking knowledge - there are several ways of dealing with this type of knowledge, for instance assuming the "worst case", or that certain hypotheses are indeed true. But what is the adequate treatment under which circumstances?


3. The consideration of non-scientific knowledge - is the quality of this type of knowledge to be assessed and if so, by whom and by which standards? Or is the measure the range of non-scientific knowledge exposed by the project, or the treatment of this knowledge in the project? Is it simply something to be registered, or does it actively flow into the solutions to the problems concerned?


4. The organisation of the process, in particular of interactions between the various scientific and non-scientific actors, synergies created by the process - doe the actors in the problem area actually enter into a discourse with each other and understand each others� positions better as a result of the project - or do they act in parallel and the task of the project is to compile the various viewpoints for a decision-maker at some distance to the project?

If we had the ready answers to these questions, there would be no need for a workshop of this kind, so we sincerely hope that the discussions over the next day or so will help to ask the right questions and to point in the direction of useful answers.

We start with a contribution by Nico Stehr on the nature of the knowledge society and its demands for the productions of new knowledge. Armin Grunwald addresses the problem of quality assurance in what he calls transdisciplinary research. His point is that the knowledge produced by projects in this area is still scientific and not a matter of beliefs or revelations, so that there most certainly is a need for standards.

Peter Healey and Harry Rothman then provide an overview of prior work in the field, particularly by researchers who are unable to be with us for the workshop today. This is followed by a session, mainly on indicators. Here, we have presentations by Andoni Ibarra/Rafael Renfigo. Philippe Jeanin reports on the evaluation of social science research, while John Rigby tells us about a social science research impact bibliography.

Tomorrow morning we start with a session containing presentations going beyond conventional indicators. Philip Balsiger reports on attempts to evaluate a program on climate research, while Jordi Molas-Gallart examines the impact of research on non-academic audiences. Stefan Kuhlmann describes some of the issues involved in the evaluation of the impact of complex knowledge and Silvio Funtowicz explains his propositions for a system of extended peer review.

This workshop is scheduled as the first of three on the subject and we will devote the remaining time to a discussion intended to identify areas needing more detailed discussion and to point the way forward to the next workshop.