Problems of Quality Assurance of Problem-Oriented Research

Armin Grunwald
Institute for Technology Assessment and Systems Analysis (ITAS)
Research Centre Karlsruhe, P.O. box 36 40, 76021 Karlsruhe, Germany
Email: [email protected]

ISSC Workshop, BBAW. Berlin, 14-15 March 2002


The quality assurance of problem-oriented, interdisciplinary research raises specific problems. It is shown that results of integrative research in technology assessment or to solve environmental problems depend on a manifold of pre-empirical decisions which are made due to ethically, politically or socially relevant criteria. In addition to the disciplinary 'internal' quality requirements, there are requirements for a external quality to be acknowledged.

The Challenge

Problem-oriented research differs in fundamental respects from classical institutionalized research (Funtowicz/Ravetz 1993, Bechmann 2000). Its cognitive interests are not only determined by establishing knowledge of nature or of the world, as is predominantly the case in the classical disciplines, but rather by the problem- and decision-making orientation of the knowledge gained. The production of solid textbooks - the succession of which made the discipline's progress tangible - is no longer the goal, but rather the elaboration of project-specific results as a contribution to knowledge as a basis for decision-making and acting. Science leaves the protected niche of presumed value-freedom, takes on a politically relevant role in the definition of social problems (cp. the field of climate change), and becomes dependent - for the conditions for its success and the criteria for its quality - on its non-scientific contexts.

Quality assessment of problem-oriented Research

The results of problem-oriented research are intended to influence social and, above all, political practice (e. g., in environment protection programmes, in sustainable development policies, or in the Rio process). It is, therefore, imperative that these results are reliable, and that their quality can be judged ex ante in order to allow for comparisons and well-founded choices between different options. Otherwise, a simple process according to the approach of trial and error would be carried out - which, in societal affairs, is extremely risky and inefficient. Quality management is of utmost importance for scientific results in problem-oriented research, because this research is designed for practical use in decision-making. 'If sustainability goals are to be achieved, science and technology development as potential forces for public good have to be guided by a quality control process based on explicit epistemological, ethical and political reflection' (Funtowicz et al. 1999).

Quality assurance in scientific research is oriented to the criteria of validity for scientific propositions: the requirements of general, personally invariant validity (transsubjectivity) and intersubjective comprehensibility. In the normal process of scientific research, these criteria are submitted to quality tests on two different levels - (1) on that of method, and (2) on that of the organization of research. Together, these both types of well-established quality assessment are denoted as internal quality where the 'internal' relates to the internalities of the respective discipline.

  1. The claim of a scientific proposition to validity must be methodically verifiable. By examining the observance of recognized scientific methods (e. g., of gathering data, of mathematical methods, of experimental technique, of statistical methods of evaluation), the validity of the statement in question can - step by step - be proven or refuted. For the implementation and acceleration of these procedures, reference is made to scientific jargon, terminologies, and basic concepts, which - in scientific disciplines - form the basis for justificatory discourses. This basis is an essential constituent of disciplinary paradigms (Thomas Kuhn). The philosophy of science in this case carries out the function of making quality assurance possible, inasmuch as it reconstructs the necessary frameworks for establishing criteria and procedures for quality assessment.
  2. But in actual institutionalized research, the validity of scientific results isn't tested down to the last step of methodological analysis. The touchstone is rather the current level of technology and knowledge ('state of the art') attained. Only in cases of particularly grave doubt is the research analyzed down to the level of the basic concepts, and the methodical foundations of the current disciplinary paradigms are taken as a criterion. For quality assurance in science's daily routine, organizational measures of self-monitoring have been established: scientific periodicals with demands on quality which are often upheld by means of peer reviews, internal assessments, and systems of expert opinions. In individual cases, even these control systems can err or be misled, and can not completely preclude forgeries, for example; on the whole, however, they seem to fulfill their purpose, so that no serious mishaps occur.

Wholesale transfer of both of these types of mechanism to interdisciplinary research is not possible, because there is no science of 'interdisciplinarity', nor is such a (meta-)discipline possible, which could constitute a meta-paradigm in relation to the disciplines. Outside of the disciplines there is neither a specialised language nor a methodological procedure available to check the quality of problem-oriented knowledge. When the validity of scientific propositions made on the basis of interdisciplinary research is to be examined, it is at the outset completely unclear, in which scientific terminology this should be done. According to which criteria should the quality of an interdisciplinary result Z, developed out of a statement A (valid relative to the disciplinary framework A') and out a statement B (valid relative to the framework B') be judged, if we have to assume that A' and B' don't have anything to do with one another? What happens to the disciplinarily assured quality in interdisciplinary aggregations and integrations?

The thesis presented in the following is, that, in addition to the internal quality, specific criteria of an external quality of integrative research must be applied. The (internal) quality of the disciplinary contributions is a necessary, but not in itself sufficient precondition for the quality of interdisciplinary research. In addition to disciplinary quality assurance, quality assurance must also be introduced into the process of knowledge aggregation (cf. part 4). Otherwise the interdisciplinary results might develop into 'science light', with unclear quality criteria, and open for a methodical 'rule of thumb'. It must be emphasized that problem-oriented research doesn't raise less stringent, but rather higher demands on quality assurance: observance of the usual disciplinary standards of quality is necessary, but not sufficient to guarantee the quality of problem-oriented research.

The orientation of problem-oriented research with respect to the formulation of social problems precludes that the criteria of scientific research are constituted exclusively within science. Interdisciplinary research refers to socially relevant extra-scientific decisions. The quality of the results to be expected then depends essentially on the 'quality', i. e., the adequacy of these decisions as to the societal problem under discussion and the responsibilities for solving it. In this way, in addition to internal quality criteria for scientific work external criteria have to be applied to problem-oriented research. In the following some specific methodical challenges to the quality assurance of problem-oriented research are analysed.

  1. Formulation of the problem
  2. Choice of basic terminology
  3. Application of classifications
  4. System Demarcations in the Disciplines Involved
  5. Basic features of modelling
  6. Differing Quality Criteria for the Knowledge Contributions involved

The orientation of research projects to contribute to the solution of social problems implies that the criteria of scientific research have to transgress the range of inner-scientific quality criteria and have to include extra-scientifical issues. It has been shown that integrative research refers to socially relevant non-scientific decisions in essential pre-empirical respects. The quality of the results to be expected then depends essentially on the 'quality', i. e., the adequacy of these decisions as to the problem under discussion and the responsibilities for solving it. In this way, in addition to internal quality criteria for scientific work external criteria have to be applied. Integrative research doesn't raise less stringent, but rather higher demands on quality assurance: observance of the usual disciplinary standards of quality is necessary, but not sufficient to guarantee the quality of interdisciplinary research. Otherwise problem-oriented research would get into danger to lose external quality. If such fundamental and pre-empirical ingredients of problem-oriented research like terminology, system boundaries or assessments on relevance were biased this deficit could not be compensated by the consecutive modelling or research.

It has become clear that, in addition to an internal quality assurance grounded on inner-scientific methods, some kind of external quality assurance must be applied which would tackle the appropriateness of such pre-empirical conventions and decisions with respect to the problem to be solved. The question now is how this external quality can be ensured by means of research organisation. Are there organisation principles assuring a maximum of external quality? What is the maximum of external quality which can be constituted by science alone? Where are the limits of science in this field?

Between Analysis and Synthesis

Breaking down a problem into its constituent elements according to goal-means- and relevance considerations is an objective of the analysis of the problem in question, before interdisciplinary research can at all begin. This pre-empirical analysis includes several of the elements discussed above relating this process to the requirements for external quality: classifications have to be applied, terminologies to be chosen and system demarcations have to be defined. The analysis problem is fundamental, and must be solved on the basis of extra-scientific criteria. In this sense also '...the translation of life-world problems into scientific problems presupposes a non-scientific standpoint from which the problems can be recognized and their relevance judged...' (Jaeger/Scheringer 1998, p. 14). The problem of synthesis is subordinate to the problem of analysis; it turns out that the answer to the question whether integration of knowledge succeeds or not, is determined basically by the foregoing analysis and pre-empirical reduction of the problem. Two aspects of these pre-empirical decisions are of major importance:

  1. deliberations on relevance for the construction of systems and for modelling, and
  2. the requirement of pragmatic compatibility on all of the levels discussed, in order to secure the quality of interdisciplinary research

Both types of quality-determining measures are governed by the specifications of the societally-defined problem. The criteria for relevance decisions as well as for compatibility requirements have to be derived from this problem by means of extra-scientific and pre-empirical deliberation. The criteria for relevance decisions and compatibility requirements are normative; it is not possible to justify them inner-scientifically. Their justification extends to social questions and requires under certain circumstances forming a political and ethical opinion. The social orientation of integrative research is not characterized merely by the application of scientific knowledge, but extends into the criteria for ascertaining scientific quality. Conflicts in the field of sustainability, for example, refer to such pre-empirical normative judgments which often are related to questions of the type: which society we want to live in, on the basis of which understanding of nature and the environment, and which conceptions of humanity are to be realized, are concretized. Sustainability conflicts therefore concern fundamental elements of society's self-concept and have the following characteristics in common with political conflicts: plurality of value systems concerned, lack of a concrete focus group, and interlacing of systems.

In this situation, the provision of knowledge for orientation and action for political measures in the field of environment policy can't be value-neutral. 'Scientific practice is not fundamentally 'value-free' but it has always needed to find its justifications by prevailing social concerns' (Funtowicz et al. 1999). The normative plane has to be taken into consideration as well. How one should (re)act doesn't follow at all from models alone, because the direction of action is intimately bound to normative convictions, e. g., on a just society. This has been acknowledged in the sustainability discussion, where it has been explicitly reflected, which normative convictions can form a basis for sustainable policy (KopfmŸller et al. 2001). When, however, one attempts to gain clues for future action out of a descriptive simulation of the system Earth, then this attempt must either go awry, because such models can only generate and extrapolate trends (and are therefore predictive rather than action-orienting), or the normativity needed is obtained surreptitiously, and is introduced into the deliberations as pre-empirical input. In this case, however, the normativity itself would no longer be open to transparent criticism which could lead to grave biasses with consequences for decreasing social and political acceptance.

Quality assessment of problem-oriented research in this sense mediates between analysis and synthesis. It directs its attention to the entirety of the area defined for problem-solving, without losing contact to technical details. External quality assessment can, in fact, be defined as a means for the construction and the cognition of complex systems, which makes the connections between the global view of the system on the problem level from the outside, and the internal analysis of the system's structures down to questions of disciplinary details on the inside. External quality assessment concerns itself with the reductional, structural, and aggregational problems of knowledge and skills. In this process, the problems of relevance and compatibility mentioned above have to be kept in mind. Decisive for quality in this connection is not the representation or comprehension of reality, but rather the construction and integration of purposive forms of knowledge.


Bechmann, G. (2001): Problemorientierte Forschung. In: Jahrbuch des Instituts für Technikfolgenabschätzung und Systemanalyse. Karlsruhe

Bechmann, G., Frederichs, G. (1996): Problemorientierte Forschung: Zwischen Politik und Wissenschaft' in G. Bechmann (ed.) Praxisfelder der Technikforschung. Campus, Frankfurt, pp. 11-40

Coenen, R. (20001, ed.): Transsektorale Forschung zum Globalen Wandel. Campus, Frankfurt

Funtowitz, S., O'Connor M, Ravetz, J. (1999):Challenges in the utilisation of science for sustainability. In: M. Catizzone (ed.): From Ecosystem Research to Sustainable Development. European Commission, Ecosystems Report No 26, Brussels

Funtowitz, S., Ravetz, J. (1993): The Emergence of Post-Normal Science. In: R. von Schomberg (ed.): Science, Politics and Morality. Kluwer Academic Publisher, London

Grin, J., Grunwald, A. (2000, eds.): Vision assessment: shaping technology in 21st century society. Towards a repertoire for technology assessment. Springer, Heidelberg et al.

Grunwald, A. (2000): Technik für die Gesellschaft von morgen. Möglichkeiten und Grenzen gesellschaftlicher Technikgestaltung. Campus, Frankfurt

Grunwald, A. (2001): Integrative Forschung zum Globalen Wandel: Herausforderungen und Probleme. In: R. Coenen (ed.): Transsektorale Forschung zum Globalen Wandel. Campus, Frankfurt (im Druck).

Jaeger, J., Scheringer, M. (1998): Transdisziplinarität: Problemorientierung ohne Methodenzwang. GAIA 7, pp. 10-25

Kopfmüller, J., Brandl, Jörissen, J., V., Paetau, M., Banse, G., Coenen, R., Grunwald, A. (2001): Nachhaltige Entwicklung integrativ betrachtet. Edition Sigma

Mittelstraß, J. (1974): Die Möglichkeit von Wissenschaft. Suhrkamp

Rotmans, J. (1998): Global Change and Sustainable Development: Towards an Integrated Conceptual Model. In: Schellnhuber/Wenzel 1998, S. 421-450

Schellnhuber, H.-J. (1998): Earth Systems Analysis - The Scope of the Challenge. In: Schellnhuber/Wenzel 1998, S. 3-195

Schellnhuber, H.-J., Wenzel, V. (1998, eds.): Earth Systems Analysis. Integrating Science for Sustainability. Springer Verlag, Heidelberg et al.

WBGU 1996, Wissenschaftlicher Beirat für Globale Umweltveränderungen der Bundesregierung, Welt im Wandel: Herausforderung für die deutsche Wissenschaft. Bonn

Back to ISSC Workshops