Social Science Research Impact Mapping and Measurement
John Rigby (PREST, University of Manchester)
ISSC Workshop, BBAW. Berlin, 14-15 March 2002
Assessing Impacts of Social Science Research
The question of how to assess the impact of research activity of engineering and the natural sciences has been an academic interest and a policy concern for many years. Recently, there has been some attempt to re-consider the specific and broader impact of social science research also, a move which stems from some genuine successes of social research engagement and a marginally more favourable political climate in which social structures are seen far less as intractable or the sole guarantors of stability and coherence and more as open to transformation (- witness interest in the UK in the Third Way - and statements such as 'tough on crime and tough on the causes of crime'). This development aims to capture the both the long term value of social science research carried out within narrow academic contexts - what one might regard as research for its own sake - and in terms of the contribution which it makes to societal development and government policy. There has therefore also been far greater use of social science research within the development of European research and technological development, the Fifth Framework Programme being the first to recognize and legitimise the role of social sciences within this context. Within the Targeted Socio-economic Research Programme (1994-1998) and the Key Action Improving the Socio-economic Knowledge Base (1998-2002) of Framework Six, social sciences are moving up an agenda and are carrying high expectations of success. However, the challenge for the social sciences to demonstrate their 'utility', usefulness, significance, appropriateness or relevance is large and increasing. There is no established tradition and practice of measuring the impacts of social science akin to that done on the impacts of RTD which can identify rates of return and which provide not only the possibility of measurement but the opportunity to compare different forms of action in a world of scarce resources. Both inputs and outputs are difficult to identify and hence the 'value added' is highly problematic concept. This discussion firstly considers the contexts in which social science research comes into being. It then reviews how the effects and benefits of social research have recently been conceptualized. It then looks at how such conceptualisations might be amenable to measurement, and the problems which arise for this laudable but problematic enterprise.
Conceptual Discussion - Classifying Research
There are a number of widely known classifications of the research process and its outputs. Classification can take place according to the orientation of the research activity towards a pre-defined goal. When research is conceived as a product, it can take a number of forms or embodiments; for example, academic papers, patents, professors, programmes. Research knowledges and domains can work singly or collectively towards solutions. And research whether for use or for disciplinary development takes place within institutional relationships of many kinds.
In the five-fold (revised) Frascati system attempts to classify in terms of the primary purpose of the research spending and the intended socio-economic impacts. Social science research takes a number of forms although they all involve knowledge and skills or techniques to some degree and an embodiment of these properties (or indeed a combination of such properties) within human or non-human entities.
Output is generally thought to comprises the following where basic research is concerned: the stock of knowledge, the training of graduates, the creation of instrumentation and methodologies, the creation of the capability for further problem solving within institutions or within distributed network of actors, the creation of a capability for action which may or may not of course be realized, and the generation of new firms (Salter and Martin, 2001). This typology, which applies well to the natural and engineering sciences, requires further review before it might be applied to the social sciences with a view to finding a way of attributing value to a specific instances of a particular type of impact/output.
Social science research as a resource for policy purposes in the construction, design, development and use of technology has been strictly limited with its main involvement in ex post activities, i.e. within evaluation (Guston & Sarewitz, 2002). However, there has been renewed discussion of an enhanced role for social sciences both within the context of societal innovation where it is integrated with other domain knowledges such as those from the natural and engineering sciences in so-called transdisciplinary alliances, and also within areas where it is the single disciplinary resource.
A further analysis of the orientation of the research activity suggests four main ways in which research is carried out and supplied (after Landry et al, 2001) technology push; demand pull; dissemination and facilitation; and an interactive mode each of which have different implications for the nature of the research product and the way in which it is disseminated and used.
Contests over Social Science Research
Discussions about the epistemic status of knowledge within the social sciences are a natural starting point for an analysis of the role of social research and its contribution to societal development and to government policy directly and in conjunction with other disciplines. While debate within the social sciences themselves and within the evaluation literature in particular confirms the persistence and the importance a dichotomy between research on the one hand for the sake of explication of concepts and discovery on the one hand and research for a purpose on the other, see for example Rich, (1977) Shadish et al (1991), and Leviton and Hughes (1981), although these latter authors include a third motive for research, persuasion, there is increasing awareness of the role of both power and the role of contexts in which research and research products are designated, shaped, developed and used.
In the broader context of social and political enquiry, the postmodern emphasis upon the specific, localized character of knowledge, bound inextricably to the site of its production in time and space appears to undercut all possible uses of research except the critical and the deconstructing. The theoretical and practical 'utility' of research is thereby endangered. Precisely by undermining all prospect of universal standards of judgment, recent concepts of knowledge are defined as without applicability to action or indeed, without relevance for reflection. Conclusions, and policy principles have therefore limited stability, applicability, and negligible transferability.
Despite these relativistic moves, there remain those who argue that, through embedding within close, interactive, and what must be seen to be a reflexive relationships, social inquiry can credibly retain the function of guiding conduct, see for example Bourdieu (1977), Giddens (1984) and more recently Flyvbjerg (2001). For Flyvbjerg, only a focused and directed social inquiry, based on the Aristotelian concept of practical wisdom (phronesis), promises genuine possibilities for productive engagement and investigation, provided that the research is informed by an awareness and understanding of the forms of power prevailing within the domain of enquiry. Such a definition of power owes much to Foucault and Machiavelli, but the articulation of the concept of phronesis is restricted to social rather than socio-technical contexts.
Other contributions which have also reviewed the status of knowledge product include that of Gibbons et al (1994) which bring forward new perspectives to the question of how and why research, including social science research, is carried out. This account aims to show that all research has taken on unparalleled interactivity, whether it is within the context of a science and technology increasingly driven production process in which knowledge and differences play a fundamental role Reich (1991), or within a humanities increasingly driven by outright scepticism.
This new model of knowledge partly mirrors the distinction between basic and applied, but, in considering the demands for knowledge, and the dynamic question of where knowledges 'come from and go to', their work brings into relief the relationships between different subject knowledges or disciplines, how new knowledges arise from the combination of existing disciplines (multidisciplinarity) and within the marginal territory between them (interdisciplinarity) in the context of use (transdisciplinary - the NPK term which implies a full integration at the conceptual level).
This more dynamic conception of the production of knowledge urges a more active role for the users of knowledge, with knowledge production and knowledge appropriation convergent on the NPK paradigm, and this closer engagement with researchers oriented towards users carries the implication that the position of researchers is itself far less stable and no longer tenured, being affected not only by the development of epistemic knowledge but by the constant flux of user interaction. The NPK also raises important issues of access to the setting of research agendas and the use of research findings.
There is however, opposition to the NPK presentation of events. Godin (1998) regards the argument as significantly overstating previously disciplinary rigidities or trajectories in the research process. His suggestion is that research should be seen as necessarily interdisciplinary, i.e. mode 2, with here and there the occasional special case of research within a single discipline occurring. Furthermore, the NPK vision is seen by Godin not as performative but as seeking to engender the changes which are described.
Others have investigated specific themes of the new production and use, see for example Weiss (1980) on the issue of dissemination which Rip (2001) who explores the relationships which exist between different groups of users and the need for translation between local practitioners on the one hand and those commissioning research on the other.
In a period of increasing sophistication of information technology, hopes have been raised that new mechanisms could be developed with which to link those who produce research with those who might use it. Only one major example of this kind of approach has appeared, the Public Knowledge project (PKp) of the Faculty of Education at the University of British Columbia. This initiative is modelled on various Enlightenment projects of knowledge transfer, for example Liebniz's 'demonstrative encyclopaedia' or Diderot and d'Alembert's 'Encyclopedie' and also on more recent conjectures of an universal information source such as H.G. Wells' 'world brain'. Those following this approach to the public use of research have taken the view that a significant amount of social knowledge can quite readily applied by society if only it were to be made available. However, PKp faces criticism (Gil, 2001) from those who point to its naï:veté: in failing to take full account of what is known about the process of scientific enquiry and the contexts in which it takes place.
Difficulties arise therefore from long running contests over the status of social science knowledge as 'science' which has implications both for the measurement of input - i.e. where the value comes from, and also of the value of research - the output issue, in terms of a given application and more widely.
While the reflexive relationship between social knowledge and society is put forward as condition of the modern age and also perhaps as its most desirable characteristic, this raises significant difficulties for assessing the relation between causes and effects is not necessarily considered as a
Re-stating the Problems
This world of knowledge in practice - perhaps it is a new world -, a world of flux and transience appears to be an inhospitable environment in which to begin the search for measures of utility, value, appropriateness, let alone efficiency. The rules of the game that must now be played appear to be as follows:
The following figure indicates the major discontinuities within the research process as it can be conceived currently. There three main stages, input, processing, and output, each of which represents an important iterative stage within a whole which is itself a recursive and interactive process. The new conditions outlined above require a review of the way in which impacts are considered and valued. A concerted attempt should be made to distinguish between the three different elements of the research process and assess for engagement and societal value in terms of engagement within each of the three stages.
Table 1. Interactivity in Process Three Key Areas for Impact
Towards Assessing Value
While some of the types of knowledge production that exhibit transdisciplinarity currently occur within contexts where markets operate, and where values are assigned, most research is still carried on with the support of funding organisations and sponsors. In the UK, 94% of university funding still comes from this source, although the amount is declining slightly. Let us consider the efforts to address the need to allocate resources.
Attempts to conceptualize impact and 'value' for social science research which operates singly or in tandem with other discipline areas have generally focused on finding indicators of what processes of engagement between social science knowledge and those who might apply it should look like and assessing the level of interaction which takes place between the actors (Spaapen & Sylvain, 1993). This indicators based approach, which examines structures for interaction, is paralleled by work, which suggests that 'learning organisations' (Van Langenhove, 2000) are the means by which social science might realize the goals of society itself.
Attempts to assess the health and quality of the relation of science and society will need to focus upon the viability of structures for interaction and the satisfaction of those served by research: - a) interactivity and communication between science and society; - b) assessment of the relevance of research to a wide range of what Spaapen calls 'societal environments' through both agreed metrics and which are resolved to a single indicator which is part of a Societal Quality Research Profile (SQRP).
Spaapen and Wamelink (1999) have further developed their assessment of embedding of research to generate a Research Embedment and Performance Profile which shows on a radar map the extent to which research is directed towards particular areas of use. This goes beyond the approach taken by Callon et al (1994) whose compass card depicts a range of activities carried out within a particular research activity. The compass card also lies behind The UK's Royal Academy of Engineering research excellence footprint (2000) proposal, a useful contribution incorporating both input and output activities and capability enhancement. In fact, the Spaapen view takes the research dynamic more seriously than elsewhere.
In consideration of this point concerning capability, Shove et al (1998) have argued that these processes of learning and exchange take place through longer-term associations between individual academics and users. Rather than through specific projects therefore, the value of social research comes from longer-term career commitments of individuals and their networks to address specific and related questions. Bourdieu's work on the development of intellectual capital parallels this work, but focuses upon the value of individual research investment rather than the networks in which this value can be realized. The creation of such purposive social structures as Shove describes already takes place through such activities as foresight activities and futures studies, see for example Van der Meulen and Rip (2000).
Those who have developed the view that indicators provide the only realistic means of assessment of value, such as Spaapen & Sylvain (1993) cite the increasing levels of mutual involvement between research and society as central to the difficulty of assessing the impact of social science research. Such divergences from linearity are evident across both the human sciences and the physical sciences, and present significant problems for attribution see for example the view proposed by Nowotny and Felt (1997) of a large techno-science interacting and partly defining the nature of the basic research carried out.
Others focusing upon the concept of users such as Woolgar (2000) recommend a more 'energetic engagement' between researchers and users. Of course, the more energetic these interactions, the more difficult is the attempt to control, manage and audit of outcomes.
Developing new models for production and use of research - for measuring interactivity - provide a significant challenge. Existing assessment systems reflect the dichotomy of producers and users, with traditional, well-organized peer review methods employed to assess the academic, theoretical aspects of research generally more influential than the methods based on varieties of opinion polling which have been widely used in the attempt to give a measure of 'societal value'.
Examples of Good Practice and Their Limitations
Research carried out through the use of social science, which has the aim of ultimately delivering societal benefits, can take many forms, as the examples discussed in this section will indicate. From the examples studied during this project, the following examples indicate both the key steps which underpin successful research that delivers clear benefits to society and the restrictions which currently influence the way in which that research process can develop.
Integration of the roles of researchers and users of research appears the key step towards establishing successful research activities. Such integration takes the form not only of professional networks of researchers and policy-makers, as in the case of the Asia-Pacific Migration Research Network, a project supported by UNESCO - MOST (Bedford, 2001) but also requires appropriate forms of public consultation which must take into account existing political structures (Kazemir, 2001). However, there is no guarantee that this process is fully public and engaged as solutions to problems are increasingly within private rather than public alliances and networks (Liberatore, 2001).
Such integration imposes a requirement for structural support, including the creation of institutions, the provision of resources for teaching, and the creation of evidence bases (Kasemir, 2001), (Bruce, 2001) (Bedford, 2001). Assessing the impact of these institutional resources upon the work carried out within them is also likely to become an important issue for evaluation in the same way that research infrastructure has been a concern for the physical sciences.
I. Creating InfrastructureCreating new institutions and systems, as the EU and the UK have done, the latter through the establishment of the Economic and Social Research Council Resource Centre for Evidence based policy - intended to be an analogue for Oxford's Cochrane Centre for evidence-based medicine - will assist in the attempt to provide the resources and institutional arrangements to make policy reflect research and vice versa. Larger bases of knowledge may help to broaden the basis upon which policy conclusions are made. Steps taken in the UK and also in Canada (Stipich, 1994) to produce graduates who are skilled in the areas needed to take part in the new forms of research are also needed.
II. Changes to Peer ReviewGreater involvement social scientists in research that is useful for society is also likely to depend, according to Van Langenhove (1999) upon increasing the level of interdisciplinarity which requires that the academic enterprise should be re-fashioned: 'A paradigm shift from publication-driven research towards change-driven research and a paradigm shift from disciplinary-driven research agendas towards research driven by problems and their driving forces.' (Gibbons, 1994). Scott et al (1999) give a broad definition of interactivity and show how interactive research requires a variety of academic disciplines, concluding also that ╬users' lie not only outside academe but can include both other academics and students. That evaluation systems need to take account increasingly of the broader areas of contribution made by researchers, requires Peer Review is used in conjunction with or indeed replaced by Merit Review (Renaud, 2001).
This process of integration between users and researchers is subject to a number of constraints, however, some of which are significant. At the level of developing policy-making mechanisms at levels larger than national systems where there are attempts to create international 'added value' (in terms of an evidence base or training), there should be concern that specific country differences are not overlooked. Failure to understand specific country differences will lead to inappropriate structures for embedding and integration. This problem is particularly relevant to the forthcoming European Research Area.
Limits on the scope of research to effect change are faced as ever from such non-negotiable areas as ethics and politics, which are often hidden by or hidden from social science policy researchers. Radical changes such as those envisaged by Beck (1998) that the nation-state axiomatic be discarded in a complete re-evaluation of the terminology and 'metaphors of the social realm' (Beck, 1998) see Slocum (2001), may be difficult to achieve, despite the apparent attractiveness of starting again.
The process of integration also depends upon a number of pragmatic arrangements which can ease the tensions which may exist between researchers and governments. Ground rules to cover the publication and disclosure of research need to exist for there to be trust. As Batzokas et al report (2001): 'The use of social scientists by governments is never innocent'.
It has also been noted by a number of commentators (OECD, 2001) that both the dissemination of social research findings and the evidence that such research has delivered benefits should be more widespread. Only when the benefits of social science research become more clearly visible to policy-makers will there be a commitment in terms of resources. Here, the policing and quality control regimes within the research process itself are likely to need modification.
While integration at the level of research and users is an essential precondition for the new approaches to delivering value from the social sciences, it is important also to ensure that researchers themselves are integrated to produce multidisciplinary groups that are capable of understanding the needs of users.
To measure and map social science research impacts successfully within single and multidisciplinary contexts, a focus is needed upon each of three logically separable stages of research activity: input stage, research process stage, and output stage.
At the input stage, the clear focus is upon the enrolment of actors within the specification (the input side); secondly, while the research progresses and is carried out; and thirdly, when research must be used, how is it used, how widely it is disseminated and by whom and with what benefit. These three processes are of course themselves part of a larger dialogical process which can itself be subject to analysis.
Because supporting structures are also more important to the carrying out of research within an interactive mode than within single channels, they will also need to be assessed for what they contribute to the interaction, and ultimately for the benefit of users and researchers.
As research moves into this interactive and dialogical mode, interaction both across the conventional user-producer divide, and also between users themselves and between researchers (Bedford, 2001) is important. Success also depends upon the presence of trust between users and rules for disclosure, and the creation of effective support structures for research, teaching and, of increasing importance dissemination of results.
Atkinson, A. (2001) Speech at the International Conference on Social Inclusion - Making Common European Objectives Work, Belgium
Batzokas, A. et al (2001) 'The Contribution of the Social Sciences to Knowledge and Decision Making: Reports of Four Working Groups' Social Sciences for Knowledge and Decision Making, OECD: Paris
Beck, U. (1992) The Risk Society - Towards a New Modernity, Sage: London
Bedford, R. (2001) 'A Robust Research/Policy Interface: International Migration and Social Transformation in the Asia-Pacific Region' Social Sciences for Knowledge and Decision Making, OECD: Paris
Bruce, R. (2001) 'Scientific Rationality, Public Vale and Democracy: The Failure of Genetically Modified Food and Patenting Debates in the United Kingdom' Social Sciences for Knowledge and Decision Making, OECD: Paris
Callon, M., Laredo, P. & Mustar, P. (1994) 'Panorama de la science Francaise' La Recherche, page 264, 378-384.
Cernea, M.M. (1991) 'Knowledge from Social Sciences for development Policies and Projects in Cernea, M.M. (ed.) Putting People First: Sociological Variables in rural development, second edition, Oxford University Press: New York, pages 1-41
Cohen, D.K. & Lindblom, C.E. (1979) Usable Knowledge (New Haven, CT, Yale University Press).
Cohen, L., McAuley, J. & Duberley, J. (2001) 'Continuity in discontinuity: Changing Discourses of Science in a Market Economy', Science, Technology and Human Values, Vol. 26, No. 2, pages 145-166.
European Council Lisbon (2000) Presidency Conclusions - Point 32, page 12
Felt, U. & Nowotny, H. (1992) 'Striking Gold In The 1990s - The Discovery Of High-Temperature Superconductivity And Its Impact On The Science System' Science Technology and Human Values 17 (4): 506-531
Flyvbjerg, B. (2002) Making Social Science Matter - Why social inquiry fails and how it can succeed again, Cambridge University Press, Cambridge
Gibbons, M. et al (1994) The New Production of Knowledge
Gibbons, M., & Johnston, R. (1974) 'The Role of Science in Technological Innovation' Research Policy, No. 3 pages 22-42
Gil (2001) Book Review Social Forces, June, Vol. 79, No. 4
Godin, G. (1998) 'Writing Performative History: The New Atlantis' Social Studies of Science, Vol. 28, No. 3, pages 465-483
Golub, S., 'International Labor Standards and International Trade', IMF Working Paper 97/37 Washington, April 1997
Graham S., & Marvin, S. (1994) "CHERRY PICKING AND SOCIAL DUMPING: Utilities in the 1990s" Utilities Policy, 4 (2), 113-119.
Guston, D.H. & Sarewitz, D. (2002) Real-time technology assessment, Technology in Society, Vol. 24, No.1, Pages 93-109
Harre, R. (2001) 'Causal Mechanism and Social Practices: What can Social Science
Contribute to Social Practice' Social Sciences for Knowledge and Decision Making,
Landry, R., Amara, N. Lamari, M. (2001) 'Utilization of Social Science Research in Canada' Research Policy, Vol. 30, No. 2, pages 333-349.
Leviton, L.C. & Hughes, (1991) Foundations of Programme Evaluation: Theories of Practice, Sage: London
Liberatore, A. (2001) 'From Science/Policy Interface to Science/Policy/Society Dialogue' Social Sciences for Knowledge and Decision Making, OECD: Paris
Martin, B. & Salter A. (1996) The economic benefits of publicly funded basic research: a critical review', Research Policy 30 (3): 509-532 Mar 2001
Netherlands Development Assistance Research Council (2001) 'Utilization of Research for Development Cooperation' Netherlands
Nicol, Sir W., & Salmon, T. (2000) Understanding the European Union, Longman: London
Nowotny, H. & Felt. U. (1997) After the Breakthrough: The Emergence of High-Temperature Superconductivity as a Research Field. Cambridge: Cambridge University Press
Reich, R. (1991) The Work of Nations: Preparing Ourselves for 21st Century Capitalism, Simon and Schuster: London
Renaud, M. (2001) 'Serendipity and Policy Relevance' Social Sciences for Knowledge and Decision Making, OECD: Paris
Rip, A, (2001) 'Utilisation of research: A sociology of Knowledge perspective', in Utilization of Research for Development Cooperation - Linking Knowledge Production to Development Policy and Practice' Netherlands Development Assistance Research Council
Scott, A., Skea, J., Robinson, J., & Shove, E. (1999) 'Designing Interactive' Environmental Research for Wider Social Relevance' ESRC Global Environmental Change Programme, Special Briefing No 4, May 1999
Shadish, W. et al (1991) Foundations of program evaluation: theories of practice, Newbury Park, California: Sage Publications
Shove, E. & Rip, A. (2000) 'Users and Unicorns: a discussion of mythical beats in interactive science' Science and Public Policy, Vol. 27, No. 3
Shove, E. et al (1998) Exchanging Experience of Exploitation - A study commissioned by the Economic and Social Research Council
Slocum, N. (2001) 'Realising the Future - On Collaboration Between the Policy Makers and Social Scientists to Facilitate Social Change' Social Sciences for Knowledge and Decision Making, OECD: Paris
Smith, G. (1993) 'Science to Policy and Vice Versa - A comment on the reflexive relationships between social policy and social research' A Paper for the Conference on Evaluation, Social Science and Public Policy, June 1999
Spaapen, J. & Sylvain, C. (1993) 'Assessing the Value of Research For Society' Research Evaluation Vol. 3, No. 2
Spaapen, J. & Wamelink, F. (1999) Evaluating University Research - A Method for the Incorporation of Societal Value http://www.agro.nl/nrlo/english/pdf/9912e.pdf
Stern, E. (1993) 'Ongoing and participative evaluation: purpose, design and the role in the evaluation of a large-scale R&D program' Research Evaluation Vol. 3, No. 2
Stipich, N., 1994 'Improving Social Knowledge Transfer' Research Evaluation, Vol. 4, No. 1
Tiler, C. & Boddington, A. (1993) 'Outputs, Structure and process in the evaluation of social science research centres', Research Evaluation Vol. 3, No. 2
TSER Thematic Network Contract Number Soe2 -Ct-97-3059 Sedec Network Final Report May 2000 'Comparative Social Inclusion Policies And Citizenship In Europe: Towards A New European Social Model'
Van der Meulen, B. & Rip. A. (2000) 'Evaluation of societal quality of public sector research', Research Evaluation Vol. 8, No. 1, pages 11-25
Van Langenhove, L. (1999) 'Re-thinking the social sciences? A point of view' The social Sciences at a Turning Point? OECD Workshop on the social sciences
Van Langenhove, L. (2001) 'Can the Social Sciences Act as an Agent of Change in Society? An Introduction to the Workshop on the Contribution of Social Sciences to Knowledge and Decision Marking', Social Sciences for Knowledge and Decision Making, OECD: Paris
Wallerstein, I. (2000) 'From sociology to historical social science: prospects and obstacles', British Journal of Sociology, Vol. 51, No. 1
Wallerstein, I., et al (1996) Open the Social Sciences, Stanford University Press: Stanford, California
Weiss, C. (1979) 'The many Meaning of Research Utilization', Public Administration Review, September - October
Weiss, C. (1991) Social sciences and modern states: national experiences and theoretical, Cambridge: Cambridge University Press,
Weiss, C.H. (1980) Knowledge creep and decision accretion, Knowledge: Creation, Diffusion, Utilization, 1, 3, pp. 381-404.
Weiss, Carol H. (1998) - Evaluation: methods for studying programs and policies 2nd ed. - Upper Saddle River: Prentice Hall,
Woolgar, S. (2000) 'Social basis of interactive social science' Science and Public Policy, Vol. 27, No. 3
Wright, C. & Shevchuk. L. (1994) 'Knowledge, chaos and public policy' Research Evaluation, Vol. 4, No. 1, pages 12-30
Back to ISSC Workshops