In February 2006, the Tavistock Institute was commissioned by the then Office of Science and Technology (now the Department for Business, Innovation and Skills, BIS), in association with the Economic and Social Research Council (ESRC), to design, pilot and then further refine an evaluation framework for use in evaluating Science and Society initiatives.

Context

Science in society initiatives encompass a wide variety of programmes and projects that work to integrate science and society, from large-scale awareness raising campaigns (such as ‘Science Week’), to targeted interventions focused on specific accessibility issues such as gender balance and low representation of black and minority ethnic communities in science professions.They reflect a number of policy agendas and actions – such as the ‘Science and Innovation Investment Framework – Next Steps 2004-2014’ and the ‘Science, Technology, Engineering and Mathematics (STEM) Report’ – which in turn highlight a number of issues.These include concerns about the UK lagging behind in global research and technology (RTD) investment; a decline in the numbers of students studying Science Technology Engineering and Maths (STEM) subjects; a perception that the UK needs to improve its science, technology and engineering skills base; and the recognition that science can play a major role in contributing to improved quality of life and in promoting social inclusion.

Objectives

Against this background, the aim of this project was to produce an evaluation framework for evaluating ‘science in society’ initiatives. Developing a framework was timely because: 

• Although there are a large number of programmes and initiatives aimed at raising public awareness of science and supporting greater participation in STEM education and careers, there was little inter-connection and integration across these different actions.
• There has been significant investment in Science and Society programmes and initiatives.
• The evidence based on the outcomes and impacts of this investment, and on ‘what works’, is poorly developed.
• There is no systematic ‘evaluation culture’ associated with Science and Society programmes and initiatives. Evaluation practices vary in quantity and quality, and evaluation is not adequately grounded across the board in robust, rigorous concepts, models and methods.

Methodology

The project was undertaken in three stages: 

1. Production of an evaluation framework for evaluating science in society initiatives;
2. Piloting the framework in a real-world evaluation of a science in society initiative; and
3. Revising the Framework in the light of the pilot results.

Stage one Stage one involved conducting a scoping exercise through which the project team gained an understanding of the policy environment in which OST, ESRC and the science in society programme operated.This was done via stakeholder interviews, and an ‘audit’ of relevant data sources and contracts. Following this, we reviewed current evaluation ‘state of the art’ against the mission and purposes of the programme; assessed the effectiveness and relevance of particular evaluation approaches and methods in this field, and developed an appropriate evaluation model and tools for subsequent pilot testing.Stage two In stage two the framework was piloted in an evaluation of SETNET (the Science, Engineering, Technology and Mathematics Network), leading to further refinement of the framework.

Results

The Framework combines three elements:• A model and mapping tools to explore how political, economic, social and cultural factors shape Science and Society and, in turn, the kinds of programmes and initiatives that are implemented to promote STEM policies. • A review and analysis of the broad spectrum of programmes and initiatives that have been implemented in the UK and elsewhere, and the kinds of approaches used to evaluate them. • A set of generic principles, procedures and methods that can be used to implement effective evaluation.These three elements are integrated into the Framework to provide a set of procedures, guidelines, methods and examples to improve understanding of and apply key components of evaluation design in order to support effective evaluation.