Evaluating the payback or impact from expenditure on research

Funding: Australian Research Council Discovery Grant (AUD $397,514)
Investigators: Fiona Jenkins (PI), Helen Keane, Marian Sawyer, Claire Donovan

Gender equity has still not been realised, despite decades of activism, policy and research. In some of the social sciences, women make up less than 15 per cent of the professoriate. Yet these are the disciplines that should most aid our understanding of how gender works in society. The project asks what impact women's limited influence and status in these key fields of research has upon our capacity to grapple with the social and political changes necessary for progress toward gender equality. In doing so, it builds persuasive arguments about how and why gender matters in the social sciences. By examining how we judge excellence in social science research, the project aims to contribute to advancing women in all fields.

In 1976, a German researcher linked sun bed use to skin cancer. In 2006, 30 years later, the Cooksey review of publicly funded health research stated that the UK was at risk of “failing to reap the full economic, health and social benefits of public investment in health research” due to health research not being adequately translated into health outcomes. At the same time, a 26 year old girl, Clare Oliver, was diagnosed with advanced melanoma and devoted her last three months to highlighting the dangers of the solarium industry, who she blamed for her melanoma. Clare’s personal media campaign triggered state and federal health ministers to regulate the solarium industry and restrict its use by people with melanoma-prone skin. By the time Clare had passed away, she had left behind a public health policy that had taken researchers over 30 years to achieve without her.

For health research, the societal benefits may be obvious, but assessing them is difficult.  In 2014, a new evaluation framework, The Research Excellence Framework will use a measure of how research has influenced the social, political and economic lives of citizens in order to distribute almost £1.76 billion of public funding. Including this new criterion, however, raises questions about how evaluations of societal impact will be made and what aspects are evaluated favourably.

In Clare’s case, how do we reward the research involved? Identifying which research was most responsible is challenging because it is difficult to identify which research was the most influential. Was it the research that first linked solarium use and an increased risk of melanoma in 1976? Should it be Clare’s doctor, the author of the systematic review, the researcher that sat on the government committee, or the person who drafted the policy? The incorporation of a societal impact evaluation for the REF2014 provides a unique opportunity to investigate how the research evaluators handle this new impact criterion.

The large scale, mixed methods investigation employs a series of in depth, semi structured interviews with the REF2014 evaluators, bibliometric and network analysis and text analysis of EU and UK research policies via VOSviewer.

Funding Body: ESRC

Contact: Gemma Derrick

In recent years, researchers and funders have aimed to better understand the range of impacts arising from public and charitable funding for medical research — including the resulting economic benefits. Such information provides accountability to taxpayers and charity donors, and increases our understanding of how research effectively translates to health gains. Building on a previous methodological framework developed by HERG, the aim of this work is to quantify a key element of the economic returns from publicly and charitably funded musculoskeletal research in the UK, specifically through an assessment of the net monetary benefit from the subsequent application of research-based developments in the provision of musculoskeletal health services.

Funding body: Wellcome Trust, ARUK, DoH, MRC

Contact: Matthew Glover

In 2006 the Cooksey Review of UK health research funding highlighted the ‘second gap in translation’ which related to difficulties of translating research findings into practice and thus improving patient care. The NIHR established the Collaborations for Leadership in Applied Health Research and Care (CLAHRCs) to strengthen collaborations between universities and local NHS organisations, and thereby address this gap. The NIHR Service Delivery and Organisation (SDO) Programme then developed a research programme to examine the progress of this major innovation. HERG is collaborating with a RAND Europe-led evaluation that is part of this SDO programme. The overarching question being addressed in this evaluation is how, and how effectively, do CLAHRCs narrow the second translation gap?

Funding body: National Institute of Health Research, Service Delivery and Organisation:

Contacts: Prof Steve Hanney, Prof Martin Buxton & Dr Bryony Soper (PI: Prof. Tom Laing, RAND Europe)

The annual flagship publication of the World Health Organization, the World Health Report, will focus in 2012 on the role of research in improving the health status of populations. Based on the long-running stream of research at HERG on assessing the benefits of health research, Steve Hanney was invited to be a consultant to the WHO on this and help develop some case studies for possible inclusion in the World Health Report 2012. This proposed World Health Report was eventually delayed to become the World Health Report 2013.

Funding body: World Health Organization

The National Institutes of Health commissioned a scoping exercise to identify the best approach to use to evaluate the outcomes of their Mind-Body Interactions and Health Program. The scoping exercise recommended the evaluation would best be informed by the Buxton and Hanney Payback Framework. Steve Haney was invited to play a consultancy role for the Madrillon Group which was successful in its bid to undertake the evaluation. The Payback Framework is informing the data collection and analysis in the evaluation in various ways.

Funding body: National Institutes of Health (USA)
Contacts: Prof Steve Hanney

In this project Steve Hanney is assisting a team from the International Centre for Diarrhoeal Disease Research, Bangladesh (ICDDR,B) to develop a new format for engaging health policymakers with research based evidence and presenting them with relevant information. This project will attempt to explore the current levels of engagement between health policymakers and researchers in Bangladesh, and examine the impact of previous engagements. The project will then attempt to develop some more routine systems for enhancing the levels of engagement and developing tools to inform policymakers about the systematic evidence available on priority issues in health and the health system.

Funding body: Alliance for Health Policy and Systems Research/WHO
Contacts: Prof Steve Hanney (Lead Researcher: Dr Tracey Koehlmoos, ICDDR,B)

Steve Hanney successfully applied to the MRC Methodology Programme for funding to build on an earlier project that examined how far the translation of research into eventual impact on clinical practice could be identified by tracing citations through a series of generations of research papers. Most citation analysis is quantitative, but we are developing ways of using more qualitative analysis to categorise citations, and thus identify the citing papers for which the original paper was of significant importance. Having refined and tested the categorisation template, the study will then pilot it on papers from bodies of mental health research. This will enable us to explore how far it is possible to trace impact through a whole series of generations of citing research papers until a point is reached in which an impact is made, for example, on a clinical guideline.

Funding body: Medical Research Council
Contacts: Teresa Jones, Prof Steve Hanney & Dr Claire Donovan

HERG collaborated with RAND Europe to lead an international project to study the impacts or payback from cardiovascular research in three countries (Australia, Canada and the UK) taking basic and clinical research that commenced in 1989-1994 as a starting point. Each of the 29 case studies is organised according to the HERG payback framework. Having identified the impacts achieved, the study is going on to attempt to identify the factors associated with higher and lower impacts in the various categories. Three major reports from the project were published in 2011, and further publications are following.

Funding body: Canadian Heart and Stroke Foundation, National Heart Foundation of Australia, Department of Health, England.

Contacts: Prof Martin Buxton & Prof Steve Hanney 

• Wooding, S., S. Hanney, A. Pollitt, J. Grant, and M. Buxton. 2014. “Understanding factors associated with the translation of cardiovascular research: a multinational case study approach.” Implementation Science 9:47.
• Wooding, S., S. Hanney, A. Pollitt, M. Buxton, and J. Grant. 2011. “Project Retrosight: Understanding the returns from cardiovascular and stroke research: The Policy Report.” Santa Monica, CA. http://www.rand.org/pubs/monographs/MG1079.html
• Pollitt, A., S. Wooding, S. Hanney, M. Buxton, and J. Grant. 2011. “Project Retrosight: Understanding the returns from cardiovascular and stroke research: Case Studies.” Santa Monica, CA.
• Pollitt, A., S. Wooding, S. Hanney, M. Buxton, and J. Grant. 2011. “Project Retrosight: Understanding the returns from cardiovascular and stroke research: Methodology Report.” Santa Monica, CA. http://www.rand.org/pubs/technical_reports/TR925.html

HERG applied the Buxton and Hanney Payback Framework to assess the returns or impact from the research funded by Asthma UK. The framework was applied to cover the work of the various types or modes of funding traditionally used by Asthma UK: projects, fellowships and professorial chairs. Comprehensive coverage of the projects and fellowships was achieved by use of a survey, and then selective case studies were undertaken on nine projects, two fellowships, both professorial chairs and also on the pioneering Medical Research Council-Asthma UK Centre in Allergic Mechanisms in Asthma jointly located at King’s College London and Imperial College London.

Funding body: Asthma UK
Contacts: Prof Steve Hanney

Staff involved: Teresa Jones, Martin Buxton, Steve Hanney

• Objectives: The Fellow is part of a small team led by the Director of the Health Economics Research Group, Professor Martin Buxton, conducting innovative research relating to the benefits from health research. The Fellow will analyse the research outputs of the NHS, using data provided to the NHS as part of their membership of the Research Outputs Database (ROD) - a dataset of biomedical research papers developed by the Wellcome Trust but now relocated at the Department of Informatics, City University.
• The Fellow will produce policy-relevant information about NHS research including:
• The amount, quality and nature of the articles resulting from NHS and other funding sources
• What journals clinicians read and think important for their clinical practice across individual medical specialties where clinical research is considered particularly important – initially psychiatry, surgery and paediatrics.

Methods:

• Quantitative analysis methods, including bibliometric analysis, will be used for the assessment of NHS research outputs.
• Expert opinion, from within each of the medical specialties under study, will be sought in the preparation of questionnaire surveys of clinicians views of the importance of peer-reviewed journals for their clinical practice.

Staff involved: Steve Hanney, Martin Buxton

Objectives: This study has two main aims. First, to establish how the impact of the NHS Health Technology Assessment (HTA) Programme should be assessed. Second, to assess the impact of the NHS HTA Programme established in 1993.

Methods: A scoping exercise was conducted and identified the HERG payback framework developed by Buxton and Hanney to be the most appropriate approach. Therefore, the primary research study will start applying this framework but at the same time a comprehensive qualitative review of approaches to assessing research impact will be undertaken to identify ways in which it might be useful to amend or elaborate on the HERG approach. The primary study will involve: analysis of documents and information on the database of completed projects held by the National Coordinating Centre for Health Technology Assessment (NCCHTA); a survey of the principal researchers leading each project funded by the programme; and selected case studies.

Main collaborators: Staff at the NCCHTA, including John Powell, Christine Solomon, Ruairidh Milne and Colin Green.

A team from the Health Economics Research Group (HERG), Brunel University and King’s College London, with funding from the NIHR Service Delivery and Organisation (SDO) Programme, will conduct an evidence synthesis to identify, analyse and synthesize the evidence related to the question: does engagement in research improve performance at a clinician, team, service and organisational level in healthcare organisations? There is no doubt that the findings from medical research can be used to improve healthcare and health outcomes. But does engaging in research help to improve performance across and within the health system? What does engagement in research actually mean? This review seeks to answer these and other questions. We shall examine the use of the terms ‘research engagement’ and ‘performance…in healthcare organisations’ in relation to the conduct, use and impact of research within healthcare systems. We shall also explore the links between these two concepts and examine how these links work, seeking to establish what mechanisms are involved and the various contexts in which they work best.

Funding Body: NIHR
Contacts: Prof. Stephen Hanney

Publications:

- Hanney, S., A. Boaz,T. Jones, and B. Soper. 2013. “Engagement in research: An innovative three stage review of the benefits for healthcare performance.” Health Serv Deliv Res, 1:8.

HERG is collaborating on a project led by RAND Europe to evaluate a major initiative from The Health Foundation: Engaging with Quality in Primary Care. This initiative builds on a previous one from The Health Foundation, the Engaging with Quality Initiative, which was also evaluated by the RAND Europe/HERG team. The initiatives were inspired by the claim that clinicians are attentive to the need to improve quality in healthcare but are often not sufficiently or effectively engaged in this process. In both initiatives a series of projects were funded and, in practice, the evaluations have had a formative element as well as providing a summative evaluation of the whole initiative.

Funding body: The Health Foundation

Contacts: Prof. Martin Buxton, Prof. Steve Hanney & Dr. Bryony Soper (PI: Prof. Tom Laing, RAND Europe)

A team led by HERG and including RAND Europe and the Office of Health Economics will explore how far understanding the nature and extent of time lags in the research process, and where these could and should be addressed by policy, could examined by adopting a process marker model recently developed by Trochim et al (2011) to understand the research process. Trochim et al identified that a key challenge in understanding the research process was the diverse range of models for research translation, with various translation gaps which are differently named and conceptualised in different frameworks. This diversity of models, which in many cases are not clearly defined leads to challenges in comparing and aggregating findings across different studies, a problem echoed by Morris et al (2011) in their work on time lags. This might be combated by a process marker model, in which specific events along the translation pathway are defined as markers, which can be clearly defined and thus the stage of translation understood across studies. Time between process markers could also serve to provide a standardised measure of time lags which is transferrable between studies and can be easily understood. By using this format, it becomes less important whether studies are using a consistent measure to describe time lag, as long as they clearly explain which markers they are studying.

Therefore, we suggest this new approach has great potential to provide a more clearly defined a consistent approach to both the understanding of research translation more generally and to time lags in particular. We propose to test this model by attempting to apply it in a series of case studies to explore the time lags in the development of a number of treatment advances in two fields: mental health and cardiovascular research.

Funding Body: MRC

Contacts: Prof Steve Hanney

This research is an evaluation of the returns from research funded by Australia‘s National Breast Cancer Foundation (NBCF) using the ’Payback Framework‘. It covers all of NBCF‘s research funding programmes since its establishment in 1994, and a more selective case study approach to investigate its high impact research across the various Payback domains. The research will consist of a series of elements:

1. a review of the existing material available from NBCF;
2. a survey of Chief Investigators on selected Project Grants and Fellowships;
3. 16 selected mini-case studies;
4. development of recommendations for routine data collection and monitoring of research impacts, and observations on future research strategy;
5. progress reports for the NBCF Board;
6. production of a report for NBCF.

Funding Body: National Breast Cancer Foundation

Contacts: Dr. Claire Donovan and Prof. Steve Hanney

Funding Body: Wellcome Trust, Cancer Research UK, Academy of Medical Sciences and Department of Healthy Policy Research Programme

Contacts: Prof. Martin Buxton, Prof. Steve Hanney and Matthew Glover