Bulletin advised on and rewrote hundreds of REF 2014 impact case studies for several universities in the lead up to the assessment exercise.
Universities came to us because, in order to demonstrate the impact their research had made, they recognised the need for an engaging, well-evidenced narrative that adhered to HEFCE’s guidelines.
Speaking at the Westminster Briefing event Research Management: REF, Impact & The Future in London on July 28, Dr Steven Hill, Head of Research Policy at HEFCE, confirmed that the way in which the REF case studies were written in many cases proved crucial.
He said: “Good impact was necessary but not sufficient to get a good score. It is about delivering great impact but also about conveying that impact effectively.”
Stephen Holgate CBE, Clinical Professor of Immunopharmacology at the University of Southampton and Chair of REF Main Panel A, believes the introduction of impact to the REF was a success.
“The end results really have delivered. We’ve moved into a better place as a consequence of it.”
He added that international members of the REF panel were ‘wowed’ by the outputs and “went rushing back to their countries to encourage their governments to do the same”.
Holgate confirmed the use of professional writers had a positive effect on the case studies. “The narrative was highly influential in generating confidence in the impact. Clarity and identification of benchmarks along the route were helpful.”
Holgate made the following further observations on impact case studies:
- Most case studies that were ‘unclassified’ failed to establish a clear link between the impact and the underlying research – or a link to underpinning research that was 2* quality.
- He noted that for Panel A only one of the six underpinning research outputs needed to be 2*. “I hope they don’t change that,” he remarked.
- There were difficulties in defining the exact contribution of the research to the eventual impact.
- Some impact types were underrepresented in Panel A e.g. public understanding of science.
- It was surprising that not more case studies were based on research carried out by researchers who had left the institution.
- The narrative needed to be explicit in naming institutions and the people that enabled the impact to occur.
- High-scoring case studies often involved interdisciplinary teams.
- Exaggerated claims of impact shold be avoided.
- Some impacts were ‘partial’ in that they fell short of the final end product.
- Benefits of increasing employment were underplayed.
- The impact template was difficult to assess and overlapped with the “environment” section.
Looking ahead to the next REF, Holgate suggested that better ways to verify the impact claimed should be explored. And he raised the idea of rewarding what he called “super impacts”: impacts of huge reach and significance that deserve to be recognised above other 4* case studies.
Katy McKen, Head of Research Information and Intelligence at the University of Bath and a member of REF Panel C, had five lessons to share:
1. Read the exam question
- ensure research is eligible
- ensure impact is eligible
- explain any anomalies (e.g. researchers moving during REF period)
- make it easy for the panel!
2. Tell a good story
- be clear on contribution of submitting HEI to research (e.g. in collaborations)
- explain research clearly
- link impact to research
- do not overstate the impact – be realistic
- be clear on contribution of submitting HEI to impact
3. Provide appropriate and compelling evidence
- evidence should offer diverse mix of independent qualitative and quantitative sources that directly support claims being made
- panel did not have access to section 5 references (they had to submit formal audit requests to see them) so case studies that incorporated evidence from section 5 in section 4 were viewed very favourably
- follow through the ‘stories of change’ (i.e. what changed as a result of the research)
4. Communicate reach and significance of impact
- describe context in which the impact takes place to give an indication of its significance
- set out the original objectives of the research e.g. what was the intended reach? If you maximised the possible reach of your research then you were likely to score higher
5. Appropriate use of language and presentation
- make narrative coherent and present a linear, chronological story of development
- case study should be written so it is understood by a reader without specialist knowledge
- clear presentation makes it easy to read: sub-headings, adequate spacing, pictures or diagrams were well received.
- make sure the case study is self contained e.g. pack evidence from section 5 into section 4
