Table 4 Examples of evidence used to show significance and reach of impacts from research in high and low-scoring impact case studies from REF2014.

From: Writing impact case studies: a comparative study of high-scoring and low-scoring case studies from REF2014

Examples from high-scoring case studiesExamples from low-scoring case studies
• Evidence of benefits for specific beneficiary groups that have happened during the eligibility period (rather than anticipated future impacts)
• Evidence is shown to come from credible sources and is used to substantiate specific claims, e.g. official data showing 430% increase in approvals of biopesticides, or peer-reviewed analysis showing that the BBC changed its coverage based on recommendations from research
• Evidence that a new policy or practice works and has delivered benefits (e.g. via an internal or external independent review, primary or secondary data collection or testimonials) or limiting the claim to changes in policy or practice (where it is too early to assess their effect)
• Use of robust research or evaluation designs to evidence impact, with robustness demonstrated through triangulation for qualitative and mixed methods evaluations, or through statistical significance and treatment-control designs (e.g. randomised control trials)
• Research leads to an activity or other pathway, but with no evidence that these pathways led to actual impacts (in some cases the claim is for potential future impacts)
• Evidence is used vaguely, e.g. “evaluative data indicate the majority of users have…changed the way they work” without describing the number of users or the nature of the change
• The impact of future policy implementation is claimed (or implied), but the evidence only relates to policy formation
• Poorly designed evaluation undermines credibility of evidence, e.g. no baseline, before/after or comparison group to demonstrate changes were a result of the research
• Testimonials describe impacts of their organisation rather than the research, or describe engagement with researchers but no impacts
• Over-reliance on estimates (e.g. in testimonials) without more concrete evidence
• Addressing a challenge that was uniquely felt by a particular group on a sub-national scale
• Successfully helping hard-to-reach groups that others have previously not been able to reach
• Reaching significantly more than previous initiatives, e.g. poetry events that attracted “twice the national average for such events”
• Evidence of strong pathways to impact from well-respected international organisations or groups with strong influence at other relevant scales, for example via funding for research or dissemination of research via policy documents or new working practices
• Reach is claimed internationally or across multiple groups (sometimes implicitly), but convincing evidence is only presented for national (or sub-national) benefits or for a small proportion of the groups who are said to have benefited
• Claims of reach based on the global reach of an organisation or initiative using the output of research without specifying the impact the research activity or output has had on this organisation