Abstract
Any effort to understand, evaluate, and improve the impact of research must begin with clear concepts and definitions. Currently, key terms to describe research results are used ambiguously, and the most common definitions for these terms are fundamentally flawed. This hinders research design, evaluation, learning, and accountability. Specifically, the terms outcome and impact are often defined and distinguished from one another using relative characteristics, such as the degree, directness, scale, or duration of change. It is proposed instead to define these terms by the kind of change rather than by the degree or temporal nature of change. Research contributions to a change process are modeled as a series of causally inter-related steps in a results chain or results web with three main kinds of results: (i) the direct products of research, referred to as outputs; (ii) changes in the agency and actions of system actors when they are informed/influenced by research outputs, referred to as outcomes; and (iii) tangible changes in the social, economic, environmental, or other physical condition, referred to as realized benefits. Complete definitions for these terms are provided, along with examples. This classification aims to help focus research evaluation appropriately and enhance appreciation of the multiple pathways and mechanisms by which scholarship contributes to change.
Introduction
There are high expectations from the public, research funding agencies, and researchers themselves to contribute to and document impact resulting from their research (Bornmann, 2012; Edler et al., 2012; Wilsdon et al., 2015). Any effort to understand, evaluate, and improve the impact of research must begin with clear concepts and definitions. Currently there is a debilitating lack of clarity and consistency in the use of key terms that describe the results of any intervention, including changes engendered by research. The terms output, outcome, and impact, which are terms used in a typical logic model, are used ambiguously and the most common definitions for these terms are fundamentally flawed (Belcher and Palenberg, 2018). This hinders evaluation, learning, and accountability in academic research as much or more than in any other field. This essay, based on the authors’ experience with conceptualizing and assessing research impact in the social sciences and humanities, applied research, and research-for-development contexts, takes a systems perspective on research impact and offers precise sub-categories of impact to improve clarity.
Established concepts used in research evaluation such as “impact factor” and “high impact research” refer to measures of publication and citations of research, but do not measure actual use or value beyond the academic realm (DORA, 2012; Hicks et al., 2015). There has been increasing attention to the non-academic impacts of research (Bornmann, 2012; Oancea, 2019; Williams, 2020). Alla et al. (2017) conducted a systematic review of definitions of research impact, finding 108 definitions in 83 publications. However, they noted a dominance of what they called bureaucratic definitions and a widespread failure to actually define the term explicitly. The most highly cited definitions were those of the Research Excellence Framework (“an effect on, change or benefit to the economy, society, culture, public policy or services, health, the environment or quality of life, beyond academia” (REF, 2011, p. 26)), the Research Councils of the UK (“the demonstrable contribution that excellent research makes to society and the economy” (Economic and Social Research Council, 2021, para.1)), and the Australian Engagement and Impact Assessment framework (“the contribution that research makes to the economy, society, environment or culture, beyond the contribution to academic research” (Australian Research Council, 2018, p. 5)). While these broad, all-encompassing concepts give attention to societal benefits beyond academia, they all lack precision and require further classification to be useful analytically. They also fail to recognize that research typically contributes to change within complex social, economic, technical, and environmental systems, in conjunction with many other factors. Based on their review, Alla et al. (2017) re-emphasize the need for conceptual clarity, while offering their own definition specific to the mental health field: “Research impact is a direct or indirect contribution of research processes or outputs that have informed (or resulted in) development of new (mental) health policy/practices, or revisions of existing (mental) health policy/practices, at various levels of governance (international, national, state, local, organizational, health unit)” (p. 9).
Gow and Redwood (2020) also give considerable attention to the lack of clarity in interpretation of impact. They devote a chapter to discuss impact terminology and suggest a four-part impact typology: Instrumental; Conceptual; Capacity Building, and Procedural. They do not provide precise definitions for these sub-components of impact, and the authors themselves note that the categories are not mutually exclusive.
The term outcome is also widely used to refer to a step in a results chain. Like impact, outcome is also used ambiguously to refer to everything from the products of research to intermediate and shorter-duration changes stimulated by research, and it is often used as a synonym for impact. Most results chains conceptualize outcomes as resulting from outputs and as precursors to impact. The terms outcome and impact are typically distinguished from one another relatively, based on the degree, directness, scale, or duration of change. For example, the influential OECD (2010) glossary of evaluation terms defined outcomes as “The likely or achieved short-term and medium-term effects of an intervention’s outputs” (p. 28) and impacts as “Positive and negative, primary and secondary long-term effects produced by a development intervention, directly or indirectly, intended or unintended” (p. 24). As Belcher and Palenberg (2018) discuss in detail, these definitions do not support clear, unambiguous distinctions between the terms or the concepts they are intended to define. Of particular relevance is the fact that the temporal dimension of these definitions is not helpful for analytical purposes such as research design, evaluation, learning, and accountability.
All the above impact definitions refer to a ‘contribution’ made by research, but devote most of their attention to the locus of change (i.e., beyond academia). They offer little to help specify, understand, or analyze the nature of the contribution research makes, or to ascertain definitively what is included and what is excluded in the definition. To help clarify the concept and advance thinking about research impact, we therefore propose two more precise sub-categories of impact that are defined absolutely, by the kind of change, rather than relatively, by the degree or temporal nature of change. We recognize that change processes happen in complex systems. Research contributes to a change process within a system and can be modeled as a series of causally inter-related steps in a results chain or results web. There are three main kinds of results from research: (i) the products and services of research, produced directly by a research program, which we refer to as outputs; (ii) changes in the agency of other actors when they use and/or are influenced by research outputs, which we refer to as outcomes; and (iii) tangible changes in the social, economic, environmental, or other physical condition, which we refer to as realized benefits. Complete definitions for these terms are provided below, along with examples. This is a classification of the types of contributions of research and scholarship within a theory of change, not a hierarchy of value.
Societal demands for impact naturally focus on positive changes in social, economic, environmental, or other physical condition. Research is supported with the expectation that it will contribute in some way to improvements in human well-being and environmental conditions. In the development field, the term impact is often used to mean mission-level impact (i.e., changes in social, economic, environmental, and/or physical condition) (Belcher and Palenberg, 2018). However, the term impact is used commonly and ambiguously in standard English language, and in the academic realm it has both a particular meaning (often measured by citations) and a general meaning that includes what we have called outcomes as well as realized benefits (and costs), as exemplified by the definitions cited above. The term is so imprecise in its common usage, and so loaded with pre-existing definitions, that it would be difficult to re-define. We have therefore elected not to propose a new or restricted definition of the term impact. Rather, we are proposing a classification of sub-categories of impact, which are based on the nature of the change. We use “impact” as an overarching term to denote any change caused in whole or in part by an action or set of actions, including research actions.
Proposed definitions
Output
Knowledge, including new insights, technical innovations, institutional models, and other direct products and services produced by a research program. Outputs are produced by actions within a program’s (including partners) sphere of control (see Fig. 1).
-
Examples of outputs include: new research methods, data sets, analyses, discoveries, histories, new theories, policy analyses or recommendations, and artistic performances. Outputs may also include processes such as discussion fora, networking, or capacity building done as part of a research process.
-
Outputs are the actual knowledge, innovations, and services produced by research as well as the media that communicate knowledge and innovation, such as books, journal publications, policy briefs, or patents.
Outcome
A change in knowledge, attitudes, skills, and/or relationships (KASR), ideally manifest as a change in behavior (B), that results in whole or in part from the research process and its outputs. Outcomes may be at the individual, group, organizational, or higher scales.
-
Outcomes occur in actors beyond the research boundary; that is, outside the sphere of control and within the spheres of influence and interest.
-
By this definition, a change in an individual, group, or organization’s KASR is an outcome.
-
If a change in KASR leads to an action or set of actions (a change in behaviorFootnote 1) by an actor in the system, that action may in turn contribute to changes in other actors’ KASR and behavior. Such downstream changes are also defined as outcomes. A change in KASR is an outcome by this definition, but it can only contribute to further change if it leads to some action.
-
In research evaluation, outcomes can be disaggregated into academic outcomes, which refers to influences and changes within the academic realm, and societal outcomes, which refers to changes outside the academic realm.
-
Examples of academic outcomes include adoption and use of new methods, replication of studies, use of data sets, or use of new theories by other researchers.
-
Examples of societal outcomes include changesFootnote 2 in understanding of risk or vulnerabilities; changes in public understanding, values, and attitudes; adoption of new technologies or organizational practices; licensing of patents; new partnerships with community groups; skills and capabilities inculcated through the research experience; shared knowledge and public discourse; new policy or regulations; or creation of a social enterprise.
Realized benefits
A change in economic, social, or environmental condition resulting in whole or in part from a chain of events to which research has contributed. This can manifest as a change in flow or change in state. Benefits/costs may be realized at individual, group, organizational, or higher scales.
-
Realizing tangible social, economic, and/or environmental benefits oftenFootnote 3 involves actors outside the program’s/researcher’s sphere of influence and is the ultimate stage of a complex pathway and change process to which the research has contributed.
-
Examples of realized benefits include: changes in income (flow) or wealth (state), changes in the level of press freedom (state), changes in carbon emissions (flow) or water quality (state), changes in levels of experienced racism, or changes in a person’s or a community’s mental health status.
-
Realized benefits may be positive or negative in the same way an investment can yield a negative return; that is, the change process to which research contributes may have negative or harmful social, economic, and/or environmental consequences for some or all stakeholders. Such negative consequences are sometimes termed “grimpacts” (Derrick et al., 2018, p. 1199).
Research outputs, outcomes, and realized benefits in a theory of change
Figure 1 illustrates a research programFootnote 4 theory of change. The three spheres reflect the fact that the relative influence of any intervention declines as interactions with other actors and processes increase (Hearn, 2010; Montague, 2000). The program has a high level of control over program activities and outputs in the sphere of control. Beyond the program boundary, research outputs inform, influence, and support other actors and their actions (outcomes), alongside many other influences and processes, in the sphere of influence. Ideally, the actions of those other actors will then contribute to realized benefits in the sphere of interest.
In practical terms, the sphere of control includes actions and outputs that can be produced directly by the researcher or research team. This includes actions and outputs produced by collaborators as part of their commitments to a program. If an actor must be persuaded through the provision of knowledge, tools, or advocacy, this change occurs in the sphere of influence. The concept of the sphere of influence attempts to capture the idea that change happens when the KASR of other actors (i.e., not part of the research team) change. These kinds of changes are classified as outcomes of the research if they result in whole or, more likely, in part from the research process and/or output(s). If there are co-produced outputs, it implies that the research process has resulted in KASR and behavior in other actors that would not have happened in the absence of the research, and this change is an outcome. Individually or collectively, changes in behavior that result in part or in whole from the research can lead to realized benefits.
The research program itself is represented in Fig. 1 as a stylized sequence of activities, from top to bottom, within the sphere of control. Activities include developing partnerships with other researchers and/or societal actors and (co-)defining the problem the research will address and the specific questions it seeks to answer. The research then may apply established methods and/or develop new methods to collect and analyze data and (co)create new knowledge and innovations. This list is indicative; not all steps may be present and\or they may occur in different sequence, iteratively, and with or without external actors being involved.
The program’s interactions with and influence in society is represented horizontally, from the sphere of control (program implementation), through the sphere of influence (other actors informed and influenced by research outputs), to the sphere of interest (the tangible benefits to which the research may contribute). The figure tries to represent the dynamic interactions in a complex system. The downward arrows in the sphere of control indicate that each step in the research process contributes to other actions in the research process.
In traditional academic research, the primary aim has been to create new knowledge, search for meaning, and improve understanding. However, research can contribute to outcomes and realized benefits in many ways. Moving from the left to right in the diagram (as indicated by the rightward arrows), each of the individual steps in the research process can produce outputs that contribute independently as well as in combination. For example, the process of developing a partnership may build relationships among stakeholders that have value beyond the program; the research question and/or new methods could stimulate attention and additional research on an important topic; open data policies are increasing the likelihood that data sets will be made available for other uses beyond a program. Each of the steps can contribute to changes in KASR and changes in behavior (B) by other actors. The research process may also be informed and influenced by societal engagement, as represented by the leftward arrows moving from partners, stakeholders, and society back to the program.
The rightward arrow to the second step within the sphere of influence illustrates how changes in KASRB (outcomes) among partners, stakeholders, and society more generally can lead to changes in policy and practice (outcomes) and higher-level system transformations (outcomes), that ultimately lead to changes in social, economic, or environmental condition (realized benefits) in the sphere of interest. This highlights the important role of collaborations and partnerships in co-creating and advancing the use the research-based knowledge and reflects an important rationale for increased use of engaged transdisciplinary research approaches. The circular arrow at the bottom of the diagram represents ongoing stakeholder engagement throughout all stages.
Finally, the figure indicates that the focus of monitoring, evaluation, and learning (MEL) is different at each stage in the impact pathway. Within the sphere of control, the focus is on research quality, broadly defined to include considerations of relevance, credibility, legitimacy, and how research is positioned for use (Belcher et al., 2016; Ofir et al., 2016). Is the research focus, design, and implementation appropriate and sound? Within the sphere of influence, research evaluation needs to focus on whether and how research has contributed to outcomes. Is there evidence that the research has stimulated or contributed to changes in KASRB, and is it reasonable to expect further knock-on changes? In the sphere of interest, the focus is on the scale and scope of realized benefits and analysis of the relative contribution of research.
It is important to emphasize that this is a classification, not a hierarchy of value. It is intended to support research evaluation by distinguishing the kinds of changes that research can enable, catalyze, and contribute to. In order to assess what difference research makes, we need to know what kind of change we are looking for. Change happens in complex systems and, as illustrated in Fig. 1, most change happens outside the control of a research program. The kind and degree of change to which any research program contributes and the timeframe over which that change happens will depend on many other factors, including the nature of the issue, the current state of knowledge, and the political climate. In some domains of research (e.g., many Engineering and Applied Sciences), external stakeholders often have close linkages with researchers, such that the pathway through the spheres of influence and interest to realized benefits can be relatively direct and rapid. In Health research, the interface of researchers with individuals with lived experience of a disease provides engagement and learning, and enables more effective translation of research outputs to practice and realized benefits of the affected communities. The outputs of scholarship in Social Sciences and the Humanities may profoundly influence understanding, appreciation, values, and indeed the actions of individuals, organizations, or society more generally (i.e., outcomes). These kinds of changes are often difficult to observe, difficult to measure, and difficult to attribute, and occur over long timeframes, but have value in and of themselves. They may also contribute to realized benefits, but in most cases the attribution challenges are insurmountable because there are so many other causal factors. This classification aims to help focus research evaluation appropriately and enhance appreciation of contributions that scholarship makes to change in more diffuse ways. In any research evaluation, we need to look at outcomes as the primary indicator of research effectiveness.
Conclusion
There has been a great deal of discussion in the literature about research impact, how to define it, and how to measure it, but current definitions and usage remain vague and ambiguous. This essay combines two main ideas to help achieve conceptual clarity. First, we explicitly recognize that research contributes to change within systems as sequential causal processes (with feedback and iteration), in combination with other processes and other actors. We have provided a generic model of a research-to-impact process that: illustrates the declining relative influence of an intervention in a system, shown as spheres of control, influence, and interest; indicates typical actions within a research process; appreciates that individual actions in the research process may make valuable contributions independently as well as in combination, especially in engaged co-produced research; and identifies that the focus of monitoring, evaluation, and learning is different at each stage in the process. Second, we propose that it is practical and useful to classify research results into different kinds. Outputs are the products and services produced directly by research. Outcomes are the changes in KASR experienced by other actors who have been influenced by the outputs of research. Those changes in KASR may also contribute to changes in behavior and, thereby, to subsequent outcomes. Realized benefits are tangible changes in the social, economic, environmental, or other physical conditions. In this framing, research impact includes both outcomes and realized benefits. This classification aims to help focus research evaluation appropriately and enhance appreciation of the multiple pathways and mechanisms by which scholarship contributes to change.
Notes
Change in behavior is understood broadly. It is any action that would not otherwise have taken place. It could be something as simple as one person telling another what they have learned, to transformative changes in individual, organizational, institutional, or societal policies or practices. We are asking “Who does what differently as a result of the research?”
Change is assessed against a (hypothetical) counterfactual; i.e., what would have happened in the absence of the intervention. Thus, the change may be a decision to maintain the status quo or to avoid implementing a program.
In some types of research, such as participatory action research, benefits may be realized by participants.
We use the term “program” to refer to a body of research work done by an individual researcher or a team of researchers. The discussion could equally refer to a “project”.
References
Alla K, Hall WD, Whiteford HA, Head BW, Meurk CS (2017) How do we define the policy impact of public health research? A systematic review. Health Res Pol Syst 15(1):1–12. https://doi.org/10.1186/s12961-017-0247-z
Australian Research Council (2018) Engagement and Impact EI 2018 Assessment Handbook. https://www.arc.gov.au/engagement-and-impact-assessment/ei-key-documents
Belcher BM, Rasmussen KE, Kemshaw MR, Zornes DA (2016) Defining and assessing research quality in a transdisciplinary context. Res Eval 25(1):1–17. https://doi.org/10.1093/reseval/rvv025
Belcher B, Palenberg M (2018) Outcomes and impacts of development interventions: toward conceptual clarity. Am J Eval 39(4):478–495. https://doi.org/10.1177/1098214018765698
Bornmann L (2012) Measuring the societal impact of research: research is less and less assessed on scientific impact alone—we should aim to quantify the increasingly important contributions of science to society. EMBO Rep 13(8):673–676. https://doi.org/10.1038/embor.2012.99
Derrick GE, Faria R, Benneworth P, Budtz-Petersen D, Sivertsen G (2018) Towards characterising negative impact: Introducing Grimpact. Proceedings of the 23rd International Conference on Science and Technology Indicators 2018, Leiden, The Netherlands. https://research.utwente.nl/files/57761417/STI2018_paper_201.pdf
DORA (2012) The San Francisco Declaration Research Assessment. https://sfdora.org/read/
Economic and Social Research Council (2021) What is impact?. https://esrc.ukri.org/research/impact-toolkit/what-is-impact/
Edler J, Georghiou L, Blind K, Uyarra E (2012) Evaluating the demand side: new challenges for evaluation. Res Eval 21(1):33–47. https://doi.org/10.1093/reseval/rvr002
Gow J, Redwood H (2020) Impact in international affairs: the quest for world-leading research (1st edn.). Routledge
Hearn S (2010) Outcome mapping: planning, monitoring and evaluation. Outcome mapping learning community. https://www.slideshare.net/sihearn/introduction-to-outcome-mapping
Hicks D, Wouters P, Waltman L, De Rijcke S, Rafols I (2015) Bibliometrics: the Leiden Manifesto for research metrics. Nat News 520(7548):429–431. https://doi.org/10.1038/520429a
Montague S. (2000) Circles of influence: an approach to structured, succinct strategy. http://www.pmn.net/wp-content/uploads/Circles-of-Influence.pdf
Oancea A (2019) Research governance and the future(s) of research assessment. Palgrave Commun 5(1):1–12. https://doi.org/10.1057/s41599-018-0213-6
OECD-DAC (2010) Glossary of key terms in evaluation and results based management. https://www.oecd.org/dac/evaluation/2754804.pdf
Ofir Z, Schwandt T, Duggan C, McLean R (2016) RQ+ research quality plus: a holistic approach to evaluating research. https://idl-bnc-idrc.dspacedirect.org/bitstream/handle/10625/56528/IDL-56528.pdf?sequence=2
REF (2011) Assessment framework and guidance on submissions. Bristol, United Kingdom. https://www.ref.ac.uk/2014/media/ref/content/pub/assessmentframeworkandguidanceonsubmissions/GOS%20including%20addendum.pdf
Williams K (2020) Playing the fields: theorizing research impact and its assessment. Res Eval 29(2):191–202. https://doi.org/10.1093/reseval/rvaa001
Wilsdon J, Allen L, Belfiore E, Campbell P, Curry S, Hill S, Jones R, Kain R, Kerridge S, Thelwall M, Tinkler J, Viney I, Wouters P, Hill J, Johnson B (2015) The metric tide: report of the independent review of the role of metrics in research assessment and management. Sage Publications, London
Acknowledgements
For their sponsorship of a 2019 workshop by the Federation for the Humanities and Social Sciences and CASRAI, as well as their valuable comments and feedback at various stages in the development of this article, the authors would like to acknowledge Suzanne Board, Laura Beaupre, Yolande Chan, Kyle Demes, Heather Frost, Laura Hillier, Sandra Lapointe, Sharon Murphy, Nilgun Onder, Emile Paquin, David Phipps, Sally Rutherford, Lisa Shapiro, Karine Souffez, Louise Michelle Verrier, and David Watt. We are also indebted to Anna Hatch for her insightful comments on an earlier draft of this article. Brian Belcher’s work on this has been supported by the Canada Research Chairs Programme, the Social Sciences and Humanities Research Council (SSHRC), Ashoka Canada, and the Forests, Trees and Agroforestry Consortium Research Program.
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Competing interests
The authors declare no competing interests.
Additional information
Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this license, visit http://creativecommons.org/licenses/by/4.0/.
About this article
Cite this article
Belcher, B., Halliwell, J. Conceptualizing the elements of research impact: towards semantic standards. Humanit Soc Sci Commun 8, 183 (2021). https://doi.org/10.1057/s41599-021-00854-2
Received:
Accepted:
Published:
DOI: https://doi.org/10.1057/s41599-021-00854-2