The true impact of a paper, a researcher’s work or a journal’s value cannot be captured by any one metric but requires a more nuanced approach that combines quantitative and qualitative measures.
In linguistic terms, the metaphorical definition of impact is simple: according to the Merriam-Webster Dictionary, impact is the force of impression of one thing on another, a significant or major effect. In contrast, the meaning of impact in science is manifold, and the ways to measure it are similarly numerous. They are also imperfect.
The most widely known among them is the journal impact factor (JIF). Released yearly by Clarivate, the 2-year JIF measures the number of citations received in a given year for the papers published by the journal in the 2 previous years, divided by the number of citable items published in that journal during those 2 years. Much ink has been spilled about the ways in which the JIF is flawed, misused and gamed1. Suffice it to emphasize that it is a journal-level metric not meant to be used as a proxy for the quality of individual papers or researcher success, but instead as a measure of how frequently the average paper of a journal is cited in a particular year, and thus as an indicator of a journal’s reach in the scientific community.
The criticism about the deficiencies and rampant misuse of the JIF has led to many constructive efforts over the years to improve this metric and the strategies used for research assessment2, and to provide a wider variety of fit-for-purpose scientometrics3. Needless to say, all such metrics come with their specific limitations. Thus, the 2-year median score of the Nature journals is similar to the 2-year JIF, but as the name suggests, this calculates the median rather than the mean and thus is not affected by outliers. Nevertheless, it retains the limitation of assessing citations for papers published during a short 2-year period. The 5-year JIF extends the article publication time frame but retains the deficiencies of its more widely used 2-year counterpart. The Eigenfactor Score calculates the number of citations received in a given year by articles published by the journal in the previous 5 years, with the caveat that citations from highly cited journals are weighted to have a higher contribution to the score.
Considering the individual article, the Immediacy Index provides the average citations it receives in its year of publication, and at a journal level informs on the speed of citation of a journal’s content. For individual articles, the number of citations can act as a proxy of the scientific community’s interest in that particular study, whereas the level of coverage in mass and social media, reported as altmetrics, can be an indicator of more immediate and broader public interest — not to be conflated with a measure of true scientific value or quality.
A number of author-level metrics also exist, among which the h-index is the most widely used. A researcher is assigned a number (h) when they have an (h) number of publications with at least (h) numbers of citations. Inherent in this correlation is the limitation that researchers with lower output — e.g., early-career researchers or those with fewer collaborations — will be disadvantaged, whereas older, active researchers and prolific collaborators will benefit. It also means that the h-index is not sensitive to highly cited papers, something that the g-index, a variant of this metric, was devised to correct. Many other variants exist, each with their own set of advantages and disadvantages.
The effort to quantify impact at the article, author and journal level is understandable — numbers simplify comparisons, data are easily digestible and open to different types of analysis, and scientists are typically prone to number crunching. However, this reductionist approach fails to capture the complexity and qualitative nature of scientific contributions. For instance, the significance of a research study goes far beyond citations and publicity and instead lies firmly in its specific scientific merits. Depending on its focus, a paper on cancer research and oncology could be impactful at the academic, clinical, industry or societal level on the basis of a number of hard-to-measure, typically longer-term effects: the degree to which it advances foundational knowledge; extends biology to the understanding of disease; leads to the development or improvement of treatments and diagnostic means; moves preclinical findings to commercial or clinical translation through pharma or biotech development and clinical trials; leads to changes in clinical practice; and highlights social issues and informs health policy. A separate set of hard-to-quantify, impactful outcomes of published research are the collaborations it generates, the follow-up studies it seeds, the funding it secures, the career steps it spurs, and the recognition of author excellence it generates.
A researcher’s impact is similarly complex and is not quantifiable solely by publication output and number of citations. Beyond the cumulative effects of their published work on their specific field of study, a researcher’s impact includes the effects of mentorship; the resulting connections of trainees and collaborators; contributions to science through elevation to positions of responsibility and prestige; and advocacy and advancement of science, research and societal causes based on the researcher’s ideas, sensibilities, network and reputation in the field.
And what of a journal’s impact? Here, too, metrics reveal only part of the picture. No journal can deny their importance in providing a mathematical basis of how it is being received, and no journal can afford to ignore their pitfalls. As Nature Cancer grows, data continue to accrue and our journal metrics page is becoming populated. Imperfect though they may be, these metrics represent a journal-level indication of the interest Nature Cancer and its publications have been garnering in the scientific community. But we are more interested in the impact that goes beyond scientometrics — that hard-to-measure impact that opens novel lines of enquiry and picks up old ones in new and exciting ways, connects disciplines in a meaningful manner, fosters translation and change in practice, and aids career development and collaboration. We are in it for the long-term impact that is not necessarily reflected in time-defined mathematical calculations but is recognizable in the body of a journal’s publications, the quality of its science, the rigor and fairness of its editorial and peer-review practices, the integrity and scholarship of its editorial staff, and the spirit of community it inspires through its pages. We are in it for the long haul.
About this article
Cite this article
The meaning of impact. Nat Cancer 3, 901 (2022). https://doi.org/10.1038/s43018-022-00435-5