Main

Science has been described as an endless frontier1,3,15,16. New opportunities and challenges continuously emerge, from synthetic biology or climate change to the COVID-19 pandemic, and researchers and research organizations must consider adapting their research portfolios to address these emergent demands4,5,17,18,19. Adaptability is thus crucial for scientific and technological progress1,3,15, and adaptive success or failure can underpin the relative progress or collapse of organizations, economic regions and societies1,3,15,16,20,21.

The adaptability of research streams hinges on researchers, who must regularly consider the direction of their work and their potential to engage with new areas. Researchers face consequential choices across large or small changes in their research directions, but the degree to which research directions are adaptable depends on fundamental trade-offs and unknowns. On one hand, shifts in research may be difficult14, because the specialization of expertise12,13,22, the design of funding systems23,24 and the nature of research incentives, culture and communities7,25,26,27 may all limit the capacity of a given individual to respond effectively to changing opportunities and demands28,29,30,31,32. On the other hand, the value of novelty8,33,34 and exploration6,9,35 in creative search suggests that moving further from one’s usual research area might be particularly fruitful10,11,14,36, and new entrants or ‘outsiders’ to a given area are sometimes thought to be especially capable of transformative ideas7,37. Indeed, a researcher who continues to exploit an existing direction may face diminishing returns and miss opportunities afforded in other areas6,38. Exploring new areas might be risky, but it may also be more likely to produce high-impact insights.

Here we study the adaptability of scientists and inventors, and examine the outcomes when researchers work in areas nearer or further from their existing research portfolio. We introduce a measurement framework for research pivots and then study adaptability in both general and specific settings. We first apply the measurement framework at high scale across scientific and technological domains, studying millions of scientific articles indexed by Dimensions from 1970 to 2020 and US patents granted from 1985 to 2020 (Supplementary sections 1.1–2). The core finding is that there is a substantial pivot penalty, meaning that the further a researcher moves from their previous work, the worse the research performs in terms of citation impact, publication success and a host of other outcomes. The negative effects of pivoting occur for individual researchers, across wide-ranging fields of inquiry, and have been increasing over time. We then evaluate the pivot penalty in terms of canonical conceptual frameworks, and investigate potential mechanisms, drawing on ideas of reputation and audience32,39,40,41, as well as creativity frameworks in the production of new ideas6,8,12. Finally, we turn to case studies of substantial interest to science and in which exogenous events can elicit research pivots. We study ‘push’ events, in which existing knowledge is revealed to be incorrect or unreliable, pushing researchers away from their previous research streams. We also study a ‘pull’ event—the COVID-19 pandemic—that drew researchers into an important new research area. We find that despite the wide-ranging nature of these events, researchers pivot to an unusually large degree after these events, and the pivot penalty persists in each case. The pandemic also allows us to examine a consequential, society-scale event and the capacity of science as a whole to address the new research demands. We conclude with a discussion about the implications of these findings for researchers, research organizations and science policy.

Measurement framework

To quantify pivots for researchers, we calculated a cosine-similarity metric that measures the extent to which a given new work departs from a researcher’s previous body of work (Fig. 1a and Pivot size in the Methods). For papers, we considered the referenced journals, comparing the focal work with the previous body of work for that author. The pivot measure, Φ, varies on the [0,1] interval. It takes the value 0 (zero pivot) if the focal paper draws on exactly the same distribution of journals as the author’s previous work, and takes the value 1 (full pivot) if the focal paper draws on an entirely different set of journals. In the patent context, for which journal information is not available, we use technological field codes to measure pivots (see Pivot size and Supplementary section 2.2 for details and alternative constructions of the pivot measure).

Fig. 1: Quantifying research pivots.
figure 1

a, The pivot measure compares a focal work against previous works by the same researcher. An increasing value on the [0,1] interval indicates a larger pivot from the researcher’s previous work. In the sciences, journals are used to define research areas (pictured); in patenting, technology classes are used. b, The distribution of author pivots in 2020 (n = 8.32 million author-by-paper observations) is dispersed across the [0, 1] interval. c, The distribution of inventor pivots in 2020 (n = 166,000 inventor-by-patent observations) is dispersed across the [0, 1] interval and is bimodal. COVID-19 papers (b) showed higher median pivots than other papers in 2020. Fig. 1a, icons adapted from the Noun Project (https://thenounproject.com).

Source data

Full size image

Figure 1 shows the distribution of pivoting behaviour, focusing on the year 2020. Pivoting values ranged across the [0, 1] interval in both the science and patenting contexts, indicating that pivoting is prevalent for both scientists and inventors (Fig. 1b,c). We also observed a sharp increase in pivot size for research related to COVID-19, in that scientists who engaged with COVID-19 exhibited unusually large pivots; whereas papers not related to COVID-19 in 2020 have a median of \(\bar{\Phi }=0.60\), COVID-19 papers present a substantially larger median pivot size of \(\bar{\Phi }=0.82\) (P < 0.001, chi-squared test for median differences). The variable nature of pivot size is particularly prominent in patenting, for which we observed a bimodal distribution (Fig. 1c), showing a tendency for both small and large jumps. Supplementary section S2.2 provides further analysis of these patterns, demonstrates their robustness across alternative pivot measures and offers specific examples of pivoting.

The pivot penalty

When scientists and inventors shift away from their earlier research, a central question is how impactful their new work becomes. We first considered 25.8 million papers published from 1970 to 2015 across 154 fields. To quantify impact, we calculated a binary, paper-level indicator for whether a given work was in the upper 5% of citations received in its field and publication year42. In Fig. 2a, the data are presented as binned scatterplots, with papers grouped by pivot size into 20 equally sized groups and showing the mean rate of high-impact papers for each group (see Binned scatterplots in the Methods). Figure 2a reveals a striking fact: across the whole of science, papers with larger average pivots have a systematically lower propensity for high impact. Indeed, we observed a large, monotonic decrease in the average hit rate as the pivot size rises. The lowest-pivot work had high impact 7.4% of the time, which is 48% higher than the baseline rate (P < 0.001 in one sample t-test), whereas the highest-pivot work had high impact only 2.2% of the time, which is a 56% reduction from the baseline (P < 0.001). Figure 2b normalizes impact for individual researchers using regressions with individual fixed effects (see Regressions with individual fixed effects in the Methods), showing an impact penalty that is both substantial and less steep than in the raw data. Within a given researcher’s portfolio, the lowest-pivot work was 2.1% more likely (P < 0.001 in regression t-test) to have high impact than that researcher’s other work, and their highest-pivot work was 1.8% less likely (P < 0.001) to have high impact, again showing large deviations from the 5% baseline.

Fig. 2: The pivot penalty.
figure 2

a, In a study of 25.8 million papers published from 1970 to 2015, papers with higher pivot size have substantially lower probabilities of being high impact. b, For a particular author, relative impact for their papers declines steeply with pivot size. c, In a study of 1.72 million US patents granted from 1980 to 2015, patents with higher pivot size have substantially lower probabilities of being high impact. d, For a particular inventor, relative impact for their patents declines with pivot size. e,f, Over time, the relationship between pivot size and high-impact works has become increasingly negative in science publishing (e) and patenting (f).

Source data

Full size image

We next considered 1.72 million patents granted from 1980 to 2015 across 127 technology classes, and we similarly calculated the patent-level hit rate based on being in the upper 5% of citations received in the patent’s technology classification and application year. We again found a monotonic decrease in impact as pivot size increased (Fig. 2c). The lowest-pivot patents had high impact 8.0% of the time, which is 60% higher than the baseline rate (P < 0.001 in one-sample t-test), but the highest-pivot patents had high impact only 3.8% of the time, a 24% reduction from the baseline (P < 0.001). This decline in impact with larger pivots was robust to measuring inventor pivots at any technology-classification level, from the broadest to the narrowest (Supplementary Fig. 1). Figure 2d further normalizes impact for individual inventors and continues to show the pivot penalty.

The relationship between pivot size and impact in science has become increasingly negative over the past five decades, both in the raw data (Fig. 2e) and when looking at individual researchers (Extended Data Fig. 1a,b). Furthermore, these findings generalize widely across scientific fields. Studying each of the 154 subfields separately, the negative relationship between impact and pivot size held for 93% of fields, and the increasing severity of the pivot penalty over time occurred in 88% of all scientific fields (Supplementary Table 1). Turning to patenting, we again observed an increasingly steep pivot penalty with time (Fig. 2f). Studying 127 level-2 technology classes separately, the negative relationship between impact and pivot size held in 91% of classes, with the severity of the pivot penalty growing over time in 76% of patent classes (Supplementary Table 2). This steepening pivot penalty among inventors was also seen when using broader or narrower technological classifications (Supplementary Fig. 2). Earlier years for patenting showed flatter, less-monotonic relationships in the raw data (Fig. 2f) and within inventors’ portfolios (Extended Data Fig. 1c,d).

The findings of a substantial impact penalty are robust to many alternative measures and sample restrictions (see Supplementary sections 2.2–2.3 and 3.1–3.2 for analysis and further examination of high-pivot cases and outlier fields). Robustness tests considered: alternative time windows to determine citation impact (Extended Data Fig. 2); alternative measures of citation impact (Extended Data Fig. 3); sample restrictions to papers with larger reference counts (Supplementary Table 3); alternative pivot-size computation based on referenced papers’ field coding, as opposed to their journals (Supplementary Fig. 3); alternative field encodings for patents (Extended Data Fig. 4 and Supplementary Figs. 1, 2 and 4); and hand checks on high-pivot researchers (Supplementary section 3.1).

When examining outcomes, one can also look beyond the citation impact. For papers, we further measured whether a published paper was referenced in a future patent3,43, indicating the use of the idea beyond science. There was a large decline in patent references to high-pivot articles, with the probability of being cited in a patented invention declining by 43% (P < 0.001 in a two-sample t-test) when comparing the highest-pivot with the lowest-pivot vigintiles of papers (Extended Data Fig. 5b). We also examined the propensity for preprints to be published and found that higher-pivot preprints were published at substantially lower rates, with publication rates for the highest-pivot papers declining by 35% (P < 0.001) compared with the lowest-pivot papers, indicating another form of the pivot penalty (Extended Data Fig. 6). For patents, we considered the invention’s market value based on how a company’s stock price moved in response to the patent being issued44. The market value of a patented invention decreased steeply with pivot size, declining by 29% (P < 0.001) when comparing the highest-pivot with the lowest-pivot patents (Extended Data Fig. 7). These findings indicate that the pivot penalty also appears when considering publication success, practical use and market value, pointing to a constellation of outcomes that go beyond the citation behaviour within a community of researchers.

Altogether, we observed striking empirical regularities that generalize across science and technology. Despite the distinct nature of scientific articles and patents, the institutional contexts in which they are produced, the wide range of research fields and their alternative outcome measures, they have remarkable commonalities: for both scientists and inventors, greater pivots bring large penalties, and this increases over time.

Conceptual frameworks and mechanisms

Our findings indicate that researchers face substantial challenges when entering new subject areas, heightening concerns in innovation communities that research with wide reach or new orientations is difficult12,13,14,18,25. Entering new areas may be challenging as a matter of reception, whereby a scholar has difficulty penetrating new audiences, and it may be challenging as a matter of idea generation, as scholars can face problems generating valuable ideas outside their key areas of competency. To further inform the nature of the pivot penalty, we next examined the pivot penalty in view of both reputational perspectives and idea generation.

An established reputation in a local research community may provide impact advantages within that community but be a relative disadvantage outside it39. For example, the ‘Matthew effect’39,40 suggests advantages of established eminence within a community, but ‘typecasting’32,41,45 may undermine the reception when entering new areas. These and other reputational considerations indicate that the pivot penalty may emerge because researchers move beyond their usual audience. To test these considerations, we first examined pivots holding the researcher’s field or local audience fixed. Specifically, we examined what happens when a given researcher publishes multiple papers with different pivot sizes but in the same time frame and field, even in the same journal (Supplementary Table 4). We found that the pivot-penalty regression coefficient was approximately 26% less steep (Supplementary Table 4) when an individual published in the same journal, an attenuation that is consistent with a weakening of reputational forces when looking within a common audience, but most of the relationship remained. The pivot penalty thus persisted when the researcher published in a consistent field or before a consistent, local readership. A related approach considered impact within a given, distant audience. Recalling the findings for patented applications (Extended Data Fig. 5) and market value (Extended Data Fig. 7), the pivot penalty also appeared when examining how inventors draw on science or how investors value inventions. These evaluations are made by individuals who are far away from the focal researcher. In sum, the pivot penalty appears not simply as a matter of movement across fields, or from a local audience to a distant audience. Rather, it appears for a researcher within a given field or journal, and it appears within distant communities focused on practical use and market returns.

Reputational considerations may be further informed by considering career stage. Specifically, younger researchers, with less-formed reputations, may see less advantage (the Matthew effect) from staying in a given area or less penalty (typecasting) from venturing outside it41,46. Studying career stage, the pivot penalty was slightly stronger (1.6% steeper per year, P < 0.001, regression coefficient t-test; Supplementary Table 5) with advancing career age, consistent with these reputational frameworks. Yet the pivot penalty appears regardless of career stage, including very early in the career (Supplementary Table 5). The findings continued to indicate adaptive challenges, beyond the force of established reputations, when entering new research terrain.

Turning to frameworks of idea generation, a canonical perspective emphasizes an ‘explore versus exploit’ trade-off in creative search. Here, exploitation involves lower-risk but potentially lower-return search around the edges of one’s current focus, whereas exploration involves higher-risk but potentially higher-return forays into more-distant areas6,37,38. Related views indicate an advantage of outsiders in bringing new perspectives and driving breakthroughs10,11,47. Our analyses looked at upper-tail outcomes, but it is possible that the value of large pivots lies in even rarer, more extreme, positive outcomes. Surprisingly, however, we found that high-pivot research is increasingly under-represented at higher impact levels, whereas low-pivot research has advantages (Supplementary Fig. 5a,b). For example, studying the upper 1% and 0.1% of scientific works by citation impact, papers in the lowest decile of pivot size were over-represented by 65% and 91%, respectively (P < 0.001 in one-sample t-tests). By contrast, papers in the highest pivot size decile were under-represented by 69% and 73%, respectively (P < 0.001), among the upper 1% and 0.1% of citation impact. Rather than indicating a trade-off between risk and reward in exploratory search, or outsider advantages, these findings continue to indicate a fundamental difficulty of venturing into new areas.

Alternative idea-generation frameworks emphasize the value of specialized expertise. These frameworks link creative advantages less to outsider ideas and more to the accumulated facts, theories and methods built in an area by previous scholars47,48. The emphasis on expertise and the value of prior knowledge is consistent with Newton’s famous statement that “if I have seen further, it is by standing on the shoulders of giants”49. Furthermore, the steepening of the pivot penalty with time is consistent with increasingly narrow expertise as science progresses and knowledge deepens12,50,51. The publication findings (Extended Data Fig. 6) showing a monotonic decline in publication success rates as pivot size increased, and where the highest-pivot preprints were 35% less likely to be published in any journal, suggest the presence of substantive issues with these works, consistent with challenges in moving beyond one’s established areas of expertise. Related creativity frameworks emphasize that new works can be seen as new combinations of existing material52,53,54. Previous studies have shown that high-impact research is characterized primarily by highly conventional mixtures of prior knowledge but also tending to inject, simultaneously, a small dose of atypical combinations that are unusual in previous research8,55. Following this literature, we further measured the novelty and conventionality of combinations in a given paper and related these measures to pivot size and impact (Extended Data Fig. 8 and Supplementary Table 9). We found that high-pivot work was associated with a higher propensity for atypical combinations (Extended Data Fig. 8a), a feature also reflected in work linking inventors who switch fields to new technology combinations14. For example, 31% of the lowest-pivot vigintile of papers were characterized by high tail novelty, and 49–58% of papers in high-pivot vigintiles had high tail novelty. In other words, when pivoting, a researcher not only does something new personally, but also tends to introduce previously unseen combinations of knowledge to the broader research domain. However, at the same time, high-pivot papers showed distinctly low conventionality (Extended Data Fig. 8b), locating a key characteristic that such exploratory work tends to miss: a prevalence towards well-established mixtures of knowledge. For example, 79% of the lowest-pivot papers exhibited mixtures with high median conventionality, whereas only 27–30% of papers in high-pivot vigintiles had this characteristic. These findings indicate that researchers, as they shift to new areas personally, are equipped for novelty but limited in their relevant or conventional expertise, underscoring the difficulty researchers may face in venturing beyond their specialized knowledge.

Pivoting in response to external events

The pivot penalty indicates that larger pivots are strongly associated with lower impact. However, the research landscape is constantly shifting, and researchers must weigh opportunities nearer to and further from their current research streams. To further probe pivoting behaviour and the pivot penalty, we considered external events that may provoke researchers to pivot. External events can provide quasi-experimental settings and help to establish causal interpretations of the pivot penalty, and may further inform the tensions regarding how researchers navigate a shifting research landscape.

We first considered events that may push researchers away from an existing research stream. Specifically, previous research is sometimes revealed as incorrect or unreliable, which may encourage researchers who had been building on that work to move in new directions. Here we focus on paper retractions, which are of growing interest to the science community56,57,58. Using Retraction Watch and the Dimensions database, we identified 13,455 retractions over the 1975–2020 period. As a treatment group, we considered researchers whose work referenced a retracted paper before it was retracted (but who were not authors of the retracted study). As a control group, we considered researchers who referenced other papers appearing in the same journal and year as the retracted paper. We further used coarsened exact matching59 to match treated and control authors by their publication rates before the retraction year. We then compared pivots and hit rates between the treatment and control groups, over the four years before and the four years after retraction events, in a difference-in-differences design (Fig. 3a and Difference-in-differences).

Fig. 3: Pivots and retraction events.
figure 3

a, This difference-in-differences analysis compares treated scientists who directly cite a paper before its retraction with control scientists who cited other papers in the same journal and year as the retracted paper, but not the retracted paper. There are 164,988 treated authors who cited a retracted paper at least once (18,505 treated authors who cited it at least twice) before its retraction but are not themselves authors of the retracted papers. Pivot size and impact of papers from these treated scientists is compared with papers from equal numbers of matched control scientists before and after the year of retraction. b, Pivot size significantly increases for treated scientists relative to control scientists after the retraction (0.025 ± 0.001 s.e. pivot-size increase, P < 0.0001, regression, n = 5.82 million author-by-paper observations). The effect is larger when focusing on scientists who cited the retracted paper at least twice (0.037 ± 0.001 s.e. pivot-size increase, P < 0.0001, regression, n = 2.96 million author-by-paper observations). c, Hit rates fall for treated scientists after retraction (−0.004 ± 0.001, P < 0.0001, n = 5.82 million), and again the effect is stronger for those citing the retracted paper at least twice (−0.007 ± 0.001, P < 0.0001, n = 2.96 million). d,e, Year-by-year analysis comparing treated and control authors further shows that the increase in pivot size is statistically significant (P < 0.001) starting immediately in the retraction year (d) and the decrease in hit rate becomes statistically significant (P < 0.05) starting the year after the retraction (e). In b–e, bars and markers represent the difference-in-differences regression coefficients, and the whiskers show the 95% confidence interval derived from the regression standard errors (see Difference-in-differences). Fig. 3a, icons adapted from Apple.

Source data

Full size image

We found that pivot sizes increased markedly after a retraction event (Fig. 3b). Consider first the 164,988 treated researchers who referenced a retracted paper at least once before its retraction. The mean pivot size for these researchers’ works after the retraction increased by 2.5% (P < 0.001) compared with control researchers’ works. We also studied a smaller treatment group of 18,505 researchers who referenced a retracted paper multiple times, indicating more intensive use. For this group, pivoting was larger, with mean pivot sizes increasing by 3.7% (P < 0.001) after the retraction, compared with the control authors (Fig. 3b). Turning to paper impact, treated authors experienced a 0.4% decline (P < 0.001) in hit rate after the shock, compared with control authors (Fig. 3c). Among treated authors who drew on the retracted study multiple times, we saw not only larger pivots (Fig. 3b), but also a larger 0.7% decline (P < 0.001) in hit rates after the retraction event (Fig. 3c).

Difference-in-differences analyses on a year-by-year basis reinforced these findings. Figure 3d shows an increase in pivoting starting in the retraction year. Similarly, Fig. 3e shows a sustained decline in hit rates after the retraction. Two-stage least-squares regressions, with the retraction event as an instrument, further show that these ‘push’ pivots predict substantial declines in impact (Supplementary Table 6). Robustness tests using hit rates and citation counts over alternative periods, or using alternative definitions of the treated group, showed confirmatory results (Supplementary section 2.7.1, Supplementary Table 6 and Supplementary Figs. 6 and 7). We further considered a smaller case study of replication failures, rather than retractions, drawing on a landmark 2015 study of reproducibility in psychology60, for which 100 papers were quasi-randomly chosen for evaluation and 64 contained non-reproducible results. Deploying the same treatment and control method as for paper retractions, this smaller study provided confirmatory results for pivoting and impact (Supplementary Section 2.7.2 and Supplementary Table 7). Altogether, we saw pivoting increases and hit-rate declines in response to these external shocks. These analyses further confirm the findings of the pivot penalty, now in response to external events that push authors into new areas.

Beyond push-type events, researchers may also be pulled into new subject areas when new and important research questions emerge. This leads to our second case study, analysing how researchers shifted to engage with the COVID-19 pandemic. The advent of the pandemic enabled large-scale investigation of individual researcher pivots and further showed how science as a whole responds to a new and consequential demands on the research community. Indeed, confronted by COVID-19, the world looked to science to understand, manage and construct solutions, all in a rapid fashion. Given that few researchers were studying coronaviruses or pandemics before 2020, and none were studying COVID-19 specifically, the emergence of COVID-19 called on researchers across the frontiers of knowledge to consider shifting their work to address new, high-demand research questions61,62,63.

Figure 4 shows that pivoting to address COVID-19 was widespread. Although the earliest papers on COVID-19 did not appear until January 2020 (refs. 64,65), by May 2020, 4.5% of all new scientific papers were related to COVID-19 (Fig. 4a). Furthermore, although fields differed in the share of their papers that engaged COVID-19, all fields produced at least some COVID-19-related research (Fig. 4b and Extended Data Fig. 9). Health sciences exhibited the greatest COVID-19 orientation, but the social-science fields of economics, education and law also addressed COVID-19 intensely, speaking to the pandemic’s socio-economic challenges66,67. Furthermore, studying each field that had at least 20 COVID-19 papers, mean pivot sizes were larger for COVID-19 papers than for other papers in that field (Fig. 4d; mean difference positive for 100% of fields, t-test significant at P < 0.05 for 97% of fields). Figure 4c also tracks a cohort of scientists across the body of their work, comparing authors who wrote a COVID-19 paper in 2020 with a control set of authors who did not (Supplementary section S2.8). We found that pivot size presented a clear jump for COVID-19-related work, for which COVID-19 authors pivoted to an unusual degree compared with their own publication history (P < 0.001 in t-tests of means), their non-COVID 2020 papers (P < 0.001) and the control authors (P < 0.001). In sum, unusually large individual pivots were a widespread phenomenon as scientists sought to address COVID-19.

Fig. 4: Pivots and the COVID-19 pandemic.
figure 4

a, Science rapidly shifted to COVID-19 (COVID) research in 2020, when COVID-19 publications rose to 4.5% of all science publications in May 2020 and maintained high rates thereafter. b, Health sciences and social sciences featured the strongest responses, but all scientific fields engaged in COVID-19 research. c, Scientists who wrote COVID-19 papers pivoted to a greater extent than they did in their previous work, in their other 2020 work, or than matched control scientists did. d, Comparing COVID-19 and non-COVID-19 papers in each field in 2020, unusually large pivots have been a universal feature of COVID-19 research. e, COVID-19 papers experienced an impact premium, but the pivot penalty appeared in both COVID-19 and non-COVID-19 work. Comparing at the median pivot sizes (dashed lines), the COVID-19 impact premium was substantially offset by the pivot penalty, given its larger median pivot size. f,g,h, Engaging new collaborators was particularly common for COVID-19 researchers, who worked with new collaborators to an unusual degree compared with their own previous work, their other 2020 publications and with control scientists (f). Nonetheless, the pivot penalty persisted for big and small teams (g) and when engaging new or existing co-authors (h). i,j, Higher-pivot work was substantially less likely to acknowledge funding support in the sciences as a whole (blue) and among COVID-19 papers (red). COVID-19 papers were particularly unlikely to acknowledge grant support (i), yet the pivot penalty appeared even among both funded work and non-funded work (j). k, Although individual, collaborative and funding features sharply conditioned the adaptive response of science, in regression analysis they did not individually or collectively overcome the fundamental pivot penalty. Coronavirus icon adapted from the Noun Project (https://thenounproject.com).

Source data

Full size image

We next turned to impact. Given that 2020 papers have had less chance to receive citations68, we examined journal placement, for which each journal was assigned the historical hit rate of its publications within its field and year (Supplementary section 2.3). Figure 4e considers papers published in 2020, separating them into 82,900 COVID-19 papers and 2.63 million non-COVID papers. We found a premium associated with COVID-19 papers, as reflected by the upward shift in journal placement, consistent with the extreme interest in the pandemic. Yet the negative relationship between pivoting and impact persisted: comparing the highest-pivot and lowest-pivot bins, COVID-19 and non-COVID-19 papers had declines in hit rate of 61% and 59%, respectively (P < 0.001 in t-tests of means). Thus, scientists who ventured further from their previous work to write COVID-19 papers were not immune to the pivot penalty; rather, they produced research with less impact, on average, relative to low-pivot COVID-19 papers. The pivot penalty in COVID-19 papers also appeared net of individual fixed effects (Supplementary Fig. 8). Importantly, the pivot penalty was sufficiently steep that the COVID-19 impact premium was mostly offset by the unusually large pivots associated with COVID-19 research. For example, the upper 45% of COVID-19 papers by pivot size had lower average journal placement than did non-COVID papers with median or smaller pivot size.

In sum, the ‘pull’ nature of COVID-19 presented two extremely strong yet contrasting relationships regarding impact. On one hand, this work experienced an impact premium, consistent with the value of researching high-demand areas. On the other hand, greater pivot size markedly predicted less-impactful work. These findings underscore a central tension for individual researchers and the adaptability of science in response to external opportunities: working in a high-demand area has value, but pivoting leads to penalties that offset it.

Building on the science of science literature, we further considered numerous potential moderating factors and forms of heterogeneity that may facilitate pivots. These include researcher career stage, productivity, project-level team size, the use of new co-authors, and funding8,35,42,69 (see Binned scatterplots and Supplementary section 2.9). For example, early-career researchers may have greater creative flexibility7,26,47, and larger team size or new co-authors may extend reach33,70. When examining impact, however, we found that the pivot penalty persisted, regardless of these features (Fig. 4f–j and Supplementary Table 5). We further used regression methods to incorporate detailed controls for all these potential moderating factors together and found that the pivot penalty appears net of all these features (Fig. 4k), highlighting the depth and breadth of the adaptive challenge.

Discussion

Science must regularly adapt to new opportunities and challenges. The findings in this study, however, highlight difficulties in adapting research streams, with implications for individual researchers, research organizations, and science and society as a whole. At an individual level, a researcher must consider whether to continue exploiting a familiar research stream or explore opportunities that lie further away. Research on creativity reveals the value of exploration, novelty and outsider advantages6,7,8,9,10,11,33,34,35,37, indicating a risk-versus-reward trade-off when researchers venture further from their existing expertise. However, other viewpoints emphasize the value of deep expertise, especially in drawing on the frameworks, facts and tools built by previous scholars12,47. As Einstein observed: “Knowledge has become vastly more profound in every department of science. But the assimilative power of the human intellect is and remains strictly limited. Hence it was inevitable that the activity of the individual investigator should be confined to a smaller and smaller section”50. Consistent with both Einstein’s observation and previous studies indicating increasing specialization and disadvantages when inventors switch fields12,14, we found that researchers face systematic challenges to pivoting their research, and these increase with time. This pivot penalty applies to knowledge production in both science and technology, generalizes across research subfields and extends beyond impact and publication measures to the practical use and market value of ideas, external to the research domain. Our analyses deploy numerous proxies for quality, such as citation impact, home-run rates, publication success, novelty, conventionality and applied value, but the intrinsic quality of a paper or patent is a multidimensional and open concept.

The pivot penalty also appears in response to external events that may push a researcher away from a given area or pull them into a new one. The enormous demand for COVID-19-related research attracted numerous researchers and provided an impact premium, yet the pivot penalty continued to appear strongly among scholars who reached further to engage with COVID-19 research. All told, the pivot penalty applies to a range of outcomes that are of central interest to researchers and research institutions, and it applies in high-stakes contexts for society as a whole.

The pivot penalty, and its steepening with time, raises key questions for research organizations and research policy. For example, businesses and other organizations are often displaced by new entrants52,71, despite R&D efforts by the incumbents, which often fail to understand or embrace new technological opportunities6,38,72. The pivot penalty underscores this challenge and points towards tactics such as ‘acquihires’, in which a research organization seeks to hire relevant experts, rather than expecting success by pivoting their existing personnel73,74.

More broadly, the pre-positioning of researchers seems to be a fundamental constraint on adaptability. In Pasteur’s words, “chance favours only the prepared mind”, and without the pre-positioning of relevant human capital, the COVID-19 pandemic would probably have been even more costly. Portfolio theory suggests diversified investments as a key tool to manage risk75, but the pivot penalty indicates that adjustments to the research portfolio are governed by substantial inertia76. From this perspective, investing explicitly in a diverse set of scientists is crucial from a risk-management standpoint. A diverse portfolio of investments can then have essential roles in advancing human progress in ordinary times7,77 and in expanding the capacity to confront emerging challenges.

Science and technology present evolving demands from many areas, from artificial intelligence and genetic engineering to climate change, creating complex issues, risks and urgency. This study shows that pivoting research is difficult, with researchers’ pivots facing a growing impact penalty. The pivot penalty not only appears generally across scientific fields and patenting domains, but also arises around important events in science, including when previous research areas become devalued, for example after a paper has been retracted, and when high-demand areas emerge, such as the COVID-19 pandemic. Nevertheless, studying adaptability in different settings and timescales, including longer-run research shifts, are key areas for future work. For example, researchers should consider whether to give up in the likely event of a failed pivot or instead further develop their expertise in the new area and stick to the new path. Exploring such sequential dynamics may help us to better understand how to create conditions to enable adaptive success. Finally, pivoting to address emerging challenges is not unique to science and technology, but may underpin the dynamics of success and survival for individuals, companies, regions and governments across human society5,72,78,79,80,81, indicating that the pivot penalty may be a generic property of many social and economic systems, with potential applicability in broader domains.

Methods

Pivot size

We quantified researcher pivots using a cosine-similarity metric (Fig. 1a). Specifically, in the sciences, for an author i and a focal paper j, we calculated a vector, \({{\bf{R}}}_{{\rm{i}}}^{{\rm{j}}}\), representing the distribution of journals referenced by j. Similarly, we counted the frequency in which different journals were referenced in i’s previous work, defining a vector Ri. An individual’s work includes any paper for which the individual is a listed author. The pivot measure, \({\varPhi }_{{\rm{i}}}^{{\rm{j}}}\), is then defined as 1 minus the cosine of these two vectors:

$${\varPhi }_{{\rm{i}}}^{{\rm{j}}}=1-\frac{{{\bf{R}}}_{{\rm{i}}}^{{\rm{j}}}\cdot {{\bf{R}}}_{{\rm{i}}}}{\parallel {{\bf{R}}}_{{\rm{i}}}^{{\rm{j}}}\parallel \parallel {{\bf{R}}}_{{\rm{i}}}\parallel }.$$
(1)

The measure \({\varPhi }_{{\rm{i}}}^{{\rm{j}}}\) therefore took the value 0 if the focal paper drew on exactly the same distribution of journals as the author’s previous work, and took the value 1 if the focal paper drew entirely on new journals for that author. The measure featured in the main text calculates pivoting in the focal paper compared with the past three years of the author’s work. We also calculated our measure by using all previous work of a given author, arriving at similar conclusions (Supplementary section 2.2.1 and Extended Data Fig. 10). Finally, we considered the pivot measure based on the fields of the cited references, rather than their journals, and again found confirmatory results for our main analyses using this alternative measure (Supplementary section 2.2.1 and Supplementary Fig. 3).

For patents, given that journal information was not available, we used technological field codes to define the reference vectors. Specifically, we used the distribution of Cooperative Patent Classification technology field codes among a patent’s cited references to build the reference vectors and cosine similarity metric in equation (1). These technology codes are hierarchical, providing alternative levels of granularity in defining technology areas. Our main analyses used the detailed level-4 technological classification (comprising 9,987 distinct technology areas). We further examined all possible classification levels in the Supplementary Information, considering pivoting from the broadest level-1 classification level (9 sections) to the most detailed level-5 classification (210,347 subgroups). Intuitively, the pivot distribution for inventors shifted left when using broader technology categories (Extended Data Fig. 4), so that inventors pivoted less from their broadest technology areas (the section or section-class level). Regardless of the technological classification used, the pivot penalty was robust (Supplementary Fig. 1).

Outcome measures

We used citation-based and non-citation-based outcome measures. Our citation-based measures normalized outcomes for each work by its field and year. For papers, the primary citation measure was an indicator for being in the upper fifth percentile among all articles published in the same year and the same field. The field designation was the L1 field of research designation, for which there are 154 fields in the Dimensions database. For patents, we similarly used an indicator for being in the upper fifth percentile of citations received among all patents from the same year and technology area, using the Cooperative Patent Classification class-level designation, for which there are 128 technology areas.

As presented in the Supplementary Information, we considered numerous alternative citation-based measures. These include smoother (non-binary) outcomes, in which a paper’s citation count is normalized by the mean citation counts to articles in the same field and publication year. We further considered the outcome as the percentile rank of the paper’s citations among all articles published in the same field and year. To examine time frames, we further considered citation counts over two, five and ten-year forward citation windows. Finally, we considered alternative binary indicators to further emphasize the locus of the very highest-income work, defining a ‘hit’ paper as being alternatively in the upper 10, 5, 1, 0.5 or 0.1% percent of all publications in a given field and year. Supplementary section 2.3.2 provides further details and associated robustness tests for all these alternatives.

Among the non-citation-based measures, we considered numerous other outcomes. These included measures of publication success, for which we considered preprints from 2015 to 2018 and examined whether they were successfully published over an ensuing five-year window. Drawing on the Reliance on Science database43, we examined whether a paper appeared as a prior art reference in a future patent, providing an indicator for the usefulness of the idea beyond science3. We considered journal placement for recent work. For patents, we also used stock-market event study data44, providing a market-value measure for patents in publicly traded firms. Supplementary section 2.3.3 provides further details and results for all these outcomes.

Binned scatterplots

To reveal potentially nonlinear relationships between pivot size and outcome variables, we use binned scatterplots82. In Fig. 2a, papers are ordered by average pivot size along the x axis and binned into 20 evenly sized groups. Each marker is placed at the mean (x,y) value within each group. Binned scatterplots of raw data are further presented in Figs. 2c,e,f and 4e,g–j, and Extended Data Figs. 2, 3, 5–8 and 10b. Student’s t-tests are used to test mean differences from baseline rates (one sample t-tests) or when comparing outcomes between high and low pivot-size vigintiles (two sample t-tests) in raw data. For simplicity, we report P < 0.001, but note that with observation counts in the millions, these mean tests tend to reject common means with extremely high t-statistics and extremely low P values.

Figure 4e uses the binned-scatterplot approach for papers in the year 2020, splitting them into articles related or unrelated to COVID-19. Similarly, Fig. 4g–j presents binned scatterplots, further splitting the 2020 papers according to the noted criteria (team size, use of new collaborators and funding). In Fig. 4k, we account for multivariate controls. We consider regression of the form

$${{\rm{Impact}}}_{{\rm{i}}}=\alpha +f({{\rm{Pivot}}\_{\rm{size}}}_{{\rm{i}}})+{\boldsymbol{\theta }}{{\bf{X}}}_{{\rm{i}}}+{\varepsilon }_{{\rm{i}}},$$

where Xi is a vector of control variables with associated vector of coefficients Θ; f(Pivot_sizei) allows for a nonlinear relationship between the outcome and pivot size; and εi is the error term. Control variables include fixed effects for average previous impact, author age, team size, number of new collaborators and an indicator variable for funding. In practice, we ran two regressions to residualize pivot size and impact, net of the controls, following the Frisch–Waugh–Lovell theorem. Figure 4k presents the binned scatterplot for the residualized measures.

Regressions with individual fixed effects

The panel regression with individual fixed effects in general takes the form:

$${{\rm{Impact}}}_{{\rm{ipt}}}={\mu }_{{\rm{i}}}+{\gamma }_{{\rm{t}}}+f({\rm{Pivot}}\_{{\rm{size}}}_{{\rm{ipt}}})+{\boldsymbol{\theta }}{{\bf{X}}}_{{\rm{ipt}}}+{\varepsilon }_{{\rm{ipt}}},$$

where i indicates a given researcher, p indicates a given work (paper or patent) and t indexes the year (publication year for a paper and application year for a patent); μi are individual-fixed effects, γt are time-fixed effects and Xipt is a vector of other potential control variables. Observations are at the paper-by-researcher level. As before, we allowed for potentially nonlinear relationships between pivot size and impact, and hence took a non-parametric approach. Specifically for Extended Data Fig. 1, we generated bins of pivot size and included indicator dummies for a work appearing in the relevant bin. Given the very large number of individual fixed effects, we ran these models in Stata using the reghdfe command suite83. Standard errors are clustered at the researcher level. The statistical significance of different pivot-size bin coefficients was calculated using t-tests. For simplicity, we report P < 0.001, but note that with observation counts in the millions, these tests present extremely high t-statistics and extremely low P values.

Difference-in-differences

When studying external shocks, we continued to use the researcher panel data model with individual-fixed effects. We implemented standard difference-in-differences methods, comparing treated researchers with control researchers, before and after the external event. The regressions take the form:

$${{\rm{Pivot}}\_{\rm{size}}}_{{\rm{ipt}}}={\mu }_{{\rm{i}}}+{\gamma }_{{\rm{t}}}+\beta {{\rm{Treat}}\_{\rm{Post}}}_{{\rm{ipt}}}+\gamma {{\rm{Post}}}_{{\rm{ipt}}}+{\varepsilon }_{{\rm{ipt}}}$$
$${{\rm{Impact}}}_{{\rm{ipt}}}={\mu }_{{\rm{i}}}+{\gamma }_{{\rm{t}}}+\beta {{\rm{Treat}}\_{\rm{Post}}}_{{\rm{ipt}}}+\gamma {{\rm{Post}}}_{{\rm{ipt}}}+{\varepsilon }_{{\rm{ipt}}},$$

where Postipt is an indicator for the period after the shock. The indicator for being in the treatment group is absorbed with an individual’s fixed effect and so does not appear separately in the regression. Treat_Postipt is an indicator for being in the treatment group after the shock and provides the reported difference-in-differences estimate in Fig. 3. The implications of the external event for pivot size and the reduced form results for impact are shown in Fig. 3b,c. We also present ‘event study’-style difference-in-differences plots in Fig. 3d,e, showing how the treatment effect evolved before and after the retraction date. Here we replace the Treat_Postipt variable and Postipt variable with a series of relative year indicators and their interactions with treatment status.

Reporting summary

Further information on research design is available in the Nature Portfolio Reporting Summary linked to this article.