Nature | Comment

Reproducibility: Team up with industry

Combining commercial and academic incentives and resources can improve science, argues Aled Edwards.

Article tools

Illustration by Pete Ellis

The scientific community is bustling with projects to make published results more reliable. Efforts are under way to establish checklists, to revamp training in experimental design, and even to fund disinterested scientists to replicate others' experiments. A more efficient strategy would be to rework current incentives to put less emphasis on high-impact publications, but those systems are entrenched, and public funders and universities are ill-prepared for that scale of change.

To catalyse change, industry must step up to the plate. I have learned this first hand, as head of the Structural Genomics Consortium (SGC), a research charity funded by business, government and other charities. If more companies contributed funds and expertise to efforts such as ours, I believe it would create a system that rewards science that is both cutting-edge and reproducible.

Here I share my experience of running a collaboration between academic and industrial researchers over the past 12 years, and distil the principles that have made for reliable research.

Then and now

In the late 1990s, several drug companies concluded that academic researchers were only half-heartedly pursuing structural information about human proteins, information crucial to the design of new drugs. They proposed pooling resources with government bodies to fund this work. And so, in 2004, drug company GlaxoSmithKline, the major UK biomedical funder the Wellcome Trust and science funders from Canada launched the SGC. Our task? To generate human protein structures and place them in the public domain.

The motivation varied. Public and charitable funders emphasized scientific novelty and high-quality publications. Industry wanted research relevant to drug-discovery efforts. For example, companies insisted on structures of human proteins rather than those from other species, even though many of these would have been of great scientific interest. All funders demanded quantifiable milestones, unrestricted use of data and reagents, and the right to withdraw support from the project if it underperformed (see 'Milestones exceeded').

Source: SGC

The number of drug companies participating has grown to eight. The SGC currently disburses more than US$20 million to 250 scientists in 6 dedicated laboratories. Our researchers include principal investigators, postdocs, technicians and graduate students. At any given time, as many as 50 scientists in industry collaborate with SGC scientists. Projects now extend beyond protein structures to the discovery of chemical probes (small molecules useful for studying protein function), and to hospital-based collaborations that explore the effects of these tools in cells from patients. All the resources we create are non-proprietary and readily available.

In 2010, as the SGC began to attract the interest of pharmaceutical companies that had never before participated in consortia dedicated to producing public information, I asked, “Why us?”, hoping to hear about our amazing scientific intellect. The answer was more prosaic: “We can repeat your work.” Back then, before reports emerged that fewer than half of biomedical papers could be reproduced (L. P. Freedman et al. PLoS Biol. 13, e1002165; 2015), that did not make much sense to me. It does now.

Ironically, it was our work with industry that had helped make our research so reliable in the first place. Our industrial colleagues helped to design processes to increase the chances that they could depend on our work. From the outset, we were operating within a system in which continued funding was tied to research that proved useful.

Eight principles

In my opinion — shared by my long-standing industrial collaborators — several mutually reinforcing factors characterize this research system. Each is essential. It is the combination, rather than any single principle, that is key.

Require full commitment, and reward efficiency. Focus and organization are necessary for success. As a precondition for receiving funding, scientists agree to dedicate all their research time to the project. This is the aspect in which the SGC differs most from other schemes involving academic partnerships with industry. For participating scientists, the advantage is efficient, predictable funding: as long as scientists achieve their milestones, expenses are covered without the need for grant applications. It also incentivizes scientists to innovate. If they achieve their targets without using all their available funds, they can use the remainder to pursue their curiosity.

Define objectives that cannot be achieved with current technology. Many scientists and public funders believe that formal milestones — a requirement for industry investment — are the antithesis of discovery research. The solution is to create 'stretch goals'. In our case, funders provided a list of 2,000 human proteins and a directive to solve the structures of 350 — knowing full well that that goal was not achievable with the technologies of the day. This worked. To meet these and subsequent milestones, SGC scientists developed new methods and have published more than 800 peer-reviewed papers (of which 60 have appeared in Nature and its eponymous sister journals).

Illustration by Pete Ellis

Establish clear quality criteria and make them public. Milestones must be unambiguous, or else they can be gamed. In our case, we asked a group of independent experts from academia and industry to craft quantitative criteria to judge research outputs. For instance, we defined how different a protein sequence needed to be from others in the Protein Data Bank to count as 'novel', and specified acceptable levels of precision and error. All these criteria were published on our website (see go.nature.com/4qncnj). SGC scientists knew that protein structures would not be 'counted' unless they met or exceeded the publicly available criteria.

Mandate data sharing. Mechanisms for sharing progress and enforcing transparency promote reproducibility. We rolled out electronic lab notebooks in 2004. There was some initial resistance, but scientists quickly realized that ready access to information from their colleagues helped their own research. Electronic lab notebooks make it easier to document and disseminate detailed experimental procedures, and also reduce the risk that researchers will cherry-pick data.

Subject work to independent oversight before public release. Participants should not be the judges of whether they have met a milestone. We created an independent advisory board of academic and industry scientists to assess the quality of each structure, probe, or other tool before it was released into the public domain. This external body prevents us from loosening the quality criteria for research outputs if achieving the original goals turns out to be harder than expected.

This oversight was more tedious and humbling than we were used to. Consortium scientists had to prepare documents defending how each criterion had been met, and external scientists who intend to use the potential tool tend to be tougher judges of robustness than the average peer reviewer.

Over the years, however, external vetting became simply part of our practice. Without a doubt, it contributed to our reputation for reproducible and meaningful research. On two occasions, SGC scientists were convinced that a chemical probe (see go.nature.com/sediul) had met our quality criteria, only to have an oversight committee raise what turned out to be legitimate concerns, which were subsequently addressed.

Enshrine public ownership for all research outputs. The value of foundational science is greater when more people use the research, so materials must be disseminated as freely as possible. We avoided legal encumbrances such as formal Material Transfer Agreements (MTAs) where possible, developed 'click-wrap' MTAs where necessary, and encouraged vendors to provide all our reagents without restriction. The result is that many scientists outside the SGC use our materials. Currently, the SGC contributes 5% of all the plasmids deposited at the repository Addgene; last year, vendors received more than 5,000 orders for chemical probes generated by the SGC.

“Done well, industry–academic collaborations could recreate the engine that powered successful enterprises.”

Ensure that industry and academic scientists collaborate. Industry must provide expertise as well as funds. Collaboration with industry scientists engenders a shared desire to succeed, and creates a sense of ownership of a project. The different motivations also create productive tension. For example, scientists in academia have strong incentives to publish rapidly. Unfortunately, this can lead to the publication of stories that are true only under narrowly defined conditions. By contrast, industry scientists push for validation using a range of orthogonal experiments; these alternative ways of evaluating the same research tool ensure that the results are broadly useful. Publications and release must happen by consensus. By balancing these desires, we achieve an optimal combination of innovation, timely dissemination and reproducibility.

Create an active governing body. Many academic projects, including a number that are co-funded by industry, create 'friends and family' governance bodies that advise, but do not govern. The SGC opted for a different model. Our board includes senior executives from every major funder. It can halt a project, change leadership and direct strategy. Any meaningful change to the project budget requires unanimous board approval. Because of this responsibility, in-person attendance has been nearly 100% for every quarterly board meeting for more than 12 years. A productive public–private-sector tension also plays out at the board. If industry requests a delay in releasing research results to further review data quality, public funders balance the scientific merits of this request with the need to disseminate the output rapidly.

Expand the model

Adherence to these principles builds an ecosystem that supports reproducible, innovative research. Scientific publications are no longer the sole units of achievement; reaching predefined milestones and making useful tools are also key to continued funding.

This approach cannot work for every scientific question or for every scientist. But there are many important scientific questions in which the long-term needs of industry and the talents of academic researchers are aligned. Here the research effort could be readily focused within a single, organized project where unrestricted access to the research results would benefit everyone. The oil and gas sector might pool resources to develop novel remediation technologies; the forestry sector might create a consortium to explore cellulose-degrading technologies; the renewables sector might collaborate to identify better energy-storage technologies.

Such public–private collaborations are in industry's interest. Companies need foundational science to innovate and for long-term profitability. In the past, researchers at BASF, Dupont and Bell Labs invented technologies that transformed science, business and daily life — from transistors to radar to synthetic fertilizers — and they won Nobel prizes. Today, support for internal company research has been slashed, and academic research is not filling the gap because it is unpredictable, unfocused and, often, unreliable. Done well, industry–academic collaborations could recreate the engine that powered successful enterprises.

Such partnerships should be more widespread. They can advance important areas of research. Equally important, the research community gains knowledge and tools, and the practices that make for robust science might diffuse beyond the collaborations to raise the quality of science overall.

Journal name:
Nature
Volume:
531,
Pages:
299–301
Date published:
()
DOI:
doi:10.1038/531299a

Author information

Affiliations

  1. Aled Edwards is chief executive of the Structural Genomics Consortium.

Corresponding author

Correspondence to:

Author details

For the best commenting experience, please login or register as a user and agree to our Community Guidelines. You will be re-directed back to this page where you will see comments updating in real-time and have the ability to recommend comments to other users.

Comments for this thread are now closed.

Comments

Comments Subscribe to comments

There are currently no comments.

sign up to Nature briefing

What matters in science — and why — free in your inbox every weekday.

Sign up

Listen

new-pod-red

Nature Podcast

Our award-winning show features highlights from the week's edition of Nature, interviews with the people behind the science, and in-depth commentary and analysis from journalists around the world.