Nature | News Feature

Reproducibility crisis: Blame it on the antibodies

Antibodies are the workhorses of biological experiments, but they are littering the field with false findings. A few evangelists are pushing for change.

Article tools

Illustration by Nik Spencer/Nature

In 2006, things were looking pretty good for David Rimm, a pathologist at Yale University in New Haven, Connecticut. He had developed a test to guide effective treatment of the skin cancer melanoma, and it promised to save lives. It relied on antibodies — large, Y-shaped proteins that bind to specified biomolecules and can be used to flag their presence in a sample. Rimm had found a combination of antibodies that, when used to 'stain' tumour biopsies, produced a pattern that indicated whether the patient would need to take certain harsh drugs to prevent a relapse after surgery. He had secured more than US$2 million in funding to move the test towards the clinic.

But in 2009, everything started to fall apart. When Rimm ordered a fresh set of antibodies, his team could not reproduce the original results. The antibodies were sold by the same companies as the original batches, and were supposed to be identical — but they did not yield the same staining patterns, even on the same tumours. Rimm was forced to give up his work on the melanoma antibody set. “We learned our lesson: we shouldn't have been dependent on them,” he says. “That was a very sad lab meeting.”

Antibodies are among the most commonly used tools in the biological sciences — put to work in many experiments to identify and isolate other molecules. But it is now clear that they are among the most common causes of problems, too. The batch-to-batch variability that Rimm experienced can produce dramatically differing results. Even more problematic is that antibodies often recognize extra proteins in addition to the ones they are sold to detect. This can cause projects to be abandoned, and waste time, money and samples.

Many think that antibodies are a major driver of what has been deemed a 'reproducibility crisis', a growing realization that the results of many biomedical experiments cannot be reproduced and that the conclusions based on them may be unfounded. Poorly characterized antibodies probably contribute more to the problem than any other laboratory tool, says Glenn Begley, chief scientific officer at TetraLogic Pharmaceuticals in Malvern, Pennsylvania, and author of a controversial analysis1 showing that results in 47 of 53 landmark cancer research papers could not be reproduced.

A few scientists who have been burned by bad experiences with antibodies have begun to speak up. Rimm's disappointment set him on a crusade to educate others by writing reviews, hosting web seminars and raising the problem in countless conference talks. He and others are calling for the creation of standards by which antibodies should be made, used and described. And some half a dozen grass-roots efforts have sprung up to provide better ways of assessing antibody quality.

But it is too soon to call the cause a movement. “There are all these resources out there, but nobody uses them and many people aren't even aware of them,” says Len Freedman, who heads the Global Biological Standards Institute, a non-profit group in Washington DC committed to improving biomedical research. “Most vendors have no incentive to change what's going on right now, even though a lot of the antibody reagents suck.”

Buyer beware

Take the example of Ioannis Prassas, a proteomics researcher at Mount Sinai Hospital in Toronto, Canada. He and his colleagues had been chasing a protein called CUZD1, which they thought could be used to test whether someone has pancreatic cancer. They bought a protein-detection kit and wasted two years, $500,000 and thousands of patient samples before they realized that the antibody in the kit was recognizing a different cancer protein, CA125, and did not bind to CUZD1 at all2. In retrospect, Prassas says, a rush to get going on a promising hypothesis meant that he and his group had failed to do all the right tests. “If someone says, 'Here is an assay you can use,' you are so eager to test it you can forget that what has been promised is not the case.”

Most scientists who purchase antibodies believe the label printed on the vial, says Rimm. “As a pathologist, I wasn't trained that you had to validate antibodies; I was just trained that you ordered them.”

Antibodies are produced by the immune systems of most vertebrates to target an invader such as a bacterium. Since the 1970s, scientists have exploited antibodies for research. If a researcher injects a protein of interest into a rabbit, white blood cells known as B cells will start producing antibodies against the protein, which can be collected from the animal's blood. For a more consistent product, the B cells can be retrieved, fused with an 'immortalized' cell and cultured to provide a theoretically unlimited supply.

Three decades ago, scientists who needed antibodies for their experiments had to make them themselves. But by the late 1990s, reagent companies had started to take over the chore.

Today, more than 300 companies sell over 2 million antibodies for research. As of 2011, the market was worth $1.6 billion, according to global consultancy Frost & Sullivan.

Devastating effects

There are signs that problems with antibodies are having broad and potentially devastating effects on the research record. In 2009, one journal devoted an entire issue to assessing the antibodies that are used to study G-protein-coupled receptors (GPCRs) — cell-signalling proteins that are targeted by drugs to treat various disorders, from incontinence to schizophrenia. In an analysis3 of 49 commercially available antibodies that targeted 19 signalling receptors, most bound to more than one protein, meaning that they could not be trusted to distinguish between the receptors.

The field of epigenetics relies heavily on antibodies to identify how proteins that regulate gene expression have been modified. In 2011, an evaluation4 of 246 antibodies used in epigenetic studies found that one-quarter failed tests for specificity, meaning that they often bound to more than one target. Four antibodies were perfectly specific — but to the wrong target.

Scientists often know, anecdotally, that some antibodies in their field are problematic, but it has been difficult to gauge the size of the problem across biology as a whole. Perhaps the largest assessment comes from work published by the Human Protein Atlas, a Swedish consortium that aims to generate antibodies for every protein in the human genome. It has looked at some 20,000 commercial antibodies so far and found that less than 50% can be used effectively to look at protein distribution in preserved slices of tissue5. This has led some scientists to claim that up to half of all commercially available antibodies are unreliable.

But reliability can depend on the experiment. “Our experience with commercial antibodies is that they are usually okay in some applications, but they might be terrible in others,” says Mathias Uhlén at the Royal Institute of Technology in Stockholm, who coordinates the Human Protein Atlas.

Researchers ideally should check that an antibody has been tested for use in particular applications and tissue types, but the quality of information supplied by vendors can vary tremendously. A common complaint from scientists is that companies do not provide the data required to evaluate a given antibody's specificity or its lot-to-lot variability. Companies might ship a batch of antibodies with characterization information derived from a previous batch. And the data are often derived under ideal conditions that do not reflect typical experiments. Antibody companies contacted for this article said that it is impossible to test their products across all experimental conditions, but they do provide reliable data and work with scientists to improve antibody quality and performance.

Many academics use Google to find products, so optimizing search results can sometimes matter more to a company than optimizing the actual reagents, says Tim Bernard, head of the biotechnology consultancy Pivotal Scientific in Upper Heyford, UK. Christi Bird, a Frost & Sullivan analyst based in Washington DC, says that researchers are often more interested in how quickly reagents can be delivered than in searching for antibodies with appropriate validation data. “It's the Amazon effect: they want it in two or three days, with free shipping.”

Researchers who are aware of the antibody problem say that scientists need to be more vigilant. “Antibodies are not magic reagents. You can't just throw them on your sample and expect the result you get is 100% reliable without putting some critical thinking into it,” says James Trimmer, head of NeuroMab at the University of California, Davis, which makes antibodies for neuroscience. Like many suppliers, NeuroMab explicitly states the types of experiment that an antibody should be used for, but scientists do not always follow the instructions.

Ideally, researchers would refuse to buy antibodies without extensive validation data or would perform the validation themselves (see 'Bad antibodies'). This is something that Rimm is passionate about: he has developed a multistep flowchart for effective validation6, which he shares with anyone who will listen. But the process is time consuming — Rimm recommends control experiments that involve engineering cell lines to both express and stop expressing the protein of interest, for example. Even he acknowledges that few labs will perform all the steps.

Some scientists buy half a dozen antibodies from different vendors, and then run a few assays to see which performs best. But they may end up buying the same antibody from different places. The largest vendors compete on catalogue size, so they often buy antibodies from smaller suppliers, relabel them and offer them for sale. Bernard says that the 2 million antibodies on the market probably represent 250,000–500,000 unique 'core' antibodies.

By necessity, many researchers rely on word of mouth or the published literature for advice. But that creates a self-perpetuating problem, in which better-performing antibodies that become available later are rarely used, says Fridtjof Lund-Johansen, a proteomics researcher at the University of Oslo. “We have very good antibodies on the market,” he says, “but we don't know what they are.” Lund-Johansen is trying to change that by developing high-throughput assays that could compare thousands of antibodies at once.

Testing times

In the past decade, various projects have sprung up to try to make information about antibodies easier to find. The online reagents portal Antibodypedia (antibodypedia.com), which is maintained by the Human Protein Atlas, has catalogued more than 1.8 million antibodies and rated the validation data available for various experimental techniques. Antibodies-online (antibodies-online.com), another portal, set up a programme two years ago for independent labs to do validation studies, generally at the vendors' expense. But out of 275 studies, less than half of the products tested have made the cut and earned an 'independent validation' badge. The non-profit Antibody Registry (antibodyregistry.org) assigns unique identifiers to antibodies and links them to other resources. Another project, pAbmAbs (pabmabs.com/wordpress), operates in a similar way to the social-recommendation web service Yelp, by encouraging people to review antibodies.

But none of these efforts has gained much of a foothold in the scientific community. Many of the scientists contacted for this article were unaware that such resources existed.

“Antibodies are not magic reagents.”

The antibody market has grown so crowded that a reputation for quality is becoming part of some suppliers' business plans. “Now there is so much competition that you have to differentiate yourself,” says Bernard. Vendors such as Abcam in Cambridge, UK, are encouraging users to report their own data and rankings on the company's website. Abcam's analysis of purchasing behaviour shows that its customers look at data pages on average nine times before buying, suggesting that customers want more information.

Abgent, an antibody company based in San Diego, California, and a subsidiary of WuXi AppTec in Shanghai, China, tested all of its antibodies about a year ago. After reviewing the results it discarded about one-third of its catalogue. Whether that was a good decision depends on whether customers will be willing to spend more for better reagents, says John Mountzouris, site leader at the company. Already, he says, customer complaints have plummeted.

Some scientists are calling for much more radical change. In a Comment in Nature in February7, Andrew Bradbury of Los Alamos National Laboratory in New Mexico and more than 100 co-signatories proposed a massive shift in the way antibodies are produced and sold. They suggested using only antibodies that have been defined down to the level of the DNA sequence that produces them, and then manufactured in engineered 'recombinant' cells. This would circumvent much of the variability introduced by production in animals. But the proposal demands information about individual antibodies that many companies consider to be trade secrets — and the antibody marketplace and its millions of products would have to be essentially demolished and reconstructed.

Uhlén, a co-signatory on the Comment, regards the plan as a distant hope. He estimates that the 'recombinant antibodies' that Bradbury hopes for would each cost 10–100 times more to generate than the conventional sort, and that they would not necessarily perform better. “At the end of the day, how the binder works in the application is more important,” he says. “Having a sequence for sure doesn't tell you if it works.” Other efforts are under way to find cheap, fast, reliable ways of making antibodies without immunizing animals, for example by expressing and optimizing them in viruses.

The pressure to characterize currently available antibodies is surging. As part of efforts to improve reproducibility, some researchers have started to discuss enlisting an independent body to establish a certification programme for commercial antibodies. And several journals (including Nature) ask authors to make clear that antibodies used in their papers have been profiled for that particular application.

The quality will creep, rather than leap, forward, says Trimmer, who hopes to see a positive-feedback loop: as scientists become aware of artefacts, they will be more likely to challenge results and uncover more artefacts. Already, he says, the widespread insouciance about antibody validation has started to fade. “It's turning around a little bit,” he says. “We need to keep talking about it.”

Journal name:
Nature
Volume:
521,
Pages:
274–276
Date published:
()
DOI:
doi:10.1038/521274a

References

  1. Begley, C. G. & Ellis, L. M. Nature 483, 531533 (2012).

  2. Prassas, I. & Diamandis, E. P. Clin. Chem. Lab. Med. 52, 765766 (2014).

  3. Michel, M. C., Wieland, T. & Tsujimoto, G. Naunyn-Schmiedeberg's Arch. Pharmacol. 379, 385388 (2009).

  4. Egelhofer, T. A. et al. Nature Struct. Mol. Biol. 18, 9193 (2011).

  5. Berglund, L. et al. Mol. Cell. Proteom. 7, 20192027 (2008).

  6. Bordeaux, J. et al. BioTechniques 48, 197209 (2010).

  7. Bradbury, A. & Plückthun, A. Nature 518, 2729 (2015).

Author information

Affiliations

  1. Monya Baker writes and edits for Nature in San Francisco, California.

Author details

For the best commenting experience, please login or register as a user and agree to our Community Guidelines. You will be re-directed back to this page where you will see comments updating in real-time and have the ability to recommend comments to other users.

Comments

Commenting is currently unavailable.

sign up to Nature briefing

What matters in science — and why — free in your inbox every weekday.

Sign up

Listen

new-pod-red

Nature Podcast

Our award-winning show features highlights from the week's edition of Nature, interviews with the people behind the science, and in-depth commentary and analysis from journalists around the world.