Academic screening goes high-throughput

Creating effective screens requires time, skepticism and astute trade-offs.

Researchers at academic screening centers are creating assays for projects that large pharmaceutical companies are reluctant to touch. The impetus for the screening center at University of Dundee, Scotland, for example, came from scientists who discovered promising drug targets but found themselves unable to interest commercial partners in programs to treat parasitic infections in the developing world.

Whereas industrial researchers have to weigh the costs of pursuing a product against its potential market, academic researchers can apply their more limited resources to broader questions, such as neglected diseases and basic physiology, says Peter Hodder, who directs the lead identification program at the Scripps Research Institute. “I have been screening a greater variety of targets than I was screening in the industry, since the targets coming out of academia are not necessarily connected to a billion-dollar drug.”

Industry also shies away from assays that take a long time to optimize, says Thomas 'TC' Chung of the screening center at the Sanford-Burnham Medical Research Institute, who, like many academic screeners, spent years in industry. “In pharma, our job was to reject assays that didn't fit our format,” he says. “Now our job is to reformat assays and make them work.”

Nonetheless, academic screening centers have gained the notice of industry; pharmaceutical companies are increasingly open to partnering with them, says Lorenz Mayr, an executive director of Novartis Pharmaceuticals and board member of the Society for Biomolecular Screening. Indeed, industry-academia partnerships can be a vital source of revenue for centers, especially when grant funding is uncertain.

Partnerships are not only occurring with drug companies, says Michelle Palmer, who directs screening at the Broad Institute and is president of the Society for Lab Automation and Screening. Screening centers are also becoming more integrated academically, she says. At the Broad Institute, for example, several groups are interested in how data from screening can be used: members of the Cancer Atlas Project are discussing ways to profile known drugs against cancer cell lines; small-molecule screens are being combined with RNA interference screens; and systems biologists are exploring how compounds affect gene expression and protein content.

Peter Hodder of the Scripps Research Institute says researchers must plan beyond high-throughput screens to the work that comes before and after them. Credit: Scripps Florida

Screening spurt

A global growth spurt in academic screening began in the early 2000s, fueled by increased government support for translational medicine and by declining costs of laboratory robotics. In 2004, leaders at the US National Institutes of Health (NIH) spearheaded the idea of funding scientists to develop assays amenable for medium- and high-throughput screening. The plan included creating the NIH Chemical Genomics Center and funding other screening centers at academic institutions. Researchers at these centers would help develop assays, test them against a central library of hundreds of thousands of compounds and make results publicly available. In 2008, the NIH funded the second phase of its Molecular Libraries Program, designating approximately $70 million over four years to four comprehensive screening centers as well as smaller, more specialized centers. There is also plenty of academic effort outside the nine NIH-funded centers: a directory of academic screening centers maintained by the Society for Biomolecular Screening lists over six dozen in a dozen countries.

The hope is that such efforts will identify small molecules that can, once optimized for selectivity and potency, be used to probe protein activities and signaling pathways. These 'tool compounds' can then be used for target validation—that is, to supply proof of principle that modulating a cell or protein activity could treat disease. In some cases, tool compounds can be advanced into early drug development. “We're trying to improve the discovery paradigm itself, to make the search for new medicines happen on a scale that's not decades,” says James Inglese, deputy director of the NIH Chemical Genomics Center.

It's hard to track what impact academic screening centers are having. If researchers in industry use tool compounds discovered in academia to validate a drug target, such studies could take a long time to appear in the literature or might even be kept confidential. However, NIH program codirector for the Molecular Libraries Program, Carson Loomis, says that within his program alone there have been 304 publications on probes and related analogs, which have, collectively, been cited by another 845 publications. And although the NIH's screening programs had been set up to find tool compounds rather than drug leads, about 90 of the probes have advanced into animal testing and at least two should soon be filed as investigational new drugs, a necessary step before human trials.

Assaying closer to biology

The development of new assays by academic screening centers could be just as important as the identification of new compounds. The classic high-throughput screen is an exercise in reductionism: isolated proteins are exposed to hundreds of thousands of compounds and assayed for binding or inhibition. Because this is far removed from living systems, cell-based assays have become more popular in both industry and academia. Experts interviewed for this feature estimated that half or more of the high-throughput assays in academic screening centers and industry are conducted on cells, and some are even more complicated (Box 1).

Tom Chung of the Sanford-Burnham Medical Research Institute says running phenotypic assays is a long-term relationship. Credit: Conrad Prebys for Chemical Genomics at Sanford-Burnham Medical Research Institute

Whereas biochemical assays usually focus on only one protein at a time, cell-based assays offer multiple ways to study a signaling pathway. “You can explore the pathway and target a different position within that cascade to see where you get a better signal-to-noise ratio,” explains Jose R. Perez, who manages assay development at the Broad Institute Comprehensive Screening Center. For example, an assay to look for compounds that are active against a G protein–coupled receptor (GPCR) can identify molecules that simply bind to the receptor, that cause the recruitment of other proteins to the GPCR complex, that elicit a secondary messenger response in the form of cAMP or calcium concentration, or that cause changes in gene expression.

Cell-based assays frequently rely on cells engineered to contain reporter genes whose transcription can be readily detected. Popular reporters include fluorescent proteins, as well as the firefly enzyme luciferase and the bacterial enzyme beta-lactamase, whose respective activities generate bioluminescent and fluorescent signals.

But such assays are becoming less common, says Perez. “Reporter assays are often fraught with artifacts.” Specially created reporter cell lines are often unstable, so the reporter construct is lost as cells are grown over time. More misleading, some compounds that are interpreted as affecting gene transcription might instead modulate the reporter protein; researchers at the NIH Chemical Genomics Center recently described how several compounds identified as transcriptional activators instead affect the activity of luciferase¹. Even when a reporter construct signals as intended, it is difficult to recapitulate all the genetic regulation of the gene whose activity is to be monitored. Many scientists are turning to techniques such as quantitative PCR that monitor endogenous gene expression. Such techniques are less convenient than those relying on reporters, says Perez, but more relevant.

James Inglese of the NIH Chemical Genomics Center sees assay development as scientific research in its own right. Credit: Maggie Bartlett, NIH–National Human Genome Research Institute

In contrast, perhaps looking at many cell markers simultaneously could be another way to find compounds with a desired effect, says Steven Altschuler at The University of Texas Southwestern Medical School. Using high-content screening, a microscopy-based technique that is used to assess many features of a cell, he and his colleagues designed computer algorithms that could analyze images of cells monitored for a fairly generic set of cell-signaling markers and reliably predict which cells had been exposed to certain drug classes². However, screeners need to be careful to pick an appropriate readout, he says. “You don't want to do an assay that's looking for something just above the noise level, because then you'll spend all your time trying to separate signal from noise.”

Always optimize

The chief challenges for cell-based assays are maximizing signal over background (a property known as the signal window) and a related parameter of robustness, known as the Z-factor, which incorporates means and errors associated with positive and negative controls. Finding the best conditions generally requires multifactor analysis of key variables. In particular, researchers must find the time point when a signal is strongest but before it plateaus; measuring signals after they saturate can provide misleading data on how effective different compounds are.

Robots are used for high-throughput screening at the Conrad Prebys Center for Chemical Genomics at Sanford-Burnham Medical Research Institute. Credit: Conrad Prebys Center for Chemical Genomics at Sanford-Burnham Medical Research Institute

Getting the best signal and least noise every day for months on end is difficult because cells are sensitive to many environmental variables, says Julie Frearson, who developed screening assays as a biotechnology professor at the University of Dundee. Microscopy experiments, which require fewer cells, are often the best method for using unmodified cells, but such experiments demand even greater attention to reducing variability. “You are looking at much smaller windows of readout, so you have less room for error, but since these are living cells, you get more error,” she says. “You have to be religious about your cell culture routines.” It is more than establishing a consistently reliable cell bank and finding the best way to seed cells into wells and add compounds; practices such as how plates are stacked in an incubator matter as well. “Things that you would think are trivial are far from trivial,” she says.

Even under optimal conditions, an assay will not be uniform. Noise is part of automation, says Paul Johnston of the University of Pittsburgh Drug Discovery Institute. “You have to understand the statistical variability of that assay before you start running it.” That means running plates many times with many controls under different conditions as well as running plates with validation libraries containing pharmacologically active compounds; data from such runs can indicate what the hit rate of an assay will be, identify potential sources of interference and alert researchers when something goes wrong.

Assays must also be optimized to reduce the time and number of steps as much as possible and to keep the number of parameters measured manageable (see, for example, http://www.ncgc.nih.gov/guidance/hts_assay_guidance_criteria.html). Screens that follow cell response over time are often converted to fixed endpoints, for example. Or a screen hunting for molecules that prompt stem-cell differentiation might look for earlier indicators of cell fate. Cost per well must also be considered. Excluding employees and equipment, most cell-based assays should cost around five or ten cents per well, though that might go up to as much as fifty cents for certain high-content assays that use several antibodies or fluorescent dyes at once, says Sanford-Burnham's Chung. Assay developers may try to simplify a screen relying on several antibodies to the single most-informative one or find ways to replace expensive, specialty antibodies with more commonly used, less expensive ones. Replacing an antibody-generated signal with one from a fluorescent reporter could reduce costs.

The right look at the right cells

A mixed benefit of cell-based assays is that they can be used to find compounds that promote altered cell phenotypes without knowing which proteins are involved. Some researchers look for compounds that modulate a particular signaling pathway. Others study phenomena such as cell proliferation and growth.

“Phenotypic assays are often easy to run but frustrating to make sense of,” says John Reed, principal investigator of the screening center at the Sanford-Burnham Medical Research Institute. In a program looking for compounds that modulate responses to endoplasmic reticulum stress, for example, researchers pursued over 30 secondary screens and counterscreens to weed out toxic compounds and pin down other compounds to particular pathways. Without this kind of validation, identified compounds could actually hinder rather than help scientific understanding when used as research tools; pursuit into drug development could waste millions of dollars and expose human subjects to unnecessary risks (Box 2). But extensive follow-up screening is often more than researchers bargained for, Reed says. “If you're going to go with cell-based assays, you need to have the commitment to deconvolute the pathways.”

The right assays are also important for development of identified molecules. Even though programs screen hundreds of thousands of molecules, identified molecules are only starting points. Chemists then tweak molecules' structure to boost desired properties and eliminate undesirable ones, and additional assays are necessary to keep them from working in the dark. “That's something that academic researchers aren't always thinking about,” says Palmer. “If the chemists are going to work with you, you need to have the appropriate assays to inform their chemistry decisions.”

And there is no guarantee that screened cells represent the biological events relevant for physiology or disease biology. The cells most frequently used for screening, HeLa cells, CHO cells and HEK-293 cells, are selected because they grow well and are easily transfected with reporter genes. Scientists would prefer cells from particular tissues, but these vary in quality and rarely grow well in plates.

Many researchers are hopeful that induced pluripotent stem cells could at last provide access to more-relevant cell types. But although some screening centers have begun pilot screens with cells differentiated from induced pluripotent stem cells, their use has yet to become widespread. Ian Wilmut, who directs the Centre for Regenerative Medicine at the University of Edinburgh, cautions that even once cells have been differentiated into cell types affected by disease, representing that disease in a dish could be difficult; modeling that disease in such a way that it is amenable to high-throughput screening will be more difficult still. The neurodegeneration he hopes to study occurs over decades in complex tissue, Wilmut says. There is no guarantee that relevant differences between cells from diseased and healthy patients will appear in a Petri dish over a matter of weeks, let alone that the difference will be something, such as the amount or localization of protein, which translates readily into a high-throughput assay.

For these kinds of reasons, many researchers feel that the future of academic screening lies in increasing collaborations with scientists who have developed deep expertise in specific areas of physiology or disease. Though researchers in several laboratories are working out ways to process more cells more quickly—such as using flow cytometry, microfluidics, cell chips or microarrays—miniaturization just to save time and reagents is becoming less important, particularly because screening cells in 1,536-well plates is now common, says Novartis' Mayr.

Researchers in both industry and academia are talking less about increasing throughput and more about representing biology. “There was a big push to move reporter assays into 1,536-well plates because they were convenient for automation,” says Perez. “I'm not convinced that a lot came of that. Everyone loves a simple assay, but if the simplicity comes at the cost of relevance, it's not worth it.”

Table 1 Suppliers guide: Companies offering screening products and services

Box 1: Whole-organism screening: when cells are not enough

More difficult but often more physiologically relevant than cell-based screening is screening whole organisms. The prominence and numbers of publications based on these types of experiments have been increasing. Just this year, three teams led by researchers at GlaxoSmithKline³, National Institute of Allergy and Infectious Diseases⁴ and St. Jude Children's Research Hospital⁵ published results of screens against the malaria parasite. In another instance, two other teams placed zebrafish larvae in 96-well plates, then looked to see how compounds affected the larval rest-wake patterns or responses to light^6,7.

A microscopy-based screen detects the number of fluorescent parasites in unmodified human macrophages. Credit: M. DeRycker, Drug Discovery Unit, University of Dundee

Julie Frearson, who oversaw assay development at the University of Dundee in Scotland, says academic researchers have more freedom to develop unique assays because they can focus first on biology rather than automation. She and colleagues wanted to develop a technique that could identify small molecules that prevented the parasite that causes leishmaniasis from growing in infected macrophages. One possibility was to develop a label-free screen to detect infection. Computer-assisted object identification allowed researchers to count the number of parasites per macrophage, but the assay was too variable to be practical for primary screening. Another option was to modify the macrophages and parasites to express fluorescent proteins and rely on microscopy, but the research team wanted to avoid genetically engineering the human host cells. After working out the correct washing protocols, the team found a way to run screens in 384-well plates using unmodified human cells with a fluorescent but still infectious parasite. The workflow is more complicated than what might be put forward in industrial, highly automated settings, says Frearson. “We focus on the assays and use whatever automation is necessary to make the process work.”

Box 2: Artifact alert

Screening centers often track their progress by the number of compounds that they identify as active, but researchers have to be careful that the results reflect biologically relevant events, warns James Inglese, deputy director of the NIH Chemical Genomics Center. “The assay's the goose that lays the golden egg,” he says. “The compound is the golden egg, but geese produce stuff besides golden eggs and it's annoying to step through all that.”

Researchers already run some wells without cells to screen out compounds that are naturally colored or fluorescent and so could give a positive signal by their presence alone. Other artifacts are less intuitive⁸. Even readily detectable signals from cell-based screens may not be as straightforward as assay developers would wish. Cell populations even in a single well can show heterogeneous responses to different drugs. Thus the signal from a well might be the average of two distinct populations and so misrepresent them both.

Even if cell populations are homogenous, screening hundreds of thousands of compounds means hundreds of thousands of chances for researchers to find compounds that fool them. “Your false positives are going to outnumber your real hits,” says Jonathan Baell, a medicinal chemist at The Walter and Eliza Hall Institute of Medical Research, who believes that much of the screening literature is polluted with nonspecific artifacts—molecules he has dubbed pan-assay interference compounds or PAINS⁹. Whereas companies often have a sort of tribal knowledge about identifying true binders in their own libraries, more easily deceived inexperienced researchers are enjoying increasingly easy access to high-throughput screening technologies; they are both eager to publish and may lack resources for rigorous follow-up. Researchers who take the publications at face value could design faulty experiments around these molecules, he says.

But the urge to publish itself may right such problems if academics deposit results in public databases, says Larry Sklar. PubChem provides data for hundreds of thousands of compounds in thousands of assays, and savvy researchers are wary of molecules that show up frequently. “If the database didn't exist, you might be tempted to use different criteria to find molecules that go into your screens,” Sklar says.