Late-stage attrition and rising costs have been a key feature of pharmaceutical research and development (R&D) in recent years (R&D productivity: on the comeback trail.Nature Rev. Drug Discov. 13, 331–332 (2014))1. One solution — building more opportunities to terminate unmarketable molecules earlier in the development process — has been known for some time2,3,4,5. However, implementing such 'quick-kill' strategies requires rigorous decision-making.

In particular, decisions to terminate projects can be especially difficult. Project terminations are often viewed as losses, and project teams tend to be loss-averse6. Project teams may commit a sunk-costs fallacy, arguing that a project is close to completion and that terminating the project would mean the loss of investment that is in fact already lost6,7. Most importantly, the marginal costs that are associated with a decision to continue development may seem trivial compared with the potential losses that are associated with a decision to terminate a potentially marketable molecule. This, coupled with the organizational and market incentives to load the pipeline with 'promising' candidates, may make R&D teams and management reluctant to terminate scientific projects6,8,9. In pharmaceutical companies, this creates a bias in favour of continued development — the progression-seeking bias — that allows projects to advance even in the light of strong evidence to support their termination.

Such progression-seeking behaviour ignores the late-stage attrition costs and opportunity costs that are associated with false positives. False positives incur further R&D costs until their eventual termination later in the development process — the direct costs associated with late-stage attrition2,10. In addition, whereas the opportunity costs of false negatives are prominent in every executive's mind, the opportunity costs of false positives may be less obvious. As R&D resources are finite, by choosing to develop a product that is unmarketable we deny potentially marketable products the chance to be evaluated. False positives thereby incur opportunity costs, representing lost revenues to the organization and denying patients access to novel treatments.

Here, we show how reframing false positives in terms of opportunity costs could help project teams to view termination decisions as prospective gains. Reframing decisions in this way makes termination decisions easier6,8,11,12.

Calculating opportunity costs

In the following treatment, for simplicity, the 'currency' used to express costs is the number of molecules that are denied access to the market. However, the model can be readily extended to include lost revenues that are calculated to be net of development costs.

Let p be the probability that the product is marketable; that is, it has the therapeutic effect of interest and has an acceptable safety profile. Thus, p is an inherent property of the molecule.

At any particular decision point in the development process, let FP be the probability of advancing an unmarketable product (a false positive), and let FN be the probability of failing to advance a marketable product (a false negative). Note that the direct costs of progressing products to the next decision point are the same for true positives as for false positives.

The total cost of false negatives is proportional to the probability that we fail to advance a marketable molecule. This is simply given by pFN. Note that as p increases, we have more opportunities to make false-negative decisions, resulting in higher opportunity costs.

The total cost of false positives is proportional to the probability that an unmarketable product is progressed for development. This is given by (1 − p)FP. As p increases, we have fewer opportunities to make false-positive decisions, resulting in lower opportunity costs.

Assuming we have limited R&D resources, these false positives deny us the opportunity to evaluate a further proportion, (1 − p)FP, of the candidates. And of these, a proportion equal to p are marketable, of which (1 − FN) are expected to be progressed to the following stage.

This means that the probability that a marketable product is denied access to the next development stage by a false-positive decision is given by:

Using this simple framework, we can now explore the opportunity costs of false positives and false negatives.

Worked example

Let the probability that a molecule is marketable, and the probabilities of false positives and false negatives be 10%, 50%, and 10% respectively, corresponding to p = 0.10, FP = 0.50, and FN = 0.10. The values for p and FP approximate those from industry data for Phase I clinical development4,13. Although FN is largely unknown, a range of values for FN can be explored using sensitivity analysis.

Imagine we screened 1,000 molecules. As p = 0.10, 100 of those molecules are potentially marketable. False negatives deny 1,000 × pFN = 1,000 × 0.10 × 0.10 = 10 molecules access to the later stages of development and access to the market.

False positives deny 1,000 × (1 − p)FP × p(1 − FN) = 1,000 × (1 − 0.10) × 0.50 × 0.10 × (1 − 0.10) = 40.5 molecules access to the later stages of development. Assuming the FN rates for the subsequent stages — Phase II, Phase III and submission to launch — are each 0.10, a proportion (1 − FN)3 = (1 − 0.10)3 are denied access to the market. Thus, of our 40.5 marketable molecules denied access to the later stages, 40.5 × (1 − 0.10)3 = 29.5 molecules are denied access to the market.

At this stage of development, false positives incur much higher opportunity costs than do false negatives — almost 30 molecules are lost to false positives, compared with just 10 that are lost to false negatives. It makes sense to employ a ruthless, quick-kill strategy that is directed at early termination of marginal projects5. By purging the pipeline of failing projects, we are likely to free up development capacity, creating further opportunities to discover and develop novel medicines.

It is instructive to examine how the balance of opportunity costs, for false positives and for false negatives, shifts for a range of probabilities p, FP and FN. Figure 1 shows opportunity costs expressed as the proportion of molecules that are denied access to the next phase of development, as a function of a range of probabilities, p, for a development phase at which the notional false-negative rate, FN, is 0.20, and for false-positive rates that approximate early-, mid-, and late-stage development of FP = 0.67, FP = 0.50, and FP = 0.33, respectively4,13.

Figure 1: Opportunity costs, measured as the proportion of lost molecules that arise from false positives and false negatives as a function of the probability that a product is marketable.
figure 1

Let p be the probability that the product is marketable, FP be the probability of advancing an unmarketable product (a false positive) and FN be the probability of failing to advance a marketable product (a false negative). The grey, dashed diagonal line indicates the cost of false negatives where FN = 0.20. Only when the FP curves intersect with this line does the number of molecules that are denied access to the next stage by false negatives begin to exceed that of false positives. Up until this point, pressing for molecules to be progressed in the hope that they will be successful may reduce research and development productivity. Once the cost of false negatives is greater than the cost of false positives, it makes sense to switch from a quick-kill strategy to a strategy of rational optimism. As the rate of false positives (FP) falls, the critical probability at which it makes sense to switch to a rational optimism strategy is higher. For example, if FP = 0.67, the number of molecules that are denied access to the next phase by false positives does not exceed that of false negatives until p is 0.63 (a). However, if FP = 0.33, it makes sense to switch to rational optimism strategy at around p = 0.24 (b). In early development, when the false positive rates are likely to be high, a quick-kill strategy is favoured unless the probability, p, is high. In later development, when FP is lower and the probability that a molecule is marketable is higher, the cost of false positives is likely to be lower than the cost of false positives, favouring a strategy of rational optimism.

PowerPoint slide

Full size image

Note that as p increases, there comes a point at which the cost of false negatives exceeds the cost of false positives, favouring a strategy of rational optimism. Note also that the trade-off point in Fig. 1 shifts as the probability of a false positive, FP, increases. Of course, the exact point at which it makes sense to switch from a quick-kill strategy to a strategy of rational optimism will critically depend on the probabilities p, FP and FN. These probabilities are likely to differ for different companies and different therapeutic areas and we provide an Excel Workbook (Supplementary information S1 (table)) to allow the interested reader to explore other choices for p, FP and FN.

Concluding remarks

For a range of values for p, and for credible values for false-positive and false-negative errors, the opportunity costs that are associated with false positives are often greater than those associated with false negatives — especially during preclinical stages and early phases of clinical development. Even using fairly optimistic values for these early phases, a ruthless quick-kill strategy is likely to outperform a strategy that is 'planning for success'. Of course, at later stages, when the probability of success is likely to be higher, it is appropriate to switch to a strategy of rational optimism. However, this is not usually the case until late in clinical development — typically the start of confirmatory trials. In addition, front-loading the R&D process with more-specific assays, thereby reducing the numbers of unmarketable molecules that enter the development process, makes perfect business sense. Indeed, such assays may give an organization substantial commercial advantage, even if they are not required by regulators and do not form part of a submission.

The model can be extended further to incorporate stage-specific estimates of p, FP and FN. Industry data are available that permit the approximation of stage-specific p and FP rates (Supplementary information S1 (table)). Although there have been serious attempts to estimate the false-negative rates of individual assays, the FN rates for specific stages are largely unknown; however, this does not stop us exploring the robustness of the model to variation in FN. In addition, values of FN at specific stages can be estimated by drilling down to the specificity of the component assays. However, the main value of the model stems from making the real cost of false positives explicit. Reframing false positives as prospective gains makes termination decisions easier11.

In reality, there are yet further costs that are associated with false positives. As well as the additional development costs and opportunity costs associated with late-stage attrition, there are substantial market-revaluation costs associated with false positives. There may be a short-term effect, observed as a fall in share prices, to reflect expected delays in return on investment. In addition, decisions to terminate molecules late in the development process can lead to a loss in confidence in an organization's R&D processes and capabilities, leading to further market revaluations and share-price adjustments. Whereas larger companies may recover from such set-backs, such market revaluations are often catastrophic for smaller companies.