Health policy is hampered by uncertainty. During a novel outbreak, much is uncertain: the mode of transmission; the duration of latency, infection and immunity; and whether the outbreak will fade or turn into an epidemic. The uncertainty may be structural (which model is most appropriate?), parametric (what is the basic reproduction number, R0?) and/or operational (what is the potential efficacy of a vaccine?). This month, we discuss how uncertainty affects forecasts of disease dynamics1,2 and optimization of intervention strategies.

Some model uncertainties will matter more than others, depending on our purpose. If early results suggest an R0 between 1.5 and 4.0, we already have sufficient information to justify mobilizing an epidemic taskforce since R0 > 1. However, we need more biological information to forecast the outbreak magnitude.

This information will accrue over time, and estimates of fast rates will be the first to be assessed. The latent (1/σ) and infectious (1/γ) periods can be determined quickly from patients. In contrast, to estimate immunity duration might take years. A priority in studying the outbreak is to identify and reduce the uncertainties that have the largest impact on forecasts or decisions. We here explore how uncertainty affects the timing and magnitude of the first and second infection waves and the cumulative burden of disease, B (number of infection events from the start of the epidemic).

Sensitivity analysis highlights which uncertainties most strongly impact our projections of disease dynamics. Consider a novel outbreak that starts in early January. Let’s assume a precise estimate of 1/σ = 1/γ = 7 days and explore the sensitivity to uncertainty in R0 = 1.5–4.0 and an average immunity duration that can be either short (1/ω = 1 year) or long (1/ω = 2 years). Figure 1 shows the infection waves projected by an SEIRS2 model for these scenarios.

Fig. 1: The effect of R0, and immunity duration on disease burden and trajectory.
figure 1

a, SEIRS simulation of the infected fraction (filled curve) and normalized cumulative disease burden (B, line) for an outbreak with 1/σ = 1/γ = 7 days, short (1-year) immunity duration, and R0 = 1.5 (blue) or R0 = 4 (orange). Peak times are shown by colored circles on the time axis and the positions of the first and second peaks across the range R0 = 1.5–4 are traced with black lines. Gray bars indicate November–February peak flu season assuming t = 0 is 1 January. Cumulative burden is normalized by burden at t = 4 years for R0 = 4 (B = 3.59). b, Same as in a but for long (2-year) immunity duration. SEIRS simulations with 1/µ = 76-year life expectancy, over 4 years and initial values of susceptible S(0) = 0.999, exposed E(0) = 0, infected I(0) = 0.001 and recovered R(0) = 0 fractions.

Full size image

R0 has a strong effect on the position and height of the first peak: lowering R0 through mitigation decreases and delays the peak1,2. The first peak is largely unaffected by immunity duration because loss of immunity is a slower process than infection and recovery. Thus, parametric uncertainty about immunity does not hamper the forecast of the initial peak. The first peak is also unaffected by structural uncertainty: trajectories of the permanent immunity (SEIR) and waning immunity (SEIRS) models are initially very similar.

The timing of the second wave, however, is affected by both R0 and waning of immunity. Thus, uncertainty may be very consequential for long-term forecasts and public health outcomes, especially if, for example, coinfections and hospital capacity are major determinants of mortality and morbidity. For a severe outbreak (R0 = 4) with short immunity, the second wave is projected to occur in the middle of the next flu season (t = 0.95 years; Fig. 1a), whereas if immunity is long, it is projected to occur during the following summer (t = 1.51 years; Fig. 1b), when respiratory coinfections are of lesser concern. If the outbreak is less severe or R0 is reduced by social distancing (R0 = 1.5), the second wave occurs substantially later, and in summer, for both immunity durations (t = 2.34 and t = 3.47 years, respectively), by which time effective vaccines or treatments may have been developed.

Estimates of cumulative disease burden, B, are greatly influenced by uncertainty in both R0 and immunity duration. We will express all subsequent burdens normalized to the 4-year cumulative burden of the short immunity scenario at R0 = 4 (B = 3.59 = 100%, Fig. 1). For short immunity, reducing R0 from 4 to 1.5 lowers B to 42.8% (Fig. 1a). Similarly, increasing immunity duration from 1 to 2 years at R0 = 4 lowers B to 61.8%, and lowering R0 to 1.5 further lowers B to 28.8% (Fig. 1b).

The impact of uncertainty about immunity duration can be influenced by vaccination. Suppose we vaccinate a percentage p of the population each year. The effect of even a low p = 30% on the burden is stark: we can achieve the same 42.8% burden projected for short immunity and R0 = 1.5 at a higher R0 = 1.86 (Fig. 2a). In the case of long immunity, we can achieve 42.8% at R0 = 2.96. In fact, a combination of partially effective vaccination and low R0 may shut down transmission altogether. For long immunity and R0 = 1.5 this happens at p = 26%; there is only one peak and 99.9% of all cases occur in the first 2 years.

Fig. 2: Quantifying parametric uncertainty and decisions.
figure 2

a, Effect of R0, immunity duration and an annual vaccination percentage p = 0%, 30% or 45% on normalized cumulative 4-year disease burden, B, for an outbreak with 1/γ = 1/σ = 7 days. B wavers when recurrent epidemic waves cross the 4-year time horizon at which the burden is calculated. b, Impact on B of three mitigation actions A1, A2 and A3 under two models M1 and M2. c, The expected value of perfect information (EVPI) assesses the benefit of resolving uncertainty in immunity duration before deciding on an action.

Full size image

Just as we study the sensitivity of our forecasts to uncertainty, we can ask how uncertainty affects our management decisions. Suppose we can implement one of three mitigation actions: strong social distancing with no vaccination (action A1, R0 = 1.5, p = 0%), intermediate distancing with low vaccination (action A2, R0 = 2, p = 30%) and slight distancing with stronger vaccination (action A3, R0 = 2.5, p = 45%), but are faced with uncertainty about immunity duration, which can be either short (model M1, 1 year) or long (model M2, 2 years) (Fig. 2b). How should we go about choosing an action, without knowing which model is better? Under M1, choosing A1 yields the lowest B, 42.8%, but if M2 is correct, then A2 gives the best outcome, B = 21.9% (Fig. 2c).

The impact of uncertainty about immunity duration on our choice of strategy can be expressed using the expected value of perfect information (EVPI)3, which is the potential improvement in outcomes that could be obtained if the uncertainty is resolved before making a decision. In other words, by how much could we potentially lower B if we knew which model best reflects reality?

To calculate EVPI, we first consider the best outcome in the presence of uncertainty. Though we do not know whether model M1 or M2 is better, we can dismiss action A3 because its outcome is worst for both models. Focusing on A1 and A2, we first find the optimum average B over each action: for A1 we have avgi(B) = 35.8% and for A2 we have avgi(B) = 35.7%, and the optimum is the latter (Fig. 2c). Thus, if uncertainty cannot be resolved before making a decision, and assuming long and short duration immunity are equally likely, the best action is A2.

However, if we have perfect knowledge about immunity duration, we can choose a strategy that optimizes B for each model. For M1 we have optj(B) = 42.8% (A1 is better) and for M2 we have optj(B) = 21.9% (A2 is better), and the average of these is 32.4% (Fig. 2c). We have used an equal-weight average, but if information about the likelihood of different models is available, the average can be weighted accordingly. The EVPI is the difference between the best of the averages and the average of the best: 35.7 – 32.4 = 3.3%. This may seem small, but in the context of the US population it corresponds to about 10 million fewer cases over 4 years.

In prioritizing interventions, an EVPI of zero means there is no need to resolve uncertainty because one strategy is clearly better. Such is the case for A3, with EVPI(A1, A3) and EVPI(A2, A3) both being zero. A3 is not only never the best but it is always the worst — a ‘dominated alternative’. It is also possible that one action is never the best but nevertheless performs relatively well in most cases — a bet-hedging strategy.

The results of EVPI analyses may depend on the objective. Here, our objective is to minimize the cumulative burden at the 4-year horizon, so we favor actions that lower R0 and increase vaccine-induced immunity. If instead our aim is to shorten the outbreak duration, perhaps for economic reasons (as may arise for outbreaks in livestock), then strategies with less stringent mitigation could be favored. Unclear or conflicting objectives can lead to confusion about optimal interventions.

In some cases, once an intervention is selected, it cannot be changed or reversed. Either single models with well-defined variants3 or multiple models4 can be used to make robust one-off decisions in the face of uncertainty. However, in an ongoing outbreak, we usually have an opportunity to change our interventions as we gain information about disease dynamics and intervention efficacy. Adaptive management3 can use EVPI to actively prioritize learning about uncertainties that have the largest impact on decisions while disease management is ongoing. This is vital given the time-sensitive nature of epidemics.

An interactive tool for EVPI analysis is at https://martinkrz.github.io/posepi3.