Introduction

The COVID-19 pandemic has created and continues to create a significant number of challenges to the global economy and its associated healthcare systems. Initial projections modelled by Ferguson et al. on 16 March 2020 predicted that, in the absence of any control measures, the UK could expect a peak of 510,000 deaths by the end of May 2020 directly attributed to the virus, with 15,000 deaths per day at its peak. Even with case isolation, home quarantine and social distancing, this would require 90,000 critical care beds by mid-June.1

It is important to note that the paper by Ferguson et al. was based on multiple predictor variables, but did not indicate any confidence intervals around these figures to describe the degree of uncertainty in the prediction.2,3 On the same day this report was published, the UK government placed the country into lockdown, severely restricting the population's movements. A week later, the Chief Dental Officer (CDO) for England published a letter requiring that all dental practices stop all non-urgent care for their patients and await the provision of specific regional urgent dental care centres (UDCs) to manage any acute treatment.4 Based on Ferguson et al.'s predictions, this action made complete sense. Dental staff were to be redeployed into the secondary care system and Nightingale hospitals were constructed to care for the huge number of patients projected to flood the healthcare system.5

The reality was that deaths peaked on 8 April at 889 fatalities, rather than 15,000 towards mid-May as Ferguson had predicted.6 On 23 April, the UK government told us in its daily briefing not to worry as they were being 'guided by the science', even though the number of daily COVID-19-related fatalities had dropped by 38% from its peak.7 Looking back at these and other planning and policy measures implemented over the lockdown period, what can we learn to improve future planning and response?

Assumptive normality

Much of the initial modelling and policy development assumed the virus and its impact on the UK would be normally distributed across the population - the classic 'bell curve distribution'. This interpretation of the data is called 'assumptive normality' and can lead to large errors in analysis if the data are in fact not normally distributed.8 An example of this would be the assumption that, since SARS-CoV-2 was a new virus, there would be no immunity to infection within the entire population and this would result in the high death rates predicted. Cromwell's rule states that absolutist assumptions (that an event will definitely occur or not occur) should be avoided because if they are not logically true it will force a rejection of real-world evidence, no matter how strong that evidence may be.9 Using Ferguson et al.'s models, the government applied and maintained national lockdown measures that affected the entire population. As evidence accumulated, it became obvious that older patients, those with severe co-morbidities and those with lower socioeconomic status were more likely to die from the consequences of COVID-19, and this closely tracked normal age/mortality data (Fig. 1) rather than affecting all age groups equally.10,11,12,13,14 From analysis of this data, one could conclude that the risk of fatal infection was not evenly distributed within the population (Fig. 2).

Fig. 1
figure 1

Deaths by age group (COVID-19 vs all causes 2019)

Full size image
Fig. 2
figure 2

Normal population age distribution compared to COVID-19 deaths

Full size image

This real-world evidence could also be seen in the analysis of the excess deaths registered by English and Welsh local authorities as related to COVID-19.15 When these data were charted (Fig. 3, Table 1), we could see that it was not normally distributed, but indicated a positively skewed distribution for both total excess deaths above the five-year average and non-COVID-related deaths.

Fig. 3
figure 3

Distribution of excess deaths by local authority (England and Wales, 29 February-8 August 2020)

Full size image
Table 1 Descriptive data for excess deaths by local authority (England and Wales, 29 February-8 August 2020)
Full size table

Problems occur when decisions are made based on an assumption that data are normally distributed, when in fact, they have a strong positive skew. This means the use of averages in the form of means and standard deviations can't apply.16 In terms of the population, a specific group of patients were very seriously affected, as were a small number of local authorities. One consequence of the national lockdown and restriction to healthcare provision was the high number of non-COVID-related excess deaths (n = 12,116) between 29 February and 8 August, which accounted for 20% of total excess deaths above the five-year average. These non-COVID deaths have been attributed to restrictions in access to both scheduled and emergency healthcare.17

Science alone is not the path to truth

We must not confuse 'the science' with evidence. Science is a process of objectively investigating a phenomenon and attempting to produce a consistent explanation for the observation; science is not a direct pathway to the truth.18 Evidence is defined by the Oxford English Dictionary as the 'available body of information indicating whether an opinion or proposition is true or valid'. A more appropriate attribute of science would be that: 'the scientist accepts or retains a hypothesis if and only if not only is it true or appropriately truthlike but also is acquainted with evidence that justifies belief in its being true or appropriately truthlike'.19

The science (modelling) behind the initial prediction for the pandemic by Ferguson et al. failed to match the evidence 24 days post-publication, highlighting some of the weaknesses in modelling complex interactions in the real world, especially where information is limited and there are exponential interactions.20 This is where Brooks stated: 'It is better to use the world as its own model' when he was developing artificial intelligence in autonomous mobile robots.21 This was based on the idea that models mirror the conceptual world view of the modeller. The conceptual world of the mathematical modeller will be very different from the policymaker, as it will be between the clinician working in an intensive care ward compared to a rural general practice. Each will place different weight on what they feel is important, be that in the form of a conscious or unconscious bias. These differences in individuals' world views are powerful and can lead to over-simplification, and can increase the effect of error when it comes to model development by normally acknowledging the potential for research bias, but neglecting the effects of statistical noise and variability in real-world interactions. This is described in the following equation:22

Total error = biasn x variance x noise.

'Bias' is the systematic error(s) introduced into a model by consciously or unconsciously selecting or encouraging one or several outcomes over alternatives. 'Variance' defines the spread of data around the mean or median value, which can become highly volatile in skewed distributions with long tails, such as data relating to the distribution of coronavirus within the population. Finally, 'noise' is the random irregularities in real-life data; for example, clinical staff not working like robots (that is, leaving a door open or shortcutting a procedural step through exhaustion).

Returning to Brooks's paper, the decomposition and abstraction involved in the computation of a predictive model may completely ignore the challenges posed by variance and noise imposed by the real world. Glazsiou et al. described the challenges of separating 'signal from noise' with regards to clinical treatment effects. For example, when a medical or surgical therapy produces a statistically significant beneficial effect in a research setting, but when introduced into the real world, this benefit is lost in the noise of interactions and confounding.23 The effects of noise can be demonstrated when we look at some of the coronavirus mitigation requirements.

Airborne infection and fallow time

By over-relying on the science associated with the modelling data, much of the healthcare sector has remained paralysed despite the reduction in hospitalisation and fatality rates from mid-April.24,25 In dentistry, there was very little evidence of any cross-infection problems compared to mainstream healthcare (hospitals, GP practices and care homes) during the peak of the pandemic in the UK/Europe/US/Asia. From searching the databases, there has been no evidence of a 'superspreader' event or case events involving dental practices. Where the dental team became infected, this was traced back to either conferences or more generally infection from within the clinician's social group.26,27,28,29 The main reason for the lack of cross infection is most likely due to the high level and standard use of personal protective equipment (PPE) in the dental practice. This has been in place since the mid-1980s to protect staff and patients predominantly from blood-borne viruses such as human immunodeficiency virus (HIV), hepatitis B and C, and respiratory pathogens such as tuberculosis.30 Cross-infection protocols were consolidated further following the publication and implementation of the Decontamination in primary care dental practices document.31 These high-level cross-infection policies before COVID-19 were already well integrated into practice protocols, audit and culture. It should be noted that, once a system has stable protocols and is producing effective cross-infection control, we encounter the problem of increasing difficulty in improving that system without inflicting costly trade-offs in terms of time spent with patients and expense. The hypothetical risk of airborne virus transmission from dental aerosol generating procedures (AGPs) was based on models created in the absence of a valid evidence base of harms (the denominator must be greater than zero) specific to the dental environment.32 There were five basic assumptions operating within the recommendations:

  1. 1.

    That the independent action hypothesis was true, and that a single pathogen has a non-zero probability that it could infect the host organism and result directly in the organism's death33,34,35

  2. 2.

    It was possible to remove 99% of the airborne pathogens from a functioning treatment room

  3. 3.

    All patients were asymptomatic/pre-symptomatic spreaders of coronavirus

  4. 4.

    All dental AGPs were contaminated with coronavirus

  5. 5.

    No current procedures regarding cross-infection control were sufficiently effective.

As mentioned earlier, these absolutist parameters invoke Cromwell's rule which leads to the rejection of real-world evidence. One fundamental premise of introducing a new set of standard operating procedures (SOPs) or processes is that the benefit gained has to be significantly greater than previously achievable in routine practice or else it should be considered ineffective or, at worst, harmful.

The assumption that the virus was 'novel' has led to the hypothesis that the protection offered by existing SOPs was insufficient and needed to be reinvented based on a 'precautionary approach' in the absence of evidence to the contrary. The lack of evidence of a dental superspreader event should be evidence that current practice was working. In general, the policymakers' focus appears to have only been on reducing the risk of coronavirus infection without considering fully any other potential harms that may be created.25 Instituting an untested protocol where there is no evidence of benefit has placed considerable stress on the system in the form of lack of direct training, supply chain problems, reduced patient access to care facilities and day-to-day functionality. Fallow time is a classic example of one of the new protocols that lacked robust direct evidence of benefit.36

Fallow time and its implementation in the real world

Fallow time is based on the hypothesis that removal of 99% of the droplets and aerosol from the clinical area over a time period of up to one hour will reduce the risk of airborne virus transmission.37 The reduction in concentration of that aerosol follows an exponential decay curve, but it must be remembered that the decay rate will vary with particle size (heavier droplets settling faster than smaller particles).38 Much of this evidence and the modelling for fallow time has been published by the Short Life Working Group for National Services Scotland, but fails to account for the variability and noise created by its implementation into the real world.32 One universal feature of an exponential curve is that it initially follows a roughly linear relationship for most of its path before the curve becomes tighter. This point of change is called the inflection point and roughly indicates the point of diminishing returns, where additional resource expenditure becomes less effective/efficient. Figure 4 shows the two options for fallow time advised by the CDO at that time - 60 minutes for a room with six air changes/hour and 20 minutes with 10-12 air changes/hour.39

Fig. 4
figure 4

Fallow time decay chart

Full size image

The inflection points are at 32 minutes for the 60-minute fallow time and at six minutes for the 20-minute fallow time, hypothetical aerosol reduction being 92% and 77%, respectively. These reductions in fallow time duplicate closely the real-world findings from recent 'in-practice' preprint studies, which ranged from 3-30 minutes to return to baseline depending on the duration of the dental AGP.40,41,42

A simple sample size/power calculation (G*Power) illustrates the challenges of the blanket introduction of untested protocols - for example, if we wish to run a study to prove 99% reduction in airborne droplets/particles is significantly better than 90% reduction. To do this, a tightly controlled laboratory study would require two groups of 79 samples (α = 0.05 and power is 0.8), which is a large study to organise in a laboratory. If this study was repeated in a real-world clinical setting - with the assumption that the research team have filtered out symptomatic patients (approximately 2/3) for coronavirus - we would need 75,840 treatment sessions based on current Office for National Statistics coronavirus infection data (1:160 as of 16 October 2020).43 This is a clear example of 'signal' being drowned out by the 'noise' of 'healthy people', and from a practical standpoint, such a study would be impossible to organise in a timely manner. One method to solve this problem in the future would be to use a 'stepped wedge randomised controlled trial' where alternate care models can be gradually introduced and tested without disrupting the whole infrastructure. Should the new protocol prove successful, it could be rapidly expanded from a low-risk clinical environment to a higher-risk one, or rapidly abandoned if harms prove unacceptable.44

In summary, mathematical modelling can be useful if we appreciate its limitations, but we return to Brooks's original quote again: 'It is better to use the world as its own model' and turn to the evidence.

How do we move forward when confronted with asymmetric risk in future crises?

From the examples described above, it is important that decision-makers at both a global, national and local level are able to understand the limitations of science and modelling in the absence of reliable data or evidence. In emergencies such as the COVID-19 pandemic, we have seen that data accumulates rapidly but needs to be carefully collected, cleaned and curated, and the science then needs to be supported by evidence.25 Initially, policymakers will have to draw on emergency plans based on past events, but they need to be prepared to rapidly modify those plans as the data start to diverge from that model. A clear example of this are the differences between the normally distributed models created by Ferguson's team to guide government policy over the years and the considerably smaller asymmetric curves created by the real-life events.1,45,46

A greater level of attention needs to be applied to real-time, real-world analysis as many people's lives and livelihoods cannot wait while leadership teams consolidate decision-making, authority and control of information, and then provide it on a strictly need-to-know basis. This can only be achieved by relinquishing the belief that a top-down response will engender stability, and replace it with a network of teams from both within and outside the normal operational environment to feed back intelligence, reducing the risk of groupthink.47 In addition, policymakers need to provide open and effective communication channels with their workforce and the public via frequent updates, and avoid suspending announcements for long stretches while they wait for more facts to emerge and decisions to be made.48 If a new system of emergency healthcare provision is to be introduced, it would be far better to run it in parallel to the current system initially (unless the system is failing). Shutting down a functional healthcare system before a new system is fully resourced and operational can leave large gaps in healthcare provision, which are then difficult to re-establish once the initial emergency has passed.

Conclusion

In conclusion, if we are to manage extreme emergencies such as future pandemics, we need more open channels of communication and understanding. If leadership is to be successful, it needs to both listen to and understand the limitations of emerging science and modelling, but also effectively appraise the evidence as it develops on the ground. In the case of COVID-19, many of the initial planning assumptions and models are now invalid in light of improving data, and the early benefits designed to protect the population need to be rapidly reassessed to prevent them harming the healthcare system and economy over the medium-to-long term.