To the Editor — The global demand for prophylactic and treatment options for COVID-19 has in turn created a demand for both randomized clinical trials, and the synthesis of those trials into meta-analyses by systematic review. This process has been fraught, and has demonstrated the inherent risks in current approaches and accepted standards of quantitative evidence synthesis when dealing with high volumes of recent, often unpublished trial data of variable quality.
Research into the use of ivermectin (a drug that has an established safety and efficacy record in many parasitic diseases) for the treatment and/or prophylaxis of COVID-19 has illustrated this problem well. Recently, we described flaws in one randomized control trial of ivermectin1, the results of which represented more than 10% of the overall effect in at least two major meta-analyses2,3. We described several irregularities in the data that could not be consistent with them being experimentally derived4. That study has now been withdrawn by the preprint server5 on which it was hosted. We also raised concerns about unexpected stratification across baseline variables in another randomized controlled trial for ivermectin6, which were highly suggestive of randomization failure. We have requested data from the authors but, as of 6 September 2021, have not yet received a response. This second ivermectin study has now been published6, and there is still no response from the authors in a request for data.
The authors of one recently published meta-analysis of ivermectin for COVID-193 have publicly stated that they will now reanalyze and republish their now-retracted meta-analysis and will no longer include either of the two papers just mentioned. As these two papers1,6 were the only studies included in that meta-analysis to demonstrate an independently significant reduction in mortality, the revision will probably show no mortality benefit for ivermectin.
Several other studies that claim a clinical benefit for ivermectin are similarly fraught, and contain impossible numbers in their results, unexplainable mismatches between trial registry updates and published patient demographics, purported timelines that are not consistent with the veracity of the data collection, and substantial methodological weaknesses. We expect further studies supporting ivermectin to be withdrawn over the coming months.
Since the above primary studies were published, many hundreds of thousands of patients7 have been dosed with ivermectin, relying on an evidence base that has substantially evaporated under close scrutiny.
Relying on low-quality or questionable studies in the current global climate presents severe and immediate harms. The enormous impact of COVID-19 and the consequent urgent need to demonstrate the clinical efficacy of new therapeutic options provides fertile ground for even poorly evidenced claims of efficacy to be amplified, both in the scientific literature and on social media. This context can lead to the rapid translation of almost any apparently favorable conclusion from a relatively weak trial or set of trials into widespread clinical practice and public policy.
We believe that this situation requires immediate remediation. The most salient change required is a change in perspective on the part of both primary researchers and those who bring together the results of individual studies to draw wider conclusions. Specifically, we propose that clinical research should be seen as a contribution of data toward a larger omnibus question rather than an assemblage of summary statistics. Most, if not all, of the flaws described above would have been immediately detected if meta-analyses were performed on an individual patient data (IPD) basis. In particular, irregularities such as extreme terminal digit bias and the duplication of blocks of patient records would have been both obvious and immediately interrogable from raw data if provided.
We recommend that meta-analysts who study interventions for COVID-19 should request and personally review IPD in all cases, even if IPD synthesis techniques are not used. In a similar vein, all clinical trials published on COVID-19 should immediately follow best-practice guidelines and upload anonymized IPD so that this type of analysis can occur. Any study for which authors are not able or not willing to provide suitably anonymized IPD should be considered at high risk of bias for incomplete reporting and/or excluded entirely from meta-syntheses.
Hurdles to the release of IPD from clinical trials are well described, and generally addressable with careful anonymization and integration of data sharing plans at the ethical approval stage of trial planning.
We recognize that this is a change to long-accepted practice and is substantially more rigorous than the standards that are typically currently applied, but we believe that what has happened in the case of ivermectin justifies our proposal: a poorly scrutinized evidence base supported the administration of millions of doses of a potentially ineffective drug globally, and yet when this evidence was subjected to a very basic numerical scrutiny it collapsed in a matter of weeks. This research has created undue confidence in the use of ivermectin as a prophylactic or treatment for COVID-19, has usurped other research agendas, and probably resulted in inappropriate treatment or substandard care of patients.
We recognize that by recommending IPD review by default for meta-analysis of potential therapeutic agents in COVID-19 we are calling for change to nearly universally accepted practice over many decades, but the consequent potential for patient harm on a global scale demands nothing less.
Elgazzar, A. et al. Preprint at https://www.researchsquare.com/article/rs-100956/v3 (2020).
Bryant, A. et al. Am. J. Ther. 28, e434–e460 (2021).
Hill, A. et al. Open Forum Infectious Diseases https://doi.org/10.1093/ofid/ofab358 (2021).
Davey, M. The Guardian https://www.theguardian.com/science/2021/jul/16/huge-study-supporting-ivermectin-as-covid-treatment-withdrawn-over-ethical-concerns (16 July 2021).
Elgazzar, A. et al. Preprint at https://doi.org/10.21203/rs.3.rs-100956/v2 (2020).
Shakhsi Niaee, M. et al. Asian Pac. J. Trop. Med. 14, 266–273 (2021).
Mega, E. R. Nature 586, 481–482 (2020).
The authors declare no competing interests.
About this article
Cite this article
Lawrence, J.M., Meyerowitz-Katz, G., Heathers, J.A.J. et al. The lesson of ivermectin: meta-analyses based on summary data alone are inherently unreliable. Nat Med 27, 1853–1854 (2021). https://doi.org/10.1038/s41591-021-01535-y
Ivermectin under scrutiny: a systematic review and meta-analysis of efficacy and possible sources of controversies in COVID-19 patients
Virology Journal (2022)
Systematic review and meta-analysis of ivermectin for treatment of COVID-19: evidence beyond the hype
BMC Infectious Diseases (2022)
Meta-analyses based on summary data can provide timely, thorough and reliable evidence: don’t dismiss them yet
Nature Medicine (2022)
Nature Medicine (2021)