Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

# Performance of preclinical models in predicting drug-induced liver injury in humans: a systematic review

## Abstract

Drug-induced liver injury (DILI) causes one in three market withdrawals due to adverse drug reactions, causing preventable human suffering and massive financial loss. We applied evidence-based methods to investigate the role of preclinical studies in predicting human DILI using two anti-diabetic drugs from the same class, but with different toxicological profiles: troglitazone (withdrawn from US market due to DILI) and rosiglitazone (remains on US market). Evidence Stream 1: A systematic literature review of in vivo studies on rosiglitazone or troglitazone was conducted (PROSPERO registration CRD42018112353). Evidence Stream 2: in vitro data on troglitazone and rosiglitazone were retrieved from the US EPA ToxCast database. Evidence Stream 3: troglitazone- and rosiglitazone-related DILI cases were retrieved from WHO Vigibase. All three evidence stream analyses were conducted according to evidence-based methodologies and performed according to pre-registered protocols. Evidence Stream 1: 9288 references were identified, with 42 studies included in analysis. No reported biomarker for either drug indicated a strong hazard signal in either preclinical animal or human studies. All included studies had substantial limitations, resulting in “low” or “very low” certainty in findings. Evidence Stream 2: Troglitazone was active in twice as many in vitro assays (129) as rosiglitazone (60), indicating a strong signal for more off-target effects. Evidence Stream 3: We observed a fivefold difference in both all adverse events and liver-related adverse events reported, and an eightfold difference in fatalities for troglitazone, compared to rosiglitazone. In summary, published animal and human trials failed to predict troglitazone’s potential to cause severe liver injury in a wider patient population, while in vitro data showed marked differences in the two drugs’ off-target activities, offering a new paradigm for reducing drug attrition in late development and in the market. This investigation concludes that death and disability due to adverse drug reactions may be prevented if mechanistic information is deployed at early stages of drug development by pharmaceutical companies and is considered by regulators as a part of regulatory submissions.

## Introduction

Medicines save millions of lives and are considered a cost-effective intervention, effectively combating infections and making conditions previously considered incurable now manageable. However, drugs can also cause dangerous and fatal reactions in humans, both in clinical trials and after market approval. In the US, it is estimated that 2 million serious adverse drug reactions (ADRs) occur every year in hospitalized patients, with 100,000 people dying annually1. In the UK, it is estimated that ADRs kill more than 10,000 annually2 and account for 6.5% of hospital admissions3. ADRs also result in significant costs to pharmaceutical companies when drugs have to be withdrawn4, create human suffering, and place huge burdens on health systems5 and the economy. While some ADRs may occur as a result of inappropriate use or prescribing errors5, a major question remains why drugs that have met the preclinical and clinical testing required to secure regulatory approval go on to cause adverse reactions in humans. Regulatory agencies require a standard battery of tests based on International Council for Harmonisation of Technical Requirements for Pharmaceuticals for Human Use (ICH) guidelines to ensure the safety and efficacy of new drugs before they are tested in humans. This regulatory battery relies on preclinical animal testing in rodents (typically rats) and non-rodents (typically dogs or non-human primates (NHPs)). This process is lengthy, costly6,7, and risky, considering that most new drugs in development fail to gain approval8,9. While several factors contribute to the high attrition rates in new drug development, including the difficulty of detecting rare events in small clinical trial populations, it is widely agreed that a predominant reason is the failure of preclinical animal models (as well as some long-established in vitro assays using mono-layer transformed cell cultures10,11) to accurately predict clinical efficacy12,13,14,15 and safety16,17. There is increasing evidence that the current system of drug development needs to be modernized18 and that we need to use tests that are more predictive of human outcomes12.

In the last few decades, due to the advent of molecular biology techniques and high-throughput screening, a number of tests based on human biology have been developed and commercialised. These tests employ a variety of approaches, including stem cells, -omics-based technologies, organoids, organs-on-chips, and computational (in silico) approaches. These new tests, often referred to as new approach methodologies (NAMs), can be used to study the mechanisms of toxicity of chemicals and identify endpoints of concern, thus, allowing for more targeted follow-up of promising chemical or drug candidates, without subjecting every candidate to the recommended ICH and/or OECD guideline tests. There is now enormous optimism about NAMs12,18,19 and accumulating evidence to support their use in regulatory contexts across various economic sectors20. The US Food and Drug Administration’s (FDA) Center for Drug Evaluation and Research (CDER) encourages communication with stakeholders regarding NAMs and is committed to exploring the potential for NAMs to improve regulatory efficiency and expedite drug development21. A key government initiative has been the Toxicity Forecaster Programme (ToxCast), launched by the US Environmental Protection Agency (EPA) in 2007 to investigate the safety of industrial chemicals, cosmetics, pesticides and approved drugs using in vitro mechanistic information22. ToxCast uses high-throughput screening technologies based on human biology. The cells or proteins in these assays are exposed to chemicals and assessed for changes in biological activity that may suggest undesirable effects in humans. Nearly ten thousand chemicals have been screened to date against over a thousand molecular targets, which makes ToxCast the largest public in vitro database in the world.

Drug-induced liver injury (DILI) is the most frequent cause of acute liver failure in the Western world, accounting for more than half of all cases. DILI is also responsible for 3–5% of hospital admissions for jaundice23. Its incidence is estimated to be 14–19 cases per 100,000 persons, with jaundice occurring in 30% of cases23. Drug safety has become the bottleneck of drug development, with hepatotoxicity accounting for one in every 4.5 drug failures in clinical trials and one in every three market withdrawals caused by ADRs24. While there are standard clinical diagnostic markers of DILI, animal studies have only a limited ability to predict hepatic drug safety using these markers25. The aim of this study is to take an evidence-based approach26 to investigating how well ToxCast in vitro tests compare with preclinical animal tests in predicting liver-related ADRs in humans, with human pharmacovigilance data used as the true indicator of DILI incidence in the population. The current investigation is conducted according to a pre-registered protocol27 which outlines our intent to query ten drugs selected according to the presence or absence of documented DILI in human subjects. This is the first publication based on this protocol. Here we report data on two of the ten drugs, troglitazone and rosiglitazone maleate (henceforth referred to as rosiglitazone). This pair of anti-diabetic drugs come from the same class of thiazolidinediones but have differing effects on the human liver. Troglitazone was approved in the US in 1997 but withdrawn from the US market in 2000 after reports of deaths and severe liver failure requiring transplantation. Rosiglitazone was approved in the US in 1999 and remains on the US market28,29. We selected this pair of drugs because of their distinct liver safety profiles: their regulatory status is “withdrawn” for troglitazone and “on the market” for rosiglitazone, while their DILI risk classification (based on the US FDA Liver Toxicity Knowledge Base) is “most DILI concern” for troglitazone and “less DILI concern” for rosiglitazone30.

## Results

### Evidence stream 1: systematic review of in vivo studies

The literature searches identified 9288 references. After screening the titles/abstracts for relevance, we reviewed the remaining 690 references in full text. Two hundred and seventy-one publications were retained for data extraction, 42 of which were studies of troglitazone or rosiglitazone (22 on troglitazone and 22 on rosiglitazone, with 2 studies evaluating both compounds). The other 229 publications were studies of eight other drugs that will be analysed separately (see systematic review protocol) (Fig. 1).

The included studies are presented in Table 1a (troglitazone), b (rosiglitazone) and S2. Most of the studies of troglitazone were published after drug withdrawal in 2000, probably to study the mechanisms of toxicity involved.

#### Risk of bias for the included studies

A summary of our risk of bias (RoB) assessments for the included studies is presented in Fig. 2a (animal studies) and b (human studies).

#### Animal studies

Eight of the 11 RoB questions in the OHAT tool were applicable to the animal studies (Fig. 2a). Overall, many studies failed to report the information needed for reviewers to assess potential bias. In terms of selection, exclusion, and selective reporting bias, most studies had low or definitely low RoB, with a few exceptions. However, it is important to note that a large number of studies had at least two bias domains where there was a high RoB, including performance bias and detection bias.

#### Human studies

The human studies included 8 randomised controlled trials (RCTs), 4 cohort studies (Co), 3 case series (CaS), and 5 cohort/case series studies (Co/CaS). All the human studies had some methodological challenges (Fig. 2b) that impacted confidence in the effect estimates and conclusions, with lack of adequate reporting on randomisation and blinding (RCTs), selection bias, confounding and outcome assessment (Co), and selective outcome reporting (all study designs). Given the low number of studies, it is hard to draw any conclusions about RoB for the CaS and Co/CaS studies. The studies on rosiglitazone appeared to be either better reported or to have a lower RoB than the troglitazone studies.

Meta-analysis of the effect of troglitazone and rosiglitazone on liver function in animals and humans.

The included studies had a variety of study designs, dosing regimens, and liver endpoints reported (Table 1). Among the included endpoints, ALT elevation was the most frequently reported outcome, closely followed by AST. ALP, total bilirubin, liver weight, and histopathology were infrequently reported. To summarize troglitazone’s and rosiglitazone’s effects on the liver, we conducted a meta-analysis of the reported outcomes and present the results from all species on each of the five main liver outcomes in collated forest plots (S3), with the representative forest plot for ALT (Fig. 3) and summary of all five liver outcomes (Table 2) presented below. In most studies, wide confidence intervals due to the small number of participants (animal studies) or events (human trials) restrict our ability to draw definitive conclusions about the predictive ability of any specific endpoint or animal model. Given that for most liver injury markers there were a limited number of studies on each species, we caution against over-interpretation of these results.

In the studies where mice were given troglitazone there appeared to be an increase in ALT (4 studies, total 118 mice), AST (2 studies, total 78 mice), ALP (1 study, 19 mice), liver weight (1 study, 19 mice) and inconclusive changes in total bilirubin (2 studies, total 38 mice). We found fewer rosiglitazone studies in mice. In these studies, inconclusive results are reported for ALT (2 studies, total 28 mice), AST (2 studies, total 28 mice) and liver weight (3 studies, total 51 mice). Rosiglitazone appeared to reduce ALP (1 study, 12 mice) and total bilirubin (1 study, 12 mice).

In the studies where rats were given troglitazone, results were inconclusive for ALT (4 studies, total 305 rats), AST (4 studies, total 305 rats), total bilirubin (2 studies, total 46 rats) and liver weight (2 studies, total 48 rats). However, for ALP there appears to be a dose–response increase after troglitazone compared to control (2 studies, total 38 rats). We found fewer studies on rosiglitazone in rats. In these studies, inconclusive results are reported for ALT and AST (6 studies, total 68 rats), ALP (3 studies, total 32 rats) and total bilirubin (2 studies, total 18 rats). However, there appeared to be an increase in liver weight after rosiglitazone compared to control (2 studies, total 22 rats).

Also included was a controlled one year-long study on male and female NHPs. The 24 NHPs which were given troglitazone (300, 600 or 1200 mg/kg), had lower levels of ALT, AST, ALP and total bilirubin than the control group. However, liver weights were reported to be significantly higher in the NHPs given troglitazone compared to controls.

We found relatively few published human trials on both drugs. The only study, which measured ALT in 19 patients exposed to troglitazone, found no significant difference compared to placebo at 12 and 26 weeks. There were 5 published human trials of rosiglitazone, which together indicate no significant difference in ALT after drug exposure. There was no significant change in AST in patients in 8 studies with rosiglitazone (p = 0.06), except for one study which found a transient increase in AST at earlier time points31. We found no studies which reported ALP or bilirubin levels after troglitazone exposure. In the studies where patients were given rosiglitazone there were inconclusive results for ALP and AST (5 studies with 1548 patients) and for ALP (1 study with 156 patients). However, the one study that reported bilirubin levels found that bilirubin levels increased in patients on rosiglitazone compared to placebo (1 study with 156 patients).

Histopathology findings were reported in all studies in narrative form, making it impossible to perform a quantitative analysis. These data are summarized in S4. The sole published NHP study32 reported a dose–response increase in liver weight in NHPs after troglitazone administration, a two-fold relative liver weight increase compared to controls at the highest dose in both male and female animals. Studies without a placebo control or numerical data for the endpoints were not included in the meta-analysis and are summarized in S5.

#### GRADE assessment of confidence in evidence

We graded our confidence in the effect estimates (S6a, S6b) as “low” and “very low” for the outcomes measured (ALT, AST, ALP, bilirubin and liver weight) in all species (mice, rats, NHPs and humans) for both troglitazone and rosiglitazone. We have downgraded for RoB in both animal and human studies and in both RCTs and observational studies. Because the focus of this investigation was on drug safety for human patients, the animal studies were downgraded for indirectness. We have downgraded most of the outcomes for imprecision due to wide confidence intervals (CI) and the small number of subjects. We have also downgraded for publication bias, which we strongly suspect since regulatory studies do not appear to be in the public domain.

### Evidence stream 2: analysis of in vitro ToxCast data

As an initial step in evaluating the in vitro data, we compared the pharmacological activity of troglitazone and rosiglitazone. For this comparison, we included all assays in which both drugs were tested and there was a response (i.e., AC50 was not reported as NA, indicating not available, or 1,000,000, indicating no activity within concentrations tested). This workflow resulted in a total set of 437 assays performed on both drugs (Fig. 4A). Troglitazone was active in almost twice as many assays (129) as rosiglitazone (69). The Venn diagram (Fig. 4B) further dissects the data in this set in which both drugs were active (51 tests), as well as tests in which only troglitazone or rosiglitazone were active. Troglitazone not only was active in almost twice as many tests in the ToxCast database, but also uniquely activated 78 tests, compared to detected activity in only 10 tests for rosiglitazone. The full set of “positive” tests, along with identifying information and their AC50 values for rosiglitazone vs. troglitazone are available in S7.

Next, we tried to better understand the underlying biological processes represented by the positive assays for each drug using the ToxCast assignments of assays to biological processes. Troglitazone activated more assays across all biological processes represented by the common set of 437 assays for both drugs (Fig. 5). Unsurprisingly, given the desired target of both drugs on PPAR-γ, a nuclear transcription factor, the largest effects for both drugs were seen in assays with transcriptional factor or gene expression regulation targets. However, across these three broad biological processes, troglitazone consistently activated more endpoints, indicating more potential for off-target activity, leading to more potentially undesirable side effects (Fig. 5).

These results indicate that the in vitro/cellular assays-based pharmacological activity of troglitazone is higher across all measured biological processes relative to rosiglitazone. However, the AC50 data in in vitro assays do not account for human exposure levels and thus may not be relevant to the in vivo scenario, which usually needs to be addressed in the form of an IVIVE (In Vitro to In Vivo Extrapolation) model. For this reason, we introduced the NAS score and used it (Fig. 6) to put the in vitro results in the context of the human exposure and to stratify which of the assays/molecular targets might be activated more with each drug. Thus, the NAS metric allowed us to stratify the 437 tests based on their “activation potential” in patients administered a clinically relevant dose of either rosiglitazone or troglitazone.

As indicated, all positive tests fall into three distinct clusters—those activated by rosiglitazone alone (top cluster), those activated by troglitazone alone (middle cluster), and those activated by both drugs (bottom cluster). The strongest activated assays for both drugs measure the perturbation of PPAR-γ—the desired drug target. However, troglitazone activates several other assays (either uniquely or to a greater extent relative to rosiglitazone based on the relative NAS score). Notably, molecular targets of several of these assays are implicated in liver injury/repair pathways33,34,35,36,37,38,39,40 such as MMP1 (Matrix Metalloproteinase-1), NR3C1 (glucocorticoid receptor), NR1H3 (Liver X receptor alpha), NR1H4 (Bile Acid Receptor), TIMP1 (Tissue inhibitor of Metalloproteinase-1), ICAM1 (intracellular adhesion molecule 1), CXCL9 (T-cell chemoattractant/Chemokine ligand 9), IL8 (Interleukin/Chemokine ligand 8), CD38 (multifunctional ectoenzyme CD38 molecule), VDR (Vitamin D receptor), NRF2 (antioxidant nuclear transcription factor 2), and HLA-DR (MHC Class II cell surface receptor). S8 provides a heat map of assay targets that illustrates their (potential) relative activation in tests treated with troglitazone vs. rosiglitazone. In summary, the analyses of in vitro assays from the ToxCast database showed a clear distinction between rosiglitazone and troglitazone in terms of both the number and type of off-target biological activities, with troglitazone being active in almost twice as many tests as rosiglitazone. Moreover, several tests measuring mediators of liver effects were either uniquely activated or activated potentially to a higher level by troglitazone.

### Evidence stream 3: liver-related human ADRs

#### Troglitazone

The WHO Vigibase database, over the 4-year period 1998 to 2001, contained 6021 records of adverse events on troglitazone (liver-related + non-liver-related), with 1348 unique case IDs. 49 of these are of an unknown category because the event terms are under assessment for WHO ADR terminology (WHO-ART) and 204 adverse event terms are not accepted in WHO-ART, corresponding to “general disorder and administration site conditions” (or MedDRA Preferred Term: “Unevaluable event”) for 247 unique cases. This leaves 5768 total reported adverse events in 4 years for 1348 unique cases (6021 − [49 + 204]).

#### Rosiglitazone

A total of 1,141 adverse events (liver-related + non-liver-related) were reported for rosiglitazone during the 4 years, 2000 to 2003, since first coming on the market, with 280 unique cases. Of these 1141 adverse events, 9 are of an unknown category and 4 are not accepted in WHO-ART (as described for troglitazone), leaving 1128 (1141 − [9 + 4]) total reported adverse events in 4 years for 280 unique cases. From these data we calculated the comparison of fatal liver-related events between the two drugs (Table 3).

In summary, the analysis of the real world evidence data found that in the first 4 years since drug treaent approval there was a fivefold difference between both the total number of reported adverse events, and the number of unique cases with liver-related reported adverse events. Moreover, when comparing the total fatalities caused by and concurrent with the reported liver-related events, we observed an over tenfold difference with troglitazone-related deaths in just the first year of marketing in the US (54 deaths), and an eightfold difference in fatalities during the first four years on the market. These incidents in the first year since release of troglitazone on the US market served as a basis for the US FDA’s decision to withdraw the drug's marketing authorization.

## Discussion

To our knowledge, this is the first study to combine evidence from systematic review, in vitro and pharmacovigilance data to compare preclinical animal studies with in vitro methods for their ability to detect human ADRs. Our systematic review found that the animal and human studies did not detect a clear liver safety signal using traditional liver safety biomarkers and, therefore, failed to identify the potential of troglitazone to represent a DILI hazard. In contrast, the in vitro data revealed that troglitazone had twice as much off-target activity as rosiglitazone, alerting to a potential for troglitazone ADRs. The pharmacovigilance data found a clear difference between troglitazone and rosiglitazone in terms of liver injury, with a five-fold higher relative frequency of severe liver adverse events and an eight-fold higher relative frequency of liver fatalities in patients treated with troglitazone compared with rosiglitazone within the first four years of market approval. Our findings provide further evidence to support the accuracy of mechanistic in vitro data to predict drug induced ADRs in vivo, shown previously in many studies, both specific to glitazones and more generally41,42. As troglitazone and rosiglitazone are both long established drugs with a wealth of clinical safety data (and withdrawal in the case of Troglitazone) there are also substantial examples of their application to mechanistic in vitro approaches in published literature43,44,45.

It is important to note that in the troglitazone studies, the elevated liver weights in NHPs, the elevations in ALT, AST, ALP and liver weights in mice and the elevated ALP in rats, were not strong enough to be regarded as warning signals for human DILI risk. Furthermore, the current preclinical testing regime is not sufficiently robust for predicting adverse effects in human populations with low incidences, interpersonal variability, and/or where mechanisms are unknown. The DILI induced by troglitazone appears to be idiosyncratic (i.e. rare, caused by agents that have little or no intrinsic toxic activity, unpredictable, not dose dependent, not reproducible in animal models and with a variable latency period23). This might explain the inconsistent DILI signals in animal and human studies. When a drug enters the market, the number of patients treated with the drug increases, raising the chance of detecting idiosyncratic events. Although our analysis of pharmacovigilance data found a clear difference between troglitazone and rosiglitazone liver injury, this was a retrospective analysis; what is needed is a means of preventing adverse events. Our study suggests that in vitro data, typically available in the early stages of drug development, may help identify drugs that cause DILI, and can provide insight into the mechanisms of potential adverse effects. The use of such in vitro assays currently is mostly limited to early drug discovery stages. This review provides evidence that mechanistic approaches have a great potential to support regulatory review and approval, thereby supplementing the mandated animal safety data, which are frequently not sufficiently conclusive. The ToxCast in vitro data suggest that if we expand the types of assays acceptable in regulatory submissions to include in vitro human biology-based data, potentially unsafe compounds can be prevented from entering the market and causing human suffering.

In conclusion, we found that neither published animal nor human studies, taken together, were able to accurately predict the potential of troglitazone to represent a DILI hazard in humans, while in vitro data were able to detect the hazard. Our findings indicate that the paired-compound approach to comparing various tests, pioneered here, could be used to evaluate the relevance and predictivity of in vitro human biology-based approaches, bringing a systematic, transparent, and evidence-based approach to drug development. Adopting such approaches could make new drugs safer and reduce late attrition, preventing unfortunate human ADRs and deaths and making the drug development process more financially sustainable. We also propose that transparent protocol-driven evidence-based approaches should become standard in preclinical research and that this would benefit the pharmaceutical industry, society, and first and foremost, patients.

## Methods

We combined three evidence streams: a systematic literature review of published human clinical trials and animal studies (Evidence Stream 1); in vitro data from the US EPA ToxCast database (Evidence Stream 2); and human ADRs from Vigibase, a global pharmacovigilance database of individual case safety reports run by the World Health Organisation (WHO) (Evidence Stream 3).

### Evidence stream 1: systematic review of in vivo studies

The aim of the systematic review was to investigate the effect on liver function of in vivo exposures to troglitazone and rosiglitazone in humans and standard preclinical animal models (rats, mice, dogs, and NHPs), as outlined in the registered systematic review protocol CRD4201811235327.

#### Search strategy

On February 7th 2020, PubMed, Embase (Embase.com), and Web of Science (Clarivate Analytics) were searched as outlined in the protocol27. The search strategies were developed by a medical librarian (RW) in collaboration with the review team and peer-reviewed by GV. The complete search strategies are presented in S1.

#### Eligibility criteria

##### Populations

Controlled studies of troglitazone or rosiglitazone in humans and experimental rats, mice, dogs and non-human primates (NHPs) with hepatic endpoints were included. Humans with diseases that are not a primary indication for the drugs of interest were excluded, as were patients with pre-existing liver injury. Genetically modified animal models or disease models were excluded.

##### Interventions

Studies were included that indicated the oral, intravenous, or intraperitoneal administration of troglitazone or rosiglitazone. Studies of drug combinations were excluded.

##### Controls

Human and animal studies with a control or placebo group, or pre-treatment values, were included.

##### Outcomes

Standard clinical and preclinical hepatic effects of troglitazone or rosiglitazone in experimental animals and humans were included. Specifically, plasma levels of liver enzyme tests (alanine transaminase (ALT); aspartate transaminase (AST); alkaline phosphatase (ALP); and bilirubin), histopathology results, absolute and relative liver weight, were included. Mechanistic non-standard observations (i.e. gene expression, proteomics, micro RNAs, and similar observations) were excluded.

##### Study types

English, Norwegian, Dutch, Swedish and Danish studies were included. Conference abstracts, narrative reviews, opinion papers, case reports and other publication types where the original outcome data are not reported, were excluded. We also excluded reports of general adverse events (AE) monitoring, AE database analyses and single AE case reports from Evidence Stream 1 for two reasons:

1. 1.

Pharmacovigilance reports do not include controls;

2. 2.

These events are included in the pharmacovigilance reports (Evidence Stream 3), thus including them in Stream 1 would be double counting.

#### Screening and data extraction

Two reviewers independently screened the literature for relevant studies, first using titles and abstracts and then full texts. Sysrev online software (Insilica LLC, Bethesda, MD, US) was used to manage the screening process. DistillerSR online software (Evidence Partners, Ottawa, ON, Canada) was used to extract the data from included studies.

#### Analysis

For each drug, data from human and animal studies were analyzed separately. For each outcome/endpoint, data from each species, dose, follow-up time, and study design were collected and presented in forest plots and tables. Continuous data are presented as standardized mean difference with 95% confidence intervals. Where we considered the population, dose, design, and comparison similar enough, we conducted a meta-analysis based on a random effect model. The internal validity of both animal and human studies was assessed using the OHAT risk-of-bias (RoB) tool54. Two reviewers independently assessed the OHAT criteria for each included study, with disagreements resolved by discussion and with the help of a third reviewer if necessary. Our confidence in the evidence was assessed by two reviewers as well, using the GRADE (Grading of Recommendations Assessment, Development and Evaluation) approach55.

### Evidence stream 2: in vitro mechanistic data

The in vitro data were obtained from US EPA ToxCast database which is the largest curated collection of > 1100 molecular and cellular assays for up to 10,000 chemicals, including ~ 500 FDA-approved drugs22. The US EPA contributes the ToxCast data to a US federal agency collaboration known as Tox21 (Toxicology in the 21st Century) which pools chemical research, data, and screening tools from several US federal agencies. As part of the EPA’s commient to share its chemical data openly and transparently, all ToxCast chemical screening data are publicly available via the ToxCast dashboard, which allows users to search and query the data.

Datasets of troglitazone and rosiglitazone were retrieved from the ToxCast database (invitrodb_v2) and analysis was done according to pre-registered protocol: https://doi.org/10.5281/zenodo.252909. All assays, without pre-selection, in which the drugs were tested in the ToxCast database were used in the analysis. The ToxCast data processing pipeline consists of multiple processing steps resulting in progressively refined estimates of AC50 values (compound concentration at 50% of maximal assay activity)56. For this analysis, we retrieved Level 5 data containing AC50 values from the best performing model used to fit dose–response curves. All AC50 values were used as downloaded per description above and no further processing was performed.

#### Analysis

All assays/targets and their pathways were compared across the troglitazone/rosiglitazone drug pair using normalised activation score (NAS) values. The NAS score was developed here to adjust the activities of the drugs in the in vitro tests (AC50) to human exposure. The NAS value reflects the difference between the average peak concentration that a drug achieves in plasma after its administration to humans at the highest dose (Cmax) and the AC50 value for the assay in an in vitro test. Normalisation of in vitro data to human maximal plasma concentrations was first proposed for use in the normalisation of drug-drug interactions57 and subsequently was used for normalisation of in vitro data58. The human Cmax used here is taken from the US FDA drug insert for each drug. Troglitazone has a reported Cmax of 1.61 µg/mL at the standard therapeutic dose of 400 mg/day59. Rosiglitazone has a reported Cmax of 156 ng/mL at the standard therapeutic dose of 2 mg/day60. The NAS formula used:

$$NAS = \frac{{{\varvec{C}}_{{{\varvec{max}}}} - {\varvec{AC}}_{50} }}{{{\varvec{C}}_{{{\varvec{max}}}} }}$$

The NAS value was used as a ranking metric to stratify assays using the following logic:

• All assays/targets thereof with NAS value > 0 were ranked to have a putative “higher activation potential” or higher possible off-target/toxic effect, because the Cmax concentrations in patients are expected to exceed AC50 values.

• All assays/targets thereof with NAS values >  − 4 but NAS < 0 were ranked to have a putative “modest activation potential” or modest possible off-target effect, because, in this case, drug plasma concentration is equal to and up to fivefold lower than the AC50 value.

• All assays/targets thereof with NAS values <  − 4 were ranked to have a putative “low activation potential” or unlikely to cause off-target/toxic effects at these concentrations, because drug plasma concentration is more than fivefold lower than the AC50 value.

All assays or their affected gene targets/pathways were compared across the drug pair using NAS values either directly or after grouping them within their respective biological processes as represented within these assay descriptions in the ToxCast database. Gene targets/pathways associated with the assays were identified using the ToxCast assay list of targets. Differentially affected gene targets/pathways were identified based on differences in NAS values (e.g. higher for troglitazone vs. rosiglitazone) and literature linking these gene targets/pathways with liver injury or repair pathways. The analysis was performed according to the pre-published protocol61. All analysed data files, and the R code used as part of this analysis, are available at: (https://github.com/Sri-Bandhakavi/ToxCast_Rosiglitazone_Troglitazone_EBTC_Analysis).

### Evidence stream 3: liver-related human ADRs

Vigibase is the largest continuously-updated database in the world, with over 20 million reports of suspected adverse effects of medicines submitted since 1968 by member countries of the WHO Programme for International Drug MonitoringWHO Programme for International Drug Monitoring. On January 2, 2018 individual cases where liver toxicity was reported as an adverse event were retrieved separately from VigiBase for troglitazone and rosiglitazone. Data was analysed using the general WHO Guideline for using Vigibase data62. To accommodate for the differences in total time on the market for the two drugs and latency in reporting, Vigibase ADR data for the first 4 years since each drug's approval by US FDA were used in this analysis (i.e. 1998–2001 for troglitazone and 2000–2003 for rosiglitazone).

As real world evidence, the liver-related adverse events reported in Vigibase are considered the “gold standard” for human safety in our study; these data were analysed separately for troglitazone and rosiglitazone in the following step-wise analysis:

• The number of unique cases was counted. The total liver-related adverse events count was compared with other non-liver-related adverse events for outcomes for the first 4 years after approval in the US, and plotted against the dose and demographic characteristics.

• The liver-related adverse events coded as MedDRA (Medical Dictionary for Regulatory Activities) terms in Vigibase were classified into 6 prime categories of important liver endpoints (Table 4).

• The proportional reporting ratio and probability of occurrence were calculated for 4 years each for rosiglitazone and troglitazone separately63.

All analyses were performed according to the pre-published protocol http://doi.org/10.5281/zenodo.252892264.

#### Ethics approval

Ethics approval was not required for this study as it involves the analysis of secondary data sources only. The views expressed in this manuscript do not necessarily represent those of the U.S. Food and Drug Administration.

#### Transparency statement

The lead author (the manuscript's guarantor) affirms that the manuscript is an honest, accurate, and transparent account of the study being reported; that no important aspects of the study have been omitted; and that any discrepancies from the study as originally planned and registered have been explained.

## Data availability

The datasets generated during and/or analysed during the current study are available from the corresponding author on reasonable request.

## References

1. 1.

US Food and Drug Administration, “Preventable Adverse Drug Reactions: A Focus on Drug Interactions. ADRs: Prevalence and incidence,” https://www.fda.gov/drugs/drug-interactions-labeling/preventable-adverse-drug-reactions-focus-drug-interactions (Accessed 24 Mar 2020).

2. 2.

Pirmohamed, M. et al. Adverse drug reactions as cause of admission to hospital: Prospective analysis of 18,820 patients. Br. Med. J. 329(7456), 15–19 (2004).

3. 3.

Bouvy, J. C., De Bruin, M. L. & Koopmanschap, M. A. Epidemiology of adverse drug reactions in Europe: a review of recent observational studies. Drug Saf. 38(5), 437–453 (2015).

4. 4.

Gwathmey, J. K., Tsaioun, K. & Hajjar, R. J. Cardionomics: a new integrative approach for screening cardiotoxicity of drug candidates. Expert Opin. Drug Metab. Toxicol. 5(6), 647–660 (2009).

5. 5.

Elliott, R., Camacho, E., Campbell, F., Jankovic, D., St James, M. M., Kaltenthaler, E., & Faria, R. Prevalence and economic burden of medication errors in the NHS in England: rapid evidence synthesis and economic analysis of the prevalence and burden of medication error in the UK www.eepru.org.uk/wp-content/uploads/2020/03/medication-error-report-edited-27032020.pdf (Accessed 27 Mar 2020), (2018).

6. 6.

Mullin, R. Tufts study finds big rise in cost of drug development. Chem. Eng. News http://www.cen.acs.org/articles/92/web/2014/11/Tufts-Study-Finds-Big-Rise.hl (Accessed 24 Mar 2020), (2014).

7. 7.

Wouters, O. J., McKee, M. & Luyten, J. Estimated research and development invesent needed to bring a new medicine to market, 2009–2018. JAMA 323(9), 844 (2020).

8. 8.

Thomas, D. W., et al. Clinical development success rates 2006–2015. https://www.bio.org/sites/default/files/legacy/bioorg/docs/Clinical%20Development%20Success%20Rates%202006-2015%20-%20BIO,%20Biomedtracker,%20Amplion%202016.pdf (Accessed 27 Mar 2020).

9. 9.

Harrison, R. K. Phase II and phase III failures: 2013–2015. Nat. Rev. Drug Discov. 15(12), 817–818 (2016).

10. 10.

Pampaloni, F., Reynaud, E. G. & Stelzer, E. H. K. The third dimension bridges the gap between cell culture and live tissue. Nat. Rev. Mol. Cell Biol. 8(10), 839–845 (2007).

11. 11.

Ravi, M., Paramesh, V., Kaviya, S. R., Anuradha, E. & Solomon, F. D. P. 3D cell culture systems: advantages and applications. J. Cell. Physiol. 230(1), 16–26 (2015).

12. 12.

BioIndustry Association and Medicines Discovery Catapult, State of the Discovery Nation 2018 and the role of the Medicines Discovery Catapult, http://www.md.catapult.org.uk/FlipBuilder/mobile/index.hl (Accessed 24 Mar 20)), (2018).

13. 13.

Geerts, H. Of mice and men. Bridging the translational disconnect in CNS drug discovery. CNS Drugs 23(1), 915–926 (2009).

14. 14.

Perrin, S. Preclinical research: make mouse studies work. Nature 507, 423–425 (2014).

15. 15.

Wall, R. J. & Shani, M. Are animal models as good as we think?. Theriogenology 69(1), 2–9 (2008).

16. 16.

Waring, M. J. et al. An analysis of the attrition of drug candidates from four major pharmaceutical companies. Nat. Rev. Drug Discov. 14(7), 475–486 (2015).

17. 17.

Hwang, T. J. et al. Failure of investigational drugs in late-stage clinical development and publication of trial results. JAMA Intern. Med. 176(12), 1826 (2016).

18. 18.

Alliance for Human Relevant Science, Accelerating the growth of human relevant life sciences in the United Kingdom. A white paper by the alliance for human relevant science, www.humanrelevantscience.org/wp-content/uploads/Accelerating-the-Growth-of-Human-Relevant-Sciences-in-the-UK_2020-final.pdf (Accessed 27 Mar 2020), (2020).

19. 19.

20. 20.

Archibald, K., Tsaioun, K., Kenna, J. G. & Pound, P. Better science for safer medicines: the human imperative. J. R. Soc. Med. 111(12), 433–438 (2018).

21. 21.

Avila, A. M. et al. An FDA/CDER perspective on nonclinical testing strategies: classical toxicology approaches and new approach methodologies (NAMs). Regul. Toxicol. Pharmacol. 114, 104662 (2020).

22. 22.

US Environmental Protection Agency, “Toxicity forecasting. Advancing the next generation of chemical evaluation,” www.epa.gov/chemical-research/toxicity-forecasting (Accessed Mar 24 2020), (2019).

23. 23.

Hoofnagle, J. H. & Björnsson, E. S. Drug-induced liver injury: types and phenotypes. N. Engl. J. Med. 381(3), 264–273 (2019).

24. 24.

Watkins, P. B. Drug safety sciences and the bottleneck in drug development. Clin. Pharmacol. Ther. 89(6), 788–790 (2011).

25. 25.

Olson, H. et al. Concordance of the toxicity of pharmaceuticals in humans and in animals. Regul. Toxicol. Pharmacol. 32(1), 56–67 (2000).

26. 26.

Hoffmann, S. et al. A primer on systematic reviews in toxicology. Arch. Toxicol. 91(7), 2551–2575 (2017).

27. 27.

Tsaioun, K. et al. Systematic reviews of the hepatotoxic effects of specified compounds/drugs as observed in humans and experimental animals (rats, mice, dogs and non-human primates). PROSPERO 2018 CRD42018112353. Available from https://www.crd.york.ac.uk/prospero/display_record.php?ID=CRD42018112353.

28. 28.

Scheen, A. J. Thiazolidinediones and liver toxicity. Diabetes Metab. 27(3), 305–313 (2001). PMID: 11431595.

29. 29.

Isley, W. L. & Oki, J. C. Hepatotoxicity of thiazolidinediones. Diabetes Obes. Metab. 3, 389–392 (2000).

30. 30.

Chen, M. et al. DILIrank: the largest reference drug list ranked by the risk for developing drug-induced liver injury in humans. Drug Discov. Today 21(4), 648–653 (2016).

31. 31.

Dereli, D., Dereli, T., Bayraktar, F., Ozgen, A. G. & Yilmaz, C. Endocrine and metabolic effects of rosiglitazone in non-obese women with polycystic ovary disease. Endocr. J. 52(3), 299–308 (2005).

32. 32.

Rothwell, C., McGuire, E. J., Altrogge, D. M., Masuda, H. & Iglesia, F. A. Chronic toxicity in monkeys with the thiazolidinedione antidiabetic agent troglitazone. J. Toxicol. Sci. 27(1), 35–47 (2002).

33. 33.

Duarte, S., Baber, J., Fujii, T. & Coito, A. J. Matrix metalloproteinases in liver injury, repair and fibrosis. Matrix Biol. 44–46, 147–156 (2015).

34. 34.

Saiman, Y. & Friedman, S. L. The role of chemokines in acute liver injury. Front. Physiol. 3, 213 (2012).

35. 35.

Böhm, F. et al. FGF receptors 1 and 2 control chemically induced injury and compound detoxification in regenerating livers of mice. Gastroenterology 139(4), 1385–1396 (2010).

36. 36.

Rudraiah, S., Zhang, X. & Wang, L. Nuclear receptors as therapeutic targets in liver disease: are we there yet?. Annu. Rev. Pharmacol. Toxicol. 56(1), 605–626 (2016).

37. 37.

Zimmermann, H. W., Trautwein, C. & Tacke, F. Functional role of monocytes and macrophages for the inflammatory response in acute liver injury. Front. Physiol. 3, 56 (2012).

38. 38.

Gomez-Ospina, N. et al. Mutations in the nuclear bile acid receptor FXR cause progressive familial intrahepatic cholestasis. Nat. Commun. 7(1), 10713 (2016).

39. 39.

Jadeja, R. N., Upadhyay, K. K., Devkar, R. V. & Khurana, S. Naturally occurring Nrf2 activators: potential in treaent of liver injury. Oxid. Med. Cell. Longev. 2016, 1–13 (2016).

40. 40.

Zúñiga, S., Firrincieli, D., Housset, C. & Chignard, N. Vitamin D and the vitamin D receptor in liver pathophysiology. Clin. Res. Hepatol. Gastroenterol. 35(4), 295–302 (2011).

41. 41.

Yu, L. et al. Integrating in vitro testing and physiologically-based pharmacokinetic (PBPK) modelling for chemical liver toxicity assessment: a case study of troglitazone. Environ. Toxicol. Pharmacol. 74, 103296 (2020).

42. 42.

Yokoyama, Y. et al. Comparison of drug metabolism and its related hepatotoxic effects in heparg, cryopreserved human hepatocytes, and HepG2 cell cultures. Biol. Pharm. Bull. 41, 722–732 (2018).

43. 43.

Lloyd, S. et al. Differential in vitro hepatotoxicity of troglitazone and rosiglitazone among cryopreserved human hepatocytes from 37 donors. Chem. Biol. Interact. 142, 57–71 (2002).

44. 44.

Edling, Y., Sivertsson, L. K., Butura, A., Ingelman-Sundberg, M. & Ek, M. Increased sensitivity for troglitazone-induced cytotoxicity using a human in vitro co-culture model. Toxicol. Vitr. 23, 1387–1395 (2009).

45. 45.

Usui, T., Hashizume, T., Katsumata, T., Yokoi, T. & Komuro, S. In vitro investigation of the glutathione transferase M1 and T1 null genotypes as risk factors for troglitazone-induced liver injury. Drug Metab. Dispos. 39, 1303–1310 (2011).

46. 46.

Khetani, S. R. et al. Use of micropatterned cocultures to detect compounds that cause drug-induced liver injury in humans. Toxicol. Sci. Off. J. Soc. Toxicol. 132(1), 107–117 (2013).

47. 47.

Bale, S. S. et al. In vitro platforms for evaluating liver toxicity. Exp. Biol. Med. 239(9), 1180–1191 (2014).

48. 48.

Villeneuve, D. L. et al.. Adverse outcome pathway (AOP) development I: Strategies and principles. Toxicol. Sci. 142(2), 312–320. https://doi.org/10.1093/toxsci/kfu199 (2014).

49. 49.

OECD. Guidance Document for the Use of Adverse Outcome Pathways in Developing Integrated Approaches to Testing and Assessment (IATA), OECD Series on Testing and Assessment, No. 260, OECD Publishing, Paris. https://doi.org/10.1787/44bb06c1-en (2017).

50. 50.

Jeong, J. & Choi, J. Development of AOP relevant to microplastics based on toxicity mechanisms of chemical additives using ToxCast and deep learning models combined approach. Environ. Int. 137, 105557 (2020).

51. 51.

Proctor, W. R. et al. Utility of spherical human liver microtissues for prediction of clinical drug-induced liver injury. Arch. Toxicol. 91, 2849–2863 (2017).

52. 52.

Bin Ramli, M. N. et al. Human pluripotent stem cell-derived organoids as models of liver disease. Gastroenterology 159(4), 1471–1486 (2020).

53. 53.

EFSA Panel on Contaminants in the Food Chain (EFSA CONTAM Panel), Risk to human health related to the presence of 2 perfluoroalkyl substances in food. EFSA J. https://doi.org/10.2903/j.efsa.2020.6223

54. 54.

Rooney, A. A., Boyles, A. L., Wolfe, M. S., Bucher, J. R. & Thayer, K. A. Systematic review and evidence integration for literature-based environmental health science assessments. Environ. Health Perspect. 122(7), 711–718 (2014).

55. 55.

Balshem, H. et al. GRADE guidelines: 3. Rating the quality of evidence. J. Clin. Epidemiol. 64(4), 401–406 (2011).

56. 56.

Filer, D. L., Kothiya, P., Setzer, R. W., Judson, R. S. & Martin, M. T. tcpl: the ToxCast pipeline for high-throughput screening data. Bioinformatics 33(4), btw680 (2016).

57. 57.

Obach, R. S., Walsky, R. L. & Venkatakrishnan, K. Mechanism-based inactivation of human cytochrome P450 enzymes and the prediction of drug-drug interactions. Drug Metab. Dispos. 35, 246–255 (2007).

58. 58.

Xu, J. J. et al. Cellular imaging predictions of clinical drug-induced liver injury. Toxicol. Sci. 105, 97–105 (2008).

59. 59.

US Food and Drug Administration, Application number 020720, S12, S14. (1999).

60. 60.

US Food and Drug Administration, Medication Guide (Avandia). (2019).

61. 61.

Tsaioun, K. et al.. Evidence-based methodology for identifying drugs with potential human liver toxicities using US EPA ToxCast data set (Version 6). Zenodo. https://doi.org/10.5281/zenodo.2529091 (2018).

62. 62.

W. C. C. for I. D. monitoring Uppsala Monitoring Centre, “Guideline for using VigiBase data in studies,” 2018. Available: www.who-umc.org. (Accessed: 05 Feb 2021).

63. 63.

Evans, S. J. W., Waller, P. C. & Davis, S. Use of proportional reporting ratios (PRRs) for signal generation from spontaneous adverse drug reaction reports. Pharmacoepidemiol. Drug Saf. 10(6), 483–486 (2001).

64. 64.

Tsaioun, K, et al. (2018, December 30). Evidence-based methodology for identifying drugs with potential human liver toxicities using WHO Vigibase data (Version 1). Zenodo. http://doi.org/10.5281/zenodo.2528922

## Acknowledgements

Thank you to the EBTC Tox21 working group members Alan Boobis (University College London), Patience Browne (OECD), and Daniele Wikoff (ToxStrategies), who participated in early discussions, assisted with protocol development, attended stakeholder meetings, and answered specific questions from authors that helped resolve many technical issues as we continued with this complex project. We also thank the Sysrev team for letting us be the beta users of their software for screening titles and abstracts.

## Funding

The study was initiated, conducted, and managed by the Evidence-based Toxicology Collaboration (EBTC) at the Johns Hopkins Bloomberg School of Public Health, which received funding from the Center for Alternatives to Animal Testing at Johns Hopkins University (CAAT), which in turn received funding from the Johns Hopkins Bloomberg School of Public Health (JHSPH), a non-profit institution. CAAT and JHSPH had no role in the study design; the collection, analysis, and interpretation of data; the writing of the report; or the decision to submit the article for publication. The authors and contributors made their contributions in their spare time. All authors had full access to the raw data used in the study and can take responsibility for the integrity of the data and the accuracy of the data analysis.

## Author information

Authors

### Contributions

The Evidence-based Toxicology Collaboration (EBTC) Working Group at Johns Hopkins Bloomberg School of Public Health contributed throughout the project, acting at various times as a wider stakeholder group. K.T. and H.D. planned the study and contributed equally to the conduct and coordination of the project. The specific contributions are listed as follows: G.E.V. conducted the meta-analysis and risk-of-bias and GRADE assessments of human studies; S.E.F. conducted the risk-of-bias and GRADE assessments of animal studies; S.B. conducted the US EPA ToxCast data analysis; J.M. conducted the WHO Vigibase adverse events data analysis; R.R. provided input in protocols development, screening of the literature and manuscript drafting; C.H.C.L. provided support in animal data meta-analysis in the systematic review; B.K. conducted the risk-of-bias assessments of animal studies in the systematic review, data extraction and quality control, and PRISMA adherence; R.A.W. guided the working group in developing a search strategy and conducted literature searches for the systematic review; and M.C. provided input during planning stages and stakeholder feedback at various stages of the project as well as in developing the manuscript. P.P. assembled and drafted revisions of the manuscript, which all authors contributed to. The majority of the co-authors participated in screening the literature for the systematic review. All authors have read and approved the final version of the manuscript. The corresponding author attests that all listed authors meet authorship criteria and that no others meeting the criteria have been omitted.

### Corresponding author

Correspondence to Katya Tsaioun.

## Ethics declarations

### Competing interests

The authors declare no competing interests.

### Publisher's note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

## Rights and permissions

Reprints and Permissions

Dirven, H., Vist, G.E., Bandhakavi, S. et al. Performance of preclinical models in predicting drug-induced liver injury in humans: a systematic review. Sci Rep 11, 6403 (2021). https://doi.org/10.1038/s41598-021-85708-2

• Accepted:

• Published: