In the Phase 3 REFLECT trial in patients with unresectable hepatocellular carcinoma (uHCC), the multitargeted tyrosine kinase inhibitor, lenvatinib, was noninferior to sorafenib in the primary outcome of overall survival. Post-hoc review revealed imbalances in prognostic variables between treatment arms. Here, we re-analyse overall survival data from REFLECT to adjust for the imbalance in covariates.
Univariable and multivariable adjustments were undertaken for a candidate set of covariate values that a physician panel indicated could be prognostically associated with overall survival in uHCC. The values included baseline variables observed pre- and post-randomisation. Univariable analyses were based on a stratified Cox model. The multivariable analysis used a “forwards stepwise” Cox model.
Univariable analysis identified alpha-fetoprotein (AFP) as the most influential variable. The chosen multivariable Cox model analysis resulted in an estimated adjusted hazard ratio for lenvatinib of 0.814 (95% CI: 0.699–0.948) when only baseline variables were included. Adjusting for post-randomisation treatment variables further increased the estimated superiority of lenvatinib.
Covariate adjustment of REFLECT suggests that the original noninferiority trial likely underestimated the true effect of lenvatinib on overall survival due to an imbalance in baseline prognostic covariates and the greater use of post-treatment therapies in the sorafenib arm.
Trial number: NCT01761266 (Submitted January 2, 2013).
In a Phase 3 trial (REFLECT) in patients with unresectable hepatocellular carcinoma (uHCC), the multitargeted tyrosine kinase inhibitor, lenvatinib, was shown to be noninferior to the standard-of-care treatment, sorafenib, in terms of the primary outcome of overall survival.1 The study reported that the median overall survival was longer for lenvatinib (13.6 months) compared to 12.3 months for sorafenib, but this difference was not statistically significant. The hazard ratio from a stratified Cox proportional hazards model was 0.92 with a 95% confidence interval (CI) of 0.79 to 1.06. Lenvatinib passed the noninferiority test that the upper CI of the hazard ratio for overall survival should be no greater than 1.08.1
Although superiority for lenvatinib in terms of overall survival cannot be shown based on the primary efficacy analysis, there are a number of reasons why lenvatinib may be superior to the standard-of-care treatment, sorafenib. First, the superiority of lenvatinib based on secondary end points of progression-free survival, with a reported hazard ratio in the Phase 3 trial of 0.66 (95% CI: 0.57–0.77), and objective response rate, with a reported odds ratio of 3.13 (95% CI: 2.15–4.56). Second, an imbalance in the baseline prognostic factors appeared to bias the outcomes against lenvatinib. Finally, there was a greater number of post-treatment therapies used after sorafenib compared with lenvatinib, leading the authors of the original study to speculate that: “If post-progression survival is prolonged by…post-study treatments, this could lead to a dilution of the observed overall survival treatment benefit”.1
The aim of this manuscript is to assess an alternative analysis of the overall survival data from REFLECT to identify and adjust for the imbalance in covariates.
REFLECT was a multicentre, randomised, open-label, noninferiority, phase 3 study that compared the efficacy and safety of lenvatinib vs. sorafenib as a first-line systemic treatment in patients with uHCC.1 Patients were randomly assigned in a 1:1 ratio to treatment with either lenvatinib 12 mg (if baseline bodyweight was ≥60 kg) or 8 mg (if baseline bodyweight was <60 kg) given once daily orally, or sorafenib 400 mg given twice daily orally. Patients were stratified by: geographical region; presence of macroscopic portal vein invasion (MPVI), extrahepatic spread (EHS) or both; Eastern Cooperative Oncology Group performance status (ECOG PS) 0 or 1; and bodyweight (<60 kg or ≥60 kg).1 No crossover to lenvatinib from the sorafenib arm was allowed.
The choice of covariates to consider as candidate variables in this analysis was informed by discussion among the clinical authors of the manuscript, who formed the purposive expert sample.2 Each clinician was asked to comment on the likely prognostic importance of the baseline patient characteristics from the original clinical trial,1 and all covariates that were considered as potentially important by any of the clinical experts were used in the initial determination of which variables to include in a model for overall survival.
To determine the potential importance of the candidate variables identified by the clinical experts, each variable was entered into the Cox proportional hazards regression model as a univariate adjustment of the treatment effect. This univariate analysis retained the original stratification variables of: geographical region; presence or absence of MPVI, EHS or both; ECOG PS and bodyweight.
A multivariable adjusted analysis was then developed using a forward stepwise procedure from the candidate variables identified by clinicians. In this case, “forwards stepwise” indicates that the procedure starts from a model with treatment effect as the only covariate and systematically considers each candidate variable for inclusion. Candidate covariates were included in further analysis if the p-value was <0.05. As variables are added, individual p-values for variables in the model can change—in addition to selecting additional variables for inclusion to the model, the stepwise procedure also drops existing variables from the model if the p-value becomes >0.1. In the multivariable analysis, the original stratification variables were included as potential covariates, but were not retained as stratification variables.
Sensitivity analyses were conducted on a number of aspects of the analysis: the default Wald test for the standard stepwise procedure was replaced by the Likelihood Ratio (LR) test; variable selection was also tested using the Akaike information criterion (AIC),3 which further penalises the likelihood for the number of parameters in the model; and backwards selection (where the model starts with all candidate variables) was contrasted with the forwards selection procedure for determining variable selection. In addition, post-treatment variables were included in the analysis to adjust for the potential dilution effect of the imbalance in post-progression therapies between the two treatment arms (i.e. post-randomisation confounding).
All statistical analyses were undertaken in STATA™ version 14.4
The baseline characteristics for REFLECT, which were considered by the clinical authors to have potential prognostic importance, are presented in Table 1 for the sorafenib and lenvatinib treatment groups. Age and sex are also included for information, although it should be noted that these demographic factors were not considered to impact disease prognosis by the authors. Geographical region (Western vs. Asia-Pacific), MPVI or EHS or both, ECOG PS (0 or 1), and bodyweight (<60 kg or ≥60 kg) were stratification variables in the original clinical trial. Table 1 shows these variables were balanced across the study arms. Some of the non-stratification covariates show potential imbalance: in particular, AFP is known to correlate with prognosis1,5 and here the imbalance suggests that the sorafenib arm, with a greater proportion of patients having AFP < 200 ng/ml level, may have included patients with disease who showed better prognosis.6,7 There is also evidence that suggests that the effect of sorafenib on overall survival is dependent on patients’ hepatitis status with a greater improvement in survival for sorafenib-treated patients positive for HCV compared to patients positive for HBV, in which sorafenib is less active.7 Also included in Table 1 are three post-treatment variables reflecting anti-cancer treatment therapies received post-treatment with either lenvatinib or sorafenib. The post-treatment therapy variable includes both post-treatment anti-cancer procedures (e.g. radiotherapy) and/or post-treatment anti-cancer medications.
The Forest plot in Fig. 1 shows the univariable impact on the estimated hazard ratio for lenvatinib treatment compared to sorafenib after adjusting for each covariate in Table 1. In terms of these univariable results, MPVI or EHS or both, AFP < 200 ng/mL, disease site, hepatitis B aetiology, and receipt of a previous procedure are all predictive of overall survival and adjusting for them influences the estimated hazard ratio of the treatment effect in favour of lenvatinib. The Child-Pugh score is also predictive for overall survival, but adjustment favours sorafenib. Finally, adjustments for the imbalance in the post-treatment covariates also have a strong impact on the treatment hazard ratio in favour of lenvatinib. Overall, it is clear from the univariable analysis that AFP level has the single greatest impact on the estimated hazard ratio for overall survival. Indeed, adjusting for AFP alone generates a significant treatment effect in favour of lenvatinib at conventional statistical significance levels with an estimated overall survival hazard of 0.856 with 95% CI ranging from 0.736 to 0.995.
A parsimonious multivariable model is reported in Table 2 using a forward stepwise procedure. Model 1 is the standard model that employs a Wald test for significance. Model 2 represents a sensitivity analysis where the LR test is used instead of the Wald test. As can be seen in Table 2, the final variable selection is identical. In the final column in Table 2, the AIC analysis shows that each successive covariate included in the model further reduces the AIC (i.e. all covariates contribute to the model even when penalising for the use of additional parameters). The confluence of these approaches suggests a strong basis for the multivariable adjusted hazard ratio for lenvatinib overall survival of 0.814 (95% CI: 0.699–0.948).
A sensitivity analysis is shown in Table 3 for including post-treatment variables into the model. Model 1 includes the composite post-treatment “therapy” variable and Model 2 shows this split out as post-treatment procedures and post-treatment medications. Both models show that the post-treatment anti-cancer variables are highly significant predictors of outcome. In addition, adjusting for these post-randomisation imbalances impacts the hazard ratio of the treatment arms. Comparing the AIC scores for non-nested models shows support for Model 2 compared with Model 1 despite the additional parameter. This gives an adjusted hazard ratio for lenvatinib overall survival of 0.765 (95% CI: 0.656–0.892).
As an additional sensitivity analysis, the forwards stepwise procedure was repeated as a backwards procedure (Table 4). Here, Model 1 relates to baseline variables only and Model 2 includes post-treatment variables. The backwards selection procedure identifies slightly different adjustment variables than the forward selection process, but the main effects of the model are similar with an adjusted hazard ratio for lenvatinib of 0.814 (95% CI: 0.699–0.948). Similarly, the addition of post-treatment procedure and medication variables gives a virtually identical hazard ratio to the forward selection model: 0.765 (95% CI: 0.656–0.892).
This study has reanalysed the REFLECT noninferiority study that compared lenvatinib and sorafenib for the treatment of uHCC in a multivariable modelling framework. The initial study’s strong positive result in favour of lenvatinib for the secondary end points of progression-free survival and objective response rate, the imbalance of baseline covariates favouring sorafenib, and the greater use of subsequent anti-cancer therapies after progression in the sorafenib arm, all suggested that lenvatinib could be superior to sorafenib in terms of overall survival. Our covariate-adjusted results indicate that lenvatinib treatment of uHCC may reach superiority for overall survival vs. sorafenib at conventional levels of statistical significance once baseline imbalances in important prognostic variables are controlled for. This effect is magnified when further adjustment is made for post-treatment therapy variables.
However, we remain cautious when it comes to interpretation of this analysis. This is a post-hoc analysis and cannot change the results of the original trial, though it allows us to better understand them. In particular, adjusting for post-treatment variables violates the randomisation principle and could, itself, lead to bias. More sophisticated techniques and further post-randomisation data on the reasons for receiving the different post-progression therapies would be needed to perform a full causal analysis, adjusting for imbalances in line therapies.8,9
By contrast, the use of covariate analysis in randomised controlled studies is far less contentious. Although not permitted in regulatory analyses designed to support licensing applications, the use of covariate adjustment is recommended by leading medical statisticians.10,11 This is because covariate adjustment can improve the analysis both in terms of correcting for the potential bias of imbalance in prognostic variables and also in terms of increasing the precision of estimated treatment effects. The latter effect occurs even when there is no imbalance if important prognostic variables are available because the multivariable model uses the covariate information to explain some of the “noise” in the data, increasing the precision of all estimated quantities. Note that it is the strength of the prognostic power of the imbalanced variable, rather than the statistical significance of the imbalance that is the issue. Leading medical journals, including The Lancet, where the original clinical trial was published,1 discourage the use of p-values to compare the balance of the randomised groups. Assuming appropriate randomisation, any observed imbalance occurs by chance and so arbitrary p-values are not required to make this judgement. Further, it is the prognostic importance of a given variable that determines the importance of any imbalance. This is illustrated in the analysis presented here by noting that adjusting for the imbalance of AFP level had the greatest single effect on the estimated hazard ratio for lenvatinib treatment. However, statistical testing of the imbalance results in a marginally insignificant difference between the groups in Table 1 (p-values not reported).
Indeed, a recent issue of The American Statistician is heralding the end of the p-value and significance testing.12 While this may be premature for appropriately conducted, randomised, controlled trials designed to test a specific hypothesis, the principles do apply to secondary analyses such as presented in this manuscript. For this reason, we focus throughout on the estimated treatment effect with accompanying CIs.
Covariate adjustment of the REFLECT data strongly suggests that the original noninferiority trial likely underestimated the true effect of lenvatinib on overall survival due to imbalances in baseline prognostic covariates (in particular, AFP level) and the comparatively greater use of post-treatment anti-cancer therapies in the sorafenib arm. While the scale of the impact of covariate adjustment on treatment effect varies, the analyses reported here all favoured lenvatinib. Considering the potential biases associated with adjusting for post-randomisation variables, the preferred base-case hazard ratio is that based on adjusting for baseline covariates only.
Kudo, M., Finn, R. S., Qin, S., Han, K.-H., Ikeda, K., Piscaglia, F. et al. Lenvatinib versus sorafenib in first-line treatment of patients with unresectable hepatocellular carcinoma: a randomised phase 3 non-inferiority trial. Lancet 391, 1163–1173 (2018).
Etickan, I., Musa, S. A. & Alkassim, R. S. Comparison of convenience sampling and purposive sampling. Am. J. Theor. Appl. Stat. 5, 1–4 (2016).
Burnham, K. P. & Anderson, D. R. Multimodel inference: understanding AIC and BIC in model selection. Sociol. Methods Res. 33, 261–304 (2004).
StataCorp. Stata Statistical Software: Release 14. (StataCorp LP, College Station, 2015).
Mazzaferro, V., Sposito, C., Zhou, J., Pinna, A. D., De Carlis, L., Fan, J. et al. Metroticket 2.0 model for analysis of competing risks of death after liver transplantation for hepatocellular carcinoma. Gastroenterology 154, 128–139 (2018).
Broadhurst, P., Toyoda, H., Kumada, T., Schweitzer, N., Vogel, A., Benckert, J. et al. Prognostic impact of serum alpha-fetoprotein in patients with hepatocellular carcinoma: an international collaborative study. J. Hepatol. 66, S620–S621 (2017).
Bruix, J., Cheng, A. L., Meinhardt, G., Nakajima, K., De Sanctis, Y. & Llovet, J. Prognostic factors and predictors of sorafenib benefit in patients with hepatocellular carcinoma: analysis of two phase III studies. J. Hepatol. 67, 999–1008 (2017).
Robins, J. M., Hernán, M., Siebert, U. Estimations of the Effects of Multiple Interventions. In: Ezzati, M. L. A., Rodgers, A., Mrray, C. J. L., editors. Comparative Quantification of Health Risks: Global and Regional Burden of Disease Attributable to Selected Major Risk Factors. 1. p. 2191–2230 (World Health Organization, Geneva, 2004).
Hernán, M. & Robins, J. M. Per-protocol analyses of pragmatic trials. N. Engl. J. Med. 337, 1391–1398 (2017).
Altman, D. G. Comparability of randomised groups. J. R. Stat. Soc. Ser. D. (Statistician) 34, 125–136 (1985).
Pocock, S. Clinical Trials: A Practical Approach (Chichester, Wiley, 1983).
Wasserstein, R. L., Schirm, A. L. & Lazar, N. A. Moving to a world beyond “p < 0.05”. Am. Statistician 73(sup1), 1–19 (2019).
Ethics approval and consent to participate
This is a reanalysis of existing data from the REFLECT study. All patients provided written informed consent to participate in the study as detailed in the original clinical trial manuscript (Kudo 2018). This study was conducted in accordance with the Declaration of Helsinki.
Consent to publish
No individual person’s data are used in this manuscript in any form, so no consent for publication was required.
Data are held by the sponsors: Eisai Inc., and Merck Sharp & Dohme Corp., a subsidiary of Merck & Co., Inc., to whom all requests for data access should be addressed.
Sponsorship for the study and analysis were funded by Eisai Inc., and Merck Sharp & Dohme Corporation, a subsidiary of Merck & Co., Inc., and study results were not contingent on the sponsors’ approvals nor did they conduct any censorship of the manuscript. A.B. is a director and shareholder of Avalon Health Economics and received compensation from Eisai as a consultant for this work. He has also been contracted by Bayer, Merck, Janssen, Novartis, Sword Health, Amgen, and Daichii Sankyo and received compensation outside of the submitted work. B.D. reports no conflicts of interest regarding the work at hand but has received personal fees and non-financial support for consultation for Ispen, Sanofi, and Bayer, and personal fees from Eisai, Eli Lilly, Astra Zeneca, and Incyte outside of the submitted work. K.D. is an employee of Avalon Health Economics. T.R.J.E. has received honoraria for advisory boards and/or speaker’s fees from Eisai, Celgene, Bristol-Myers Squibb, Bayer, Roche, Merck (MSD), Nucana, Karus Therapeutics, Modulate Therapeutics (all payable to the employing institution), research funding for clinical trials from multiple pharmaceutical and biotechnology companies including Eisai (all payable to the employing institution), and support to attend international scientific conferences from Bristol-Myers Squibb, Merck, Eisai, Bayer, Roche, Pierre Fabre, and Celgene. He is co-editor of the clinical study section of the British Journal of Cancer (honorarium payable to the employing institution). P.R.G. has received compensation from Eisai for this work and for adboards and lectures from Bayer, AstraZeneca, BMS, MSD, Roche, Sirtex, Ispen, and Lilly outside of the submitted work. R.A.H. received consultancy fees from Eisai for this work. He has also received compensation from Roche and Ispen and attended conferences for both Roche and BMS outside of the submitted work. C.L. has been paid honoraria for consultancy on this work. He has also received compensation for work from Ispen, Bayer, BMS, and Astra Zeneca, and has participated in a speaker’s bureau for Eisai, Ispen, Bayer, and BMS. Further, he has received research funding from Eisai, Ispen, BMS, MSD, Astra Zeneca, and Roche. U.S. has received honorarium for participating in an advisory board of Eisai and reports no other conflicts of interest. G.T. reports no conflicts of interest regarding the submitted work.
This study was funded by Eisai Inc., Woodcliff Lake, NJ, USA, and Merck Sharp & Dohme Corporation, a subsidiary of Merck & Co., Inc., Kenilworth, NJ, USA.
Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
About this article
Cite this article
Briggs, A., Daniele, B., Dick, K. et al. Covariate-adjusted analysis of the Phase 3 REFLECT study of lenvatinib versus sorafenib in the treatment of unresectable hepatocellular carcinoma. Br J Cancer 122, 1754–1759 (2020). https://doi.org/10.1038/s41416-020-0817-7