Comment on ‘BAG-1 as a biomarker in early breast cancer prognosis: a systematic review with meta-analyses'

Prognostic biomarkers play an important role in medical research, assessment of prognosis and treatment of patients. Validation of a potential biomarker takes a long evidencebased approach requiring several individual studies, a systematic review of the literature and a meta-analysis. Papadakis et al. conducted a systematic review with meta-analysis to assess BAG-1 as an effective prognostic biomarker in early breast cancer. They concluded ‘Meta-analyses suggested improved outcome with high BAG-1 [...] expression’. They identified 18 papers providing results from 20 studies (Table 2 in Papadakis et al.), assessed the quality of reporting according to the REMARK guidelines, 4 summarised key results of each study (Table 2 in Papadakis et al.) and conducted three meta-analyses (Figure 2 in Papadakis et al.). This study illustrates key steps required for an evidence-based biomarker assessment; however, we have identified several major weaknesses in the assessment of the quality of reporting and the meta-analyses. We concluded that results and inferences from this study are not justified by the assessments and analyses presented.

Prognostic biomarkers play an important role in medical research, assessment of prognosis and treatment of patients. Validation of a potential biomarker takes a long evidencebased approach requiring several individual studies, a systematic review of the literature and a meta-analysis. 1 Papadakis et al. 2 conducted a systematic review with meta-analysis to assess BAG-1 as an effective prognostic biomarker in early breast cancer. They concluded 'Meta-analyses suggested improved outcome with high BAG-1 […] expression'. They identified 18 papers providing results from 20 studies ( Table 2 in Papadakis et al. 2 ), assessed the quality of reporting according to the REMARK guidelines, 3,4 summarised key results of each study ( Table 2 in Papadakis et al. 2 ) and conducted three meta-analyses ( Figure 2 in Papadakis et al. 2 ). This study illustrates key steps required for an evidence-based biomarker assessment; however, we have identified several major weaknesses in the assessment of the quality of reporting and the meta-analyses. We concluded that results and inferences from this study are not justified by the assessments and analyses presented.

ASSESSMENT OF THE QUALITY OF REPORTING ACCORDING TO REMARK
According to Table 1 in Papadakis et al. 2 , reporting of the studies was excellent, strongly contradicting the results of a recent review. 5 As an example, we consider the rationale for sample size (Item 9) that was positively assessed in all evaluated studies, whereas it was assessed as being adequately reported in only 22, 11 and 8% in the three (sub-)studies (each including about 50 articles), as summarised in Sekula et al. 5  As all studies consider a survival-time outcome (time to event analysis), it is necessary to provide the number of events as a measure of the effective sample size. 3 Under the term 'Study power' (Item 9), the sample size is listed but not the number of events.
Item 16 of the REMARK guidelines states that 'For key multivariable analyses, report estimated effects (for example, hazard ratio) with confidence intervals for the marker and, at least for the final model, all other variables in the model.' Sekula et al. 5 found acceptable reporting in 70, 66 and 62% of studies assessed. Except for one study, all BAG-1 studies were positively assessed. However, in the manuscript by Papadakis et al 2 , several studies reported only p values 8,9 or otherwise indicated non-significance 10 . Significant shortcomings in reporting the primary studies are also found in overviews and hamper any sensible assessments of a marker's effect. Instead of the positive assessments of reporting quality of the primary studies, Papadakis et al. 2 should have clearly pointed out the weaknesses. How can a study be positively assessed if it mentions a significant effect in univariate analysis, shows a Kaplan-Meier plot illustrating large survival differences (Figures 4a, 4c in Athanassiadou et al. 6 ) without providing an estimate of the effect? Furthermore, in multivariable analysis, the effect seems to be non-significant. .' The first sentence implies that a sensible meta-analysis is not possible, an unfortunate situation seen in many systematic reviews of prognostic biomarkers. Nevertheless, Papadakis et al. 2 conducted three meta-analyses summarising two or three studies. It is seriously misleading to pick out a small number of studies and pretend to conduct a meta-analysis. In addition, they averaged effects for classification of BAG-1 as 'positive' vs 'negative', but criteria for this classification varied substantially (positivity was defined as 'H-Score >100,' 10 and 'dichotomised at a cutoff value of 40% positively staining nuclei at any intensity' 7 (see also Papadakis et al 2 )). They also combined hazard ratios from univariate and multivariable analyses (Figure 2c in Papadakis et al. 2 ).

META-ANALYSIS
In three meta-analyses, the authors consider two methods of measuring BAG-1 and two outcomes. Of the 18 papers identified, 14 are ignored in these 'meta-analyses'. Can that be described as a meta-analysis? Because associations of a biomarker with outcome might be heavily influenced by other prognostic variables correlated with the biomarker, such univariate analyses can be seriously misleading 3 (Item 15). An appropriate analysis of a single study requires a multivariable model in which the effect of interest is adjusted for potential confounders. Consequently, a suitable meta-analysis requires averaging of adjusted effects that can be combined meaningfully. Collaboration among several study groups and individual participant data are required to make populations 'broadly' comparable and to reanalyse individual studies so as to give adjusted effects that can be validly combined. 1,12 We are well aware of the serious difficulties www.nature.com/bjc encountered in conducting such a project. However, many researchers and collaborative groups have realised their necessity and IPD meta-analyses of prognostic factors have become more popular during the past decade. 13

PUBLICATION BIAS AND THE NEED FOR A COMPREHENSIVE BIOMARKER STUDY REGISTRY
Publication bias distorts the estimate of the true value of biomarkers. This important issue needs to be considered in studies trying to assess the value. 1,12,14 Unfortunately, Papadakis et al. 2 do not even mention this issue. As for randomised trials, several researchers argue for establishing a comprehensive biomarker study registry 12,15,16 .
In conclusion, in light of previous experience, we are not surprised to see so many weaknesses in reports of individual studies and in meta-analyses. The complexity of such projects is often underrated and the importance of including an experienced methodologist is still not recognised sufficiently. In addition, evidence-based assessments require substantial collaboration among researchers, including willingness to share data. The paper and the situation criticised for BAG-1 in breast cancer is not much different from the situation today for many other prognostic biomarkers. Fortunately, many researchers understand the difficulties, and several promising projects have been started, e.g., the PROGRESS partnership, which provides a framework to improve prognostic factor research (progress-partnership.org). We hope that by writing this letter, we can lessen the chance that others will make basic errors in future studies and point researchers to key steps needed when assessing the prognostic value of a biomarker. As biomarkers become more and more important tools in medical decision-making, invalid results can have serious consequences for patients.

ADDITIONAL INFORMATION
Competing interests: The authors declare no competing interests.
Note: This work is published under the standard license to publish agreement. After 12 months the work will become freely available and the license terms will switch to a Creative Commons Attribution 4.0 International licence (CC BY 4.0).