Poor reporting may infer poor science: lessons learned from asthma trials

High-quality studies are generally considered the best directors of medical decision-making and policy development. Although the randomised controlled trial (RCT) is the gold standard for intervention research, flaws in methodology and trial processes can severely affect the validity of its findings. Assessment of trial validity can be further hampered by poor reporting of its design and findings. The Consolidated Standards of Reporting Trials (CONSORT) Statement was developed and subsequently updated in an attempt to increase insight into the validity of RCTs by providing guidelines for their reporting.¹ CONSORT provides a checklist of key items that should be reported when publishing trial results, such as patient eligibility, randomisation procedure, and blinding.²

In this issue of the PCRJ, Ntala and colleagues report their findings of a systematic review of reporting quality of RCTs conducted in patients with asthma.³ The percentage of adequately reported CONSORT items varied between 39 and 91% among the 35 included studies. Less than half of the studies adequately reported over 60% of the items. Since the authors only included studies from high impact journals, overall reporting quality of asthma trials is likely to be lower.⁴ Given that the authors further restricted their search to one medical database, to a 1.5 year period, and to asthma trials published in journals from two subject fields, the findings should be interpreted as a case-study and therefore generalisability is limited.

Therefore, several questions remain. Is this reporting problem specific to asthma studies? Is the problem specific to respiratory journals? And has there been any progress in reporting quality over time?

The largest and most recent systematic review on this topic showed evidence of some improvement in reporting from studies assessed using the 1996 CONSORT statement to those assessed using the 2001 version.⁵ Completeness of reporting between this previous review and the current paper by Ntala and colleagues is roughly comparable, although considerable variation exists across individual items (see Table 1).^3,5 Whereas this may seem to suggest that no further improvement in reporting occurred, an updated systematic review of the contemporaneous evidence base is required to assess overall progress in reporting quality. Comparison with recent systematic reviews of trial reporting quality in other subject fields identified via an explorative electronic literature search quickly reveals that the problem is not confined to asthma studies or respiratory journals (Table 1).^4,6–8 Although this demonstrates that we are in good company, the extent of the matter is worrying.

Table 1 Comparison of CONSORT reporting quality in recent systematic reviews

Ntala et al.³, as well as others, provide some guidance on how to advance from here, and how to tackle this at different levels. There is evidence that endorsement of CONSORT by medical journals improves the reporting of at least a number of checklist items (Table 1).⁵ This suggests that there is a clear role for medical journal editors, not only in endorsing but also enforcing compliance of submitted manuscripts with the CONSORT checklist.⁹ Authors should be requested to complete a checklist during the submission process, which should be checked by editors and reviewers. Furthermore, there is a particular need for greater adoption of CONSORT in low and middle income countries, as demonstrated by its limited endorsement by medical journals from those countries^10,11 and the association between location of the corresponding author in a low or middle income country and lower reporting quality in the current study.³

However, we believe that the main responsibility lies with the authors of RCT reports. They should recognise that readers and reviewers will often consider reporting quality as a surrogate for methodological study quality: poor reporting thus infers poor science.^12,13 Although evidence suggests that this is likely to be a misjudgement,^14,15 it can easily be prevented by better reporting.¹² When preparing a manuscript, authors should therefore make every effort to comply with the reporting guidelines that are available for their particular study design.¹⁶ Given the reasonable chance for any research study to be included in a systematic review, authors should also familiarise themselves with commonly used quality appraisal tools, such as the Cochrane risk of bias tool applied here by Ntala and colleagues.³ Better reporting according to such checklists can contribute to higher quality ranking of a study.^12,13 It will also serve more accurate appraisal of the available body of evidence on a subject, benefitting subsequent clinical guideline development and policy making.

Whereas historically journal space constraints may have restricted the completeness of reporting, Ntala and colleagues argue that many details can now be reported easily in an online supplement.³ Publication and adequate referral to a full trial protocol, as done by only 17% of studies included in their review, can supply additional essential information.³ Ideally, full protocol access should be provided at the time of entry into a trial registry, which can nowadays be considered a prerequisite for eventual publication.⁴ Awaiting further improvement in reporting quality, authors of systematic reviews should consider contacting authors of included studies in an effort to prevent undue interpretation of unreported items as methodological shortcomings.¹³

In conclusion, there is considerable room for improvement in trial reporting quality. Journals should endorse reporting guidelines including CONSORT and specify how compliance is to be enforced. On this specific point, for example, the PCRJ can improve the detail in its ‘Guide for authors’ regarding how inadequate reporting is to be handled.¹⁷ Authors should ensure compliance of their manuscript with relevant guidelines, and editors and reviewers need to involve the corresponding checklists in their assessment of submitted work. The scientific community has a common responsibility to enable judgment of the clinical applicability of trials to be based on their actual merits rather than reporting adequacy.