Introduction

The randomised controlled trial (RCT) is the most robust design for assessing the efficacy and effectiveness of treatments.1 As a result, clinical decision-making has over decades been directed away from reliance based solely on the doctor's clinical experience towards a paradigm based on evidence derived from RCTs. The results of RCTs have subsequently been translated into guidelines containing evidence-graded recommendations which clinicians are encouraged to use as the basis of good clinical practice.2 If, however, the ‘raw material’ or trial is flawed, the conclusions cannot be trusted, hence the need to appraise critically the quality of the underpinning trial evidence.3

Quality is a multidimensional concept which relates to the design, conduct, and analysis of a trial, its clinical relevance, and its reporting.3 In most cases the RCT report is the only source for clinicians, guideline developers, and other researchers to judge the validity and generalisability of the results, so the quality of reporting of trials is of inherent interest.4 It is then of considerable concern that the quality of reporting of RCTs is often suboptimal.5

In response to these concerns about the quality of reporting of RCTs, an international group developed in the mid-1990s the Consolidated Standards of Reporting Trials (CONSORT) Statement.6 This was first published in 1996 and then updated in 2001.5 After an expert meeting in January 2007 it was further revised, resulting in the latest iteration — the CONSORT 2010 Statement.5

The current CONSORT Statement (hereafter referred to as CONSORT) comprises a checklist of essential items that should be included in reports of RCTs and a diagram for documenting the flow of participants through a trial. It is formulated primarily for use with reports of two-arm parallel-group RCTs. Many of the CONSORT data fields are, however, also relevant to a wider spectrum of trial designs such as non-inferiority, equivalence, factorial, cluster, and crossover trials.5 Extensions to the CONSORT checklist for reporting trials with some of these other designs have been published,7–9 as have those for reporting particular types of data (i.e. harms10), types of interventions (i.e. non-pharmacological treatments11 and herbal interventions12), and abstracts.13 In this systematic review we have, where appropriate, used the non-pharmacological, non-inferiority and equivalence, cluster and pragmatic extensions of CONSORT.

CONSORT criteria have been used to assess the reporting quality of RCTs in several disease areas4,14,15 and journal types.16–18 Asthma is a serious public health problem throughout the world and therefore a substantial body of research is conducted annually.19 However, there have been no recent assessments of the quality of RCTs reported in the asthma literature. The only previous study on clinical trials of asthma treatments was undertaken for the period 1984–1997 and was published in 2002 in two reports.20,21 Initially this involved a comparison between RCTs published in Spanish and English language journals,21 which was then followed by a secondary analysis of a subsection of the same dataset focusing solely on the quality of RCTs in English.20 The first article showed poorer reporting quality of the RCTs in Spanish publications and a strong association between the type of journal, type of intervention, and the comparison measure used and reporting quality. Moreover, this study highlighted the necessity for better reporting in general in the asthma literature, leading authors to advocate the more widespread use of a checklist by authors and editors in order to improve reporting standards.21

Building on this earlier work, we examined the quality of reporting of asthma clinical RCTs in the contemporary asthma literature. Our secondary aim was to investigate if there is an association between specific trial characteristics that have previously been identified in the literature in influencing reporting quality and the actual quality of the trial reports. We hypothesised that trials published in general medicine journals with high impact factors (IF) conducted or led by teams in high-income countries with industrial funding, evaluating drug interventions, and having multiple participating centres were associated with better quality.

Methods

Study selection

We searched the electronic database MEDLINE (via Ovid) using the search terms of the Cochrane Airways Group Specialised Register for asthma and RCTs for the period from January 2010 to July 2012. We included studies published in the top 10 impact factor journals in general medicine and respiratory specialty journals using the most recent available rankings at the time (i.e. 2011 rankings, Table 1),22 as long as they published clinical trials and included articles related to asthma. We included parallel or cluster design RCTs involving human subjects with asthma as the only condition being examined. The trials were evaluating the clinical effectiveness of a treatment, which was defined as having at least one clinical outcome, irrespective of whether this was a primary or secondary outcome. We excluded any condition not described strictly as asthma (e.g. recurrent wheezing), even when it was a broader term that included it. The complete search strategy and detailed inclusion/exclusion criteria were specified in advance and documented in a protocol.23

Table 1 Generalist and specialist journals and number of asthma trials identified
Full size table

Searches were independently undertaken by two reviewers (CN and PB) with support from AW and AS. Both reviewers independently screened the titles and abstracts of all retrieved articles, unmasked to study details. Full-text copies of potentially relevant studies were obtained and assessed for inclusion. Any disagreements were resolved by discussion between the reviewers; if necessary AS and AW arbitrated. Any additional material of the reports included as appendices were also obtained from the journal website or were acquired from the authors.

Data extraction

Data were independently extracted by two reviewers (CN and PB) from the selected studies using an appropriate electronic customised data extraction form23 that was previously pilot-tested on seven randomly selected studies and refined accordingly. Disagreements were resolved through discussion with AS. The following data on the general characteristics of trials were extracted: journal name, journal type (general medicine or specialty), IF, country of the team that conducted or led the study according to the corresponding author in case of multiple centres (high-income, middle income, low-income) defined using World Bank Group definitions,24 funding source (solely industry, partly industry, non-industry, unknown), trial design (parallel or cluster), conceptual framework (superiority, non-inferiority, equivalence), type of intervention (drug or non-pharmacological), number of participating centres (multiple or single centre).

We used a modified 38-item CONSORT-based checklist consisting of all the CONSORT checklist items plus one additional item from the non-pharmacological treatments extension.23 We assessed the adequacy of reporting according to the CONSORT 2010 guidelines and its extensions.5,8,9,11,12 Each item was characterised as ‘yes’ if it was clearly and adequately reported or ‘no’ if it was partially reported, unclear, or not reported at all. Each ‘yes’ answer received a score of 1 and each ‘no’ answer was scored as 0. The overall quality scoring of the trial was calculated as a proportion of the ‘yes’ rated applicable items (possible range 0–38 points). In addition, we scored the overall quality of reporting using key parameters of internal validity summarised in the Cochrane Risk of Bias (RoB) tool25 and categorised the studies into those at low to moderate risk of bias and those at high risk of bias.

Data analysis

We calculated the proportion of trials that clearly and adequately reported each CONSORT item with a 95% confidence interval (CI). An overall quality score was also calculated for each trial as a percentage of all the adequately reported applicable items with a 95% CI, which was used to inform a global assessment of the quality of reporting. The general characteristics data were presented as numbers and percentages with 95% CIs. For the CI calculations we used the Wilson (recommended) method26 of Confidence Interval Analysis (CIA) software (Version 2.2.0), this being preferred to the more commonly used Normal approximation method to calculate CIs as it has more reliable behaviour with small samples (i.e. it avoids limits crossing 0 or 1).

We then tested the associations between the following study characteristics: type and IF of journal of publication, funding source, country of the study, type of intervention, and number of participating centres and the quality of RCT (measured by the RoB tool and categorising the trials as low versus moderate to high RoB). Fisher's exact test and the Mann-Whitney test for categorical and continuous data, respectively, were used to identify variables associated with studies with a low to moderate risk of bias using SPSS software (version 20). The IF was evaluated as the median difference between IF in the low to moderate RoB group of trials minus the IF in the high RoB group. This was run in MINITAB.

Further details of our methods are reported in the study protocol paper.23

Results

Study characteristics

Our searches identified a total of 657 citations, 607 of which remained after adjusting for duplicates. After screening the titles and abstracts, we examined 52 full-text study reports and identified 35 articles that met our inclusion criteria and comprised our study sample (see Figure 1, and Appendix 1 available online at www.thepcrj.org). The inter-rater κ agreement for article selection was 0.64. Table 2 shows the general characteristics of the trials. The majority of RCTs were published in specialty journal types, and the vast majority of the trials were conducted or led by investigators based in high-income countries. More than half of the RCTs were evaluating drugs, and the most frequent design and conceptual framework were parallel and superiority trials, respectively. Over half (57.2%) of trials received industrial support.

Figure 1
figure 1

PRISMA flow chart

Full size image
Table 2 General characteristics of included randomised controlled trials
Full size table

Assessment of reporting quality

Table 3 provides information on the reporting of each CONSORT item. Seventeen of the 38 CONSORT items we examined were consistently well reported in more than two-thirds of the articles; these included the scientific background, specific objectives, outcomes, statistical methods, participant flow, baseline data, numbers analysed, limitations, generalisability, interpretation, and trial registration. In contrast, the following nine items were poorly reported in more than half of the trials: identification as a randomised trial in the title, structured summary, eligibility criteria, implementation of randomisation, implementation of intervention, recruitment, harms, access to full trial protocol, and funding.

Table 3 Number and percentage with 95% confidence interval (CI) of adequately reported items
Full size table

Table 4 presents an overall quality score for each trial as a global assessment of the quality of reporting. Four trials adequately reported <50% of the items, 15 trials adequately reported 50–60% of items, and 16 adequately reported >60% of items.

Table 4 Summary of numbers, percentages and 95% confidence intervals (CIs) of adequately reported items in each trial
Full size table

Factors associated with better reporting quality

We found that studies conducted or led by teams in high-income country settings were associated with better quality trials (Table 5). We did not identify any statistically significant associations between the other characteristics evaluated and the quality of trial reports (Table 5).

Table 5 Relative risks (RR) and 95% confidence interval (CI) of low to moderate risk of bias (RoB)/good quality trials
Full size table

Discussion

Main findings

This work has found that the overall quality of reporting of contemporary asthma trials was suboptimal in the majority of cases. The reporting of each separate item varied significantly. Access to the full trial protocol was the least well reported item, present in only approximately one in five trials. The analysis of factors associated with better quality revealed a significant relationship between the studies that were conducted or led by teams in high-income country settings.

Interpretation of findings in relation to previously published work

We identified certain trial parameters that were particularly inadequately reported, this finding being largely in common with previous reviews. The identification as a randomised trial in the title is very commonly omitted.17,27,28 A possible explanation for this could lie in the limitations on title length imposed by some journals and the related issue of restricting abbreviations (i.e. RCT) in the titles of papers. The presence of an adequately detailed structured summary/abstract was uncommon, but is generally variable in the literature.17,29 We did not use the CONSORT extension for abstracts,13 but nonetheless we believe that its rigorous enforcement by journal editors with a completed CONSORT abstract checklist requirement could help reduce this reporting deficit. The finding of poorly reported eligibility criteria is in contrast with what has been found in the literature,14,17,30 since the clinical features of the trials are usually better reported than the trial methodology and statistical features.14,31 This finding was principally due to the lack of reporting the methods of recruitment (which is subsumed under this heading), not the eligibility criteria. It may therefore be appropriate to adapt CONSORT in future iterations to encourage description of the method of recruitment as a separate item.

The implementation of randomisation also needs to be better reported, a finding in keeping with other reports.4,15,17,27 This finding may result from the perception that the clinical aspects of a trial are of greater importance than the methodology, which results in a deliberate or subconscious de-emphasis of data about the methodological aspects of RCTs. The description of non-pharmacological interventions also needs to be reported better. This limitation may have been partly due to the fact that we did not examine information published in previous abstracts, other publications, protocols, or contact authors for further information. It is common for non-pharmacological trials not to report fully all the details of an intervention, but the necessity for a complete reproducible description of the intervention and its implementation has been highlighted before.32 This key reporting element, which can otherwise take up a considerable proportion of the limited word count, could appear in electronic appendices of journal websites linked to the original publication.

The exact dates of recruitment and follow-up were also found to be poorly reported, in accordance with previous reviews.4,15 In an evaluation of safety reporting in randomised trials across seven different medical areas in 2001 it was noted that safety reporting was often inadequate and/or neglected. Key information that would take minimal space to report was often missing.33 According to our findings, the reporting of harms remains inadequate. We did not use the CONSORT extension for this item, but we suggest that authors should become more familiar with this CONSORT extension and follow its recommendations.10

Protocols still serve as a detailed guide for a wide audience to research in progress and may help contribute to reduced duplication of effort by others who are planning to do similar research. They also minimise the likelihood of reviewers introducing bias into their review by making major changes that might otherwise remain undisclosed and undetectable. The publication and access to both protocols and completed reviews allows the tracking of any changes that have taken place and an examination of what effect this may have had on the results of the review.34 Unfortunately, from our findings, access to the full trial protocol is still limited. Other reviews have also noted similar findings27,28 and, in fact, one did not identify a single RCT with access to the protocol.17 This may be partly explained by the previously limited opportunity to include online material with the manuscript or the previous availability of a journal for protocol publication. Nowadays such journals are available; in fact, some require access to the full study protocol for the submission of an RCT. On the downside, they are primarily open access and may be expensive, especially for researchers in low- and middle-income country settings.

Finally, the suboptimal reporting of sources of funding is also a finding that is common to other similar reviews.28 This is important since full financial disclosure reduces the suspicion that something of relevance to objectivity is being hidden and allows readers to form their own opinions on whether a conflict of interest exists and what relevance that has to the study.35 Journals should be more specific in their instructions to authors on the types of financial associations related to their submission that warrant disclosure.35

In contrast, 13 items in our study seem to have a high level of compliance; more than 70% of the trials have adequately reported certain items, including issues relating to internal validity (participant flow and numbers analysed) and external validity (interventions, definition of outcome measures and baseline data).

We encountered a lot of discrepancies in evaluating the limitations section of the discussion so we needed a way to operationalise this. If we did not identify possible bias according to the RoB tool, we considered that sources of potential bias were not present. In other reviews the discussion items were excluded as they were considered too subjective to evaluate.16,30

Strengths and limitations of this study

Our study has certain strengths. It is the only contemporary evaluation on the quality of reporting of RCTs in asthma and was undertaken following development and publication of a detailed protocol. The selection and extraction processes were independently undertaken by two reviewers with a robust strategy to solve disagreements between reviewers. Analysis involved the testing of a priori clearly stated hypotheses.

It also has several potential limitations. We used the CONSORT checklist to assess the quality of reporting and, in doing so, we weighted all items equally, although the relative importance of each might be different. In the absence of any suitable alternative validated quality assessment instrument,1 we used the RoB tool in order to measure the quality of the RCT and to investigate our secondary hypotheses. However, in these evaluations we did not measure the RCT methodological quality directly because we did not verify the information from the authors or their protocols. One observational study found that 52 of 54 trials with unclear allocation concealment in the trial publication were adequately concealed according to a pre-announced telephone interview of the investigators.36 Thus, when authors of RCTs do not report concealment and blinding, this does not necessarily mean that the studies were unconcealed.36 However, two observational studies that compared the content of study reports with the design features described in the protocols of RCTs had contradictory findings. One study found that many trials with unclear allocation concealment in the article also had an unclear description in the protocol.37 The other concluded that the reporting of methodological aspects of RCTs does not necessarily reflect the conduct of the trial, identifying the use of methodological safeguards despite not explicitly reporting them.38 Therefore, the quality of reporting can be taken only as an imperfect surrogate of true methodological quality.39 Nevertheless, because the report is usually the only source for clinicians and other researchers to judge the validity and generalisability of the results, the quality of the report has an important value by itself. In addition, poor reporting of RCT findings can result in overestimation of treatment effect and may potentially lead to erroneous conclusions.40,41

Our study sample was small and this may have contributed to our inability to identify some potentially important predictors of the quality of trial reporting. This is due to the limited time period that we covered, the restriction to trials published in only 20 high IF journals, and the strict definition of asthma that we used (excluding broader conditions that might include it such as recurrent wheezing). These restrictions may also affect the generalisability of our results to the rest of the literature on asthma. In addition, our study selection exclusively from top journals may overestimate the quality of asthma RCTs in general. The small study sample along with the small number of low RoB studies (two low RoB studies, nine moderate RoB studies, and 27 high RoB studies) forced us to combine low with moderate RoB studies even though this is not recommended.25 This represented a deviation from our original plan.23

Other reviews have identified the following factors associated with better quality: the time of publication, involvement of a methodologist/epidemiologist in a study, endorsement of CONSORT by the journal in which the study is published, size of the trial, and positive trial outcomes.4,14,16 Testing these additional hypotheses was, however, beyond the scope of this study. The exclusion of crossover and factorial designs trials is also a limitation. We made this decision because the 2010 CONSORT Statement is only intended for RCTs with a parallel-group design5 and so far there is only an extension for cluster trials.8

Implications for future research

Given the persistent problems with suboptimal reporting, it will be important in due course to take a further look at asthma trial data to see if improvements have been made. It would also be helpful to study whether more uniform implementation of CONSORT by journal editors can help improve the quality of reporting of this literature. Finally, future studies that aim to study factors associated with better quality need to draw on our data, which can be used to inform sample size deliberations about the volume of literature that needs to be investigated.

Conclusions

Our results suggest that the quality of reporting in the contemporary literature on asthma remains suboptimal. We have identified deficiencies in reporting specific areas that need to be improved. All of the CONSORT items that we examined were adequately described in at least one article, which indicates that improved reporting is certainly possible. Improved awareness of the CONSORT statement by researchers, reviewers, and journal editors is likely to improve reporting quality.