Introduction

The future of a potential antidepressant is highly dependent on whether the difference between active drug and placebo with respect to the primary effect parameter reaches significance at the P<0.05-level in at least two independent trials or not.1 Also, while the existence of negative trials, in addition to those showing a difference, does not prevent a drug from being approved for marketing, the outcome in terms of significance versus non-significance of all trials undertaken may impact how drugs already on the market are being valued; in this vein, the current questioning of the efficacy of the selective serotonin reuptake inhibitors (SSRIs) has to a great extent been spurred by the fact that approximately half of the placebo-controlled SSRI trials conducted by the pharmaceutical companies have failed to show significant superiority of the active drug over placebo.2 According to critics, the fact that these negative trials have seldom been published has made the scientific community overstate the efficacy of SSRIs,3, 4 some frequently cited debaters even claiming that SSRIs are in fact devoid of any specific, pharmacological antidepressant properties.2, 5

Needless to say, an effective treatment should be expected to outperform placebo in a vast majority of sufficiently sized trials; the high rate of unsuccessful SSRI trials hence is a matter of legitimate concern. It is therefore important to shed further light on why many trials are negative while others are not, that is, if the negative trials reflect a true inability of SSRIs to generate an antidepressant signal in many cohorts of depressed patients, or if the frequent failures to demonstrate a beneficial effect may be caused by methodological problems.

Constructed in the late 50s,6 the Hamilton depression rating scale (HDRS) in its most common version (HDRS-17) comprises 17 different symptoms for which a score between 0 and 2 or 0 and 4 may be given. For the majority of antidepressant drug trials, the decrease in the total score of this scale (HDRS-17-sum) has served as a primary effect parameter.7, 8 As previously pointed out,8, 9, 10, 11 there are, however, numerous reasons to question the usefulness of this measure.

First, HDRS-17 is multidimensional, indicating that relevant improvement in one domain of symptoms may be masked, due to enhanced variability, by lack of improvement in other possibly less relevant domains.12, 13, 14 Second, the included symptoms differ considerably in terms of burden of illness and many of them correlate poorly with depression severity.9, 15 Third, some items refer to several heterogeneous symptoms, and the different grades for a certain item do not always represent differences in severity but qualitatively distinct phenomena; both these aspects may contribute to the poor inter-rater reliability marring this instrument.8, 16 Fourth, many patients reporting some of the symptoms to be absent already at baseline is bound to reduce the sensitivity of the instrument by enhancing variability, as is the fact that some of the symptoms, such as backaches and headache, are common also in non-depressed subjects, and may therefore be present also after recovery. In line with this, it was recently reported that 25% of previously depressed subjects scoring 8–12 on HDRS-17 regarded themselves to be in remission.17

In spite of these well-established shortcomings of the HDRS-17, the recent debate concerning the efficacy of antidepressants is largely based on the outcome of individual studies or meta-analyses using HDRS-17-sum as an effect parameter.18, 19, 20, 21, 22 To explore whether the use of this measure may partly explain why many SSRI trials have been negative, we have re-analyzed eighteen drug company-sponsored depression trials, comprising 32 different comparisons, after replacing HDRS-17-sum as an effect parameter with a single item that, unlike many of the other items, is reported by a vast majority of the participants at baseline, that is, depressed mood. To shed additional light on the sensitivity of different measures to detect an antidepressant signal, we also calculated effect sizes, based either on mean effect sizes from each study, or on the pooled population of 6669 subjects, for (i) HDRS-17-sum, (ii) various sub-scales and (iii) all 17 items.

Materials and Methods

Data acquisition

When designing this study we aimed at including all drug company-sponsored placebo-controlled trials of reasonable size and regarding the treatment of depression in adults that had been conducted for the four major first generation SSRIs, that is, citalopram, fluoxetine, paroxetine and sertraline, at the time of their approval. To this end, we requested patient level-data from all such studies from Lundbeck (Valby, Denmark) (citalopram), GSK (Brentford, UK) (paroxetine), Eli Lilly (Indianapolis, US) (fluoxetine) and Pfizer (New York, US) (sertraline), respectively. While Eli Lilly was unable to provide us with data on fluoxetine trials for practical reasons, the relevant studies not being available in an electronic format, the other three companies granted our request. To preclude any bias in the selection of trials, we confirmed that we had access to all pertinent studies by examining the FDA Approval Packages for citalopram,23 paroxetine IR,24 paroxetine CR25 and sertraline,26 respectively.

Since the aim of this project was to examine the sensitivity of the HDRS to detect changes between active drug and placebo, studies that did not employ the HDRS-17 or an extended variant thereof were excluded from further analysis. The outcome parameter in focus of this study being that of statistical significance of individual trials, it was also deemed important to exclude trials for which a falsely negative outcome might be expected due to low statistical power; for this reason, an a priori decision was made not to include any trial with a sample size of less than 50 subjects in any treatment arm.

In the FDA report on paroxetine IR, two multi-center trials (GSK/002 and GSK/003) were presented as 10 minor studies, but since they were in fact conducted as two large trials, they are included in this analysis as such. GSK also provided data from five additional studies regarding paroxetine IR or paroxetine CR that were not available at the time of FDA approval but did meet the other inclusion criteria. Likewise, two of the placebo-controlled sertraline trials submitted by Pfizer were not mentioned in the FDA report; these were, however, not post-marketing but post-registration trials, that is, completed between the submission of the new drug application to the FDA and its approval.

In total, eight trials regarding paroxetine immediate release (IR), five regarding paroxetine controlled release (CR), three regarding citalopram and five regarding sertraline were eligible for inclusion. In two paroxetine trials and in one sertraline trial, another SSRI, fluoxetine, had served as an active comparator, which enabled us to include also three comparisons of this drug versus placebo.

Statistical analysis

First, analysis of covariance (ANCOVA) was used to calculate levels of statistical significance and effect sizes (as defined as the estimated marginal mean difference between groups divided by the root mean squared error) for all comparisons of active drug versus placebo when using either HDRS-17-sum or depressed mood as an effect parameter. Change from baseline to end point with respect to either HDRS-17-sum or depressed mood were dependent variables, treatment and study center were fixed effects and baseline severity as assessed using the corresponding scale was included as a covariate. The treatment–center interaction was assessed in all comparisons regarding individual studies, but excluded from the model if non-significant (P>0.05). The ratios of comparisons reaching statistical significance when using HDRS-17-sum or depressed mood, respectively, as an effect parameter, were compared using McNemar’s test. While differences between the two effect parameters with respect to mean effect sizes were assessed using a paired t-test, the ratio of effect size estimates improving by changing from HDRS-17-sum to depressed mood was assessed using Pearson’s χ2-test.

While we used ANCOVA to comply with how treatment groups have usually been compared with respect to HDRS-17-sum data in antidepressant trials, we acknowledge that the use of this method for comparing groups with respect to the depressed mood item could be questioned given that this item is assessed by an ordinal scale comprising merely five points. To address the possible influence of this aspect, all analyses regarding depressed mood were repeated using ordinal logistic regression.

Second, we wanted to compare the effect size for depressed mood also with those for other possible effect parameters, that is, all individual HDRS-17 items and all HDRS-17 subscales.9, 12, 13, 14, 27, 28, 29 To this end, effect sizes for these parameters were extracted for all 32 drug-placebo comparison using an ANCOVA model composed of change in the measure in question as a dependent variable, treatment and center as fixed factors, and baseline rating of the relevant measure as a covariate. These effect sizes were then compared using repeated measures ANOVA, the model consisting of the effects sizes for all parameters as the within-cases factor and whether a particular comparison was conducted pre- or post FDA approval as a between-cases factor. Following the ANOVA, paired t-tests were calculated for depressed mood versus all other individual items, all subscales and HDRS-17-sum, and also for all subscales versus HDRS-17-sum. Notably, for trials with more than one active treatment arm, the same placebo group was used for all drug-placebo comparisons, the effect size estimates revealed hence not being independent. To verify that this did not bias the results, all analyses were repeated using study-level effect sizes produced by calculating the arithmetic mean of all effect sizes from the same study.

Third, to obtain also patient level-based measures of the effect sizes for all various items and scales, a pooled analysis was conducted on all cases categorized into two groups, that is, patients receiving an SSRI (regardless of drug or dose) and those receiving placebo. Again an ANCOVA model composed of change in the measure in question as the dependent variable, treatment and center as fixed factors, and baseline rating of the relevant measure as a covariate, was used. As for the analyses of the individual studies, all calculations regarding single items were repeated using ordinal regression. As a sensitivity analysis, we also assessed the effect size for HDRS-17-sum and depressed mood (using ANCOVA) in the pooled population after having excluded the paroxetine and sertraline trials that had been conducted after FDA marketing approval.

Finally, the observation of negative effect sizes for three individual HDRS-17 items, that is, weight change (significant), gastrointestinal complaints (non-significant) and sexual functioning (non-significant), prompted us to explore to what extent this finding may be partly explained by these symptoms often emerging as side effects of SSRI treatment; to this end, we used χ2-tests to compare the number of subjects reporting an increase in symptom rating at end point as compared with baseline in those given SSRI and placebo, respectively.

To comply with the procedures applied by the pharmaceutical companies and endorsed by the authorities, all analyses were based on the intention-to-treat (ITT) population, defined as all randomized patients with at least one post-baseline measurement. For those discontinuing prematurely, the last observation carried forward approach was used. To enable comparisons of different trials, the symptom assessment at week 6 was extracted and used as an end point measure. One trial, LB/85A, was a four-week-trial; therefore, the ratings at week 4 were used. For trial GSK/874, no ratings from week 6 were available; hence, week 8 ratings were used.

Ethics

The Regional Ethical Review Board of Gothenburg, Sweden, has reviewed the study protocol and issued an advisory opinion stating no objection to the conduct of this study.

Results

Baseline characteristics of all included trials are displayed in Table 1.

Table 1 Included trials
Full size table

All treatment–center interactions were non-significant (P>0.05) but for study GSK/448, where a significant (P<0.001) interaction was found when HDRS-17-sum (but not depressed mood) was used as an effect parameter, and for study PZ/315, where a significant interaction (P=0.004) was found for depressed mood (but not for HDRS-17-sum). For GSK/448, follow-up analyses revealed this to be primarily due to one center, comprising 18 subjects, that displayed a highly aberrant outcome favouring both paroxetine groups; exclusion of this center rendered the treatment–center interaction non-significant (P=0.2). For PZ/315, sequential exclusion of the two most divergent centers, one favouring sertraline (n=10) and the other favouring placebo (n=6), yielded a non-significant (P=0.13) treatment–center interaction. Subjects from these three centers were omitted from all further analyses, hence yielding an evaluable population of 6669 patients.

Levels of significance and effect sizes for individual comparisons are summarized in Table 2. While 14 out of 32 (44%) comparisons yielded a significant difference when efficacy was measured using HDRS-17-sum, 29 out of 32 (91%) comparisons were significant when depressed mood was used as an effect parameter (P<0.001). Likewise, exchanging HDRS-17-sum for depressed mood as an effect parameter improved the effect size for 30 out of 32 comparisons (P<0.001), the mean (±s.d.) effect size for depressed mood (0.39±0.13) being higher than for HDRS-17-sum (0.24±0.13) (P<0.001). Repeating all comparisons in which depressed mood was the dependent variable using ordinal logistic regression rather than ANCOVA had only minor impact on levels of significance and did not change the rate of positive trials (Table 2).

Table 2 Effect sizes and levels of significance for all individual drug-placebo comparisons
Full size table

For the comparison-level analysis of all different effect parameters, Mauchly’s test indicated the assumption of sphericity to be violated (P<0.001); however, the P-value referring to the difference with respect to effect sizes (the within-cases factor) remained highly significant regardless of method of correction, the F-value being 47.4 (P-value after Greenhouse-Geisser correction: <0.001). In contrast, the factor indicating if the study had been conducted pre- or post-marketing (the between-cases factor) was non-significant (P=0.2) and therefore excluded from the analysis.

Pairwise comparisons showed the mean effect size for the depressed mood item to be significantly higher not only than the mean effect size for HDRS-17-sum (P<0.001), hence confirming the outcome of the previous analysis, but also than the mean effect sizes for all other individual items (P<0.001). All four subscales yielded significantly higher effect sizes than HDRS-17-sum (P<0.001) but the effect size for depressed mood was significantly higher than those for all subscales (P<0.001). Using study-level (n=18) rather than comparison-level effect sizes (n=32) yielded overall higher P-values, but all differences remained significant, depressed mood again outperforming all other individual items (highest P=0.004 for psychic anxiety), all subscales (highest P=0.01 for the Bech subscale) and HDRS-17-sum (P<0.001).

Table 3 is based on the pooled population and lists effect sizes for (i) HDRS-17-sum, (ii) some of the subscales previously proposed9, 12, 13, 14, 27, 28, 29 and (iii) all individual HDRS-17 items. In line with the comparisons of mean effect sizes presented above, all subscales yielded higher effect sizes than the HDRS-17-sum; however, for all subscales and for all other individual items, the effect size was lower than it was for the depressed mood item. Exclusion of the seven trials (comprising 12 comparisons) that were not included in the FDA reports since they were completed after the submission of the FDA application yielded somewhat higher effect sizes for both HDRS-17-sum and depressed mood, but again the effect size for depressed mood was considerably higher than that for HDRS-17-sum (Table 3).

Table 3 Effect sizes and P-values for different measures of efficacy in the pooled population.
Full size table

For three items, gastrointestinal complaints, loss of weight and sexual functioning, the effect sizes based on the pooled population were negative (though non-significantly for gastrointestinal complaints and sexual functioning). For all three, an increase in severity at end point as compared with baseline was significantly more common in SSRI-treated patients than in those given placebo, the odds ratios being 1.27 for gastrointestinal symptoms (P=0.009), 1.24 for weight change (P=.005) and 1.21 for sexual symptoms (P=0.03).

Discussion

While 18 out of 32 comparisons failed to reveal a significant superiority of the studied SSRI over placebo with respect to reduction in HDRS-17-sum, only three out of 32 comparisons were negative with respect to the reduction in depressed mood. The problem of negative SSRI trials, which has been a major argument for the questioning of antidepressants, hence appears to be partly due to the use of an insensitive measure of efficacy.

Our decision to focus on depressed mood, rather than any other item, was motivated by this symptom displaying the largest baseline severity in the pooled population (Table 3), and was further reinforced by the fact that since long it has been attributed particular importance by the FDA when evaluating antidepressant efficacy;23, 25 moreover, it is one of two key symptoms, one of which must be at hand, in the DSM definition of depression. While not claiming that assessing depressed mood only is the optimal way of recording symptom severity, or that other symptoms are irrelevant, we do suggest that a treatment faithfully outperforming placebo in reducing depressed mood can hardly be regarded as ineffective. Also, by demonstrating a consistent reduction in this symptom, our results refute the concern raised by some authors that the superiority of antidepressants sometimes observed also with respect to HDRS-17-sum might be due to other influences than a genuine antidepressant effect, such as non-specific sedation.5

Many recent attempts to reveal the true efficacy of antidepressants have been based on meta-analyses or patient level-based mega-analyses. While of considerable interest, such approaches are marred by the inevitable problem that individual studies yielding misleading results due to hidden methodological shortcomings, which, given the highly disparate outcome of different trials, are probably not uncommon in this area, may render also the overall outcome misleading. This is why we in this study chose a different approach, that is, to explore the consistency of the ability of SSRIs to generate an antidepressant signal across studies when using a potentially more sensitive measure than the conventional one. Although it may be argued that reaching a P-value of <0.05 for the primary comparison is an arbitrary definition of efficacy, the actual importance of this criterion for the fate of novel putative antidepressants, and for how established drugs are being valued, is obvious.

Although our primary purpose was to compare the outcome of trials after changing effect parameter from HDRS-17-sum to depressed mood, we also compared the mean effect sizes (based on the effect sizes from all individual comparisons) for depressed mood with those for all other individual HDRS-17 items; in addition, the effect sizes for the various uni-dimensional subscales – comprising the depressed mood item and 5-6 additional HDRS-17 symptoms9, 12, 13, 14, 27, 28, 29 – were compared both with the effect size for depressed mood and with that for HDRS-17-sum. These analyses revealed all subscales to yield higher effect sizes than did HDRS-17-sum; however, the effect size for depressed mood was significantly higher than those for all subscales as well as for all individual items. An analysis of the pooled population, comprising all subjects from all studies, was also undertaken – not because we believe that such an approach would reveal the true magnitude of the actual efficacy of SSRIs, but to obtain a patient level-based assessment of the relative magnitude of different effect sizes. The results obtained by this approach were not markedly different from those based on the means of the effect sizes for the different comparisons, again showing depressed mood to be the parameter resulting in the highest effect size.

Previous reports on effect sizes for the individual items of the HDRS-17 in patients treated with antidepressants are largely in line with our study, depressed mood being one of 3–4 items displaying the largest effect size also in these.27, 30 Minor differences in this regard are, however, notable and likely to be caused both by differences over time with respect to the studied patient populations and by differences in the pharmacological profiles of the tested drugs. For example, when considering that a previous study by Faries et al.27 revealed as high effect size for the items guilt and suicidality as for depressed mood in patients treated with tricylic antidepressants, it should be considered that these symptoms can be assumed to have been fairly common in studies conducted in the tricyclic era, hence making it easier to detect a robust effect of treatment in these days. In contrast, in our population, which is based largely on outpatient studies for which suicidal ideation has been an exclusion criterion, the mean ratings at baseline of both these items, and especially suicidal ideation, were markedly lower than the rating of depressed mood (Table 3). In contrast, the effect size for the symptom early insomnia being higher in the analysis by Faries et al.27 than in the present study may at least partly be explained by SSRIs causing disturbed sleep as a side effect,31 while tricyclic antidepressants exert a non-specific sedative effect by antagonizing histaminergic H1 receptors.32

For three items, that is, weight change, gastrointestinal complaints and sexual functioning, the present analyses revealed negative effect sizes; however, of these effects only that on weight change was statistically significant. The effect sizes of these items being negative is in line with a previous study30 based on eight comparisons of fluoxetine versus placebo, none of which was included in our analysis as they were conducted by a company producing the non-SSRI antidepressant venlafaxine, fluoxetine merely serving as an active control. For two of the three items for which we observed negative effect sizes, gastrointestinal complaints and weight change, the apparent lack of improvement may be partly explained by the very low rating already at baseline (Table 3) rendering further improvement difficult to detect. For all three items, an important factor may also be that they measure common SSRI side effects that are present also in many patients that have recovered from their illness;31, 33, 34 supporting this view, significantly more subjects treated with SSRI than placebo reported aggravation at end point as compared with baseline for these three symptoms. Although the possible negative impact on life quality of these and other side effects must be taken into account when considering the pros and cons of SSRI treatment, the present results underline that including such complaints when assessing efficacy is problematic. Not least for the purpose of future drug development, the importance of separating a drug that is effective but causing side effects from one that is ineffective should be emphasized.

Of note is that Klein and Fink questioned the use of multi-item rating scales for the evaluation of drug treatment in psychiatry already in 196335 and that numerous reports since then have demonstrated the insufficient sensitivity marring HDRS-17; that this instrument nevertheless has served as primary effect parameter in a vast majority of antidepressant trials during the past 50 years may seem surprising, not least since it would be in the interest of the pharmaceutical industry to avoid a rating scale not optimizing the chance of detecting a difference between active drug and placebo. It should, however, be considered that the primary purpose of the trials conducted by the pharmaceutical companies has been to obtain marketing approval as rapidly and safely as possible, and that one therefore has been inclined to copy the design from previous studies that have been successful in this regard, for example by using an effect parameter that is known to be acceptable to the FDA, rather than to give priority to innovation. From a scientific perspective, it is, however, imperative that both drug companies and regulatory authorities strive for an improvement with respect to antidepressant trial design, so that future such studies are better suited to reflect the true efficacy of the drugs. Moreover, the present study highlights the importance of extracting as much scientifically relevant information as possible from already conducted studies, rather than to feel restricted by the formal aspects often characterizing evidence based medicine, such as to consider only the outcome parameter named primary in the trial protocol.

It should be emphasized that refraining from using the sum of many disparate items as primary effect parameter in depression trials does not mean that one should not record different symptoms when evaluating the therapeutic profile of an antidepressant drug. Whereas a reasonable conclusion from this and many previous reports would be that HDRS-17-sum should no longer be used as a primary effect parameter in antidepressant trials; more research is warranted to shed light on the sensitivity of alternative scales (such as the Montgomery-Åsberg Depression Rating Scale10 and the various brief variants of the HDRS) on the one hand and various single-item assessments on the other. It should be noted that the inter-rater reliability is often higher for multi-item assessments than for single-item assessments and that Cicchetti and Prusoff16 have shown this to be the case also for HDRS. For the multi-item approach to be advantageous, all items, however, need to be both reliable and relevant; a high inter-rater reliability hence cannot compensate for poor validity. Of note in this context is that Cicchetti and Prusoff16 reported depressed mood to be the individudal HDRS-17 item showing the highest inter-rater reliability at end point (0.72) and that its reliability was not markedly lower than that for HDRS-17-sum (0.82).

While sufficiently large to justify the use of SSRIs, and larger than when estimated using HDRS-17-sum, the effect size obtained when using depressed mood as an effect parameter in the pooled population was merely moderate (0.40 in the total population; 0.44 after exclusion of post-marketing trials). When interpreting these numbers, one should, however, consider (i) that also comparisons with suboptimal dosage were included, (ii) that the analysis is based on symptom rating recorded too early (week 6) for a full effect to be at hand33 and (iii) that the ITT population comprised subjects dropping out before any marked effect of treatment could be expected. In addition, we were unable to address other methodological factors that may hamper the detection of differences between active drug and placebo, such as poor compliance, poor rater performance,16, 36, 37 artificial overrating at inclusion38 and an overly liberal inclusion of participants.39 Similarly, we were unable to control for a factor suggested to overstate the efficacy of active drugs, that is, the possibility that side effects may make patients and investigators realize that a patient is on active treatment.2

While these methodological problems unfortunately make it very difficult to evaluate the actual magnitude of the effect of antidepressants on the basis of modern company-sponsored trials,40, 41 it is nevertheless important that these are designed in a way enabling them to identify drugs generating an antidepressant signal so that novel drugs with antidepressant properties are not mistakenly deemed ineffective. The present data highlight the risk of missing an antidepressant signal by not using a sufficiently sensitive outcome parameter.

Recently mixed-effects repeated-measures models have been used for re-analyses of antidepressant trials to reduce the influence of the drop out factor and non-random missing data.21, 28, 42 In this study we refrained from using this strategy, in spite of its obvious virtues, as our aim was to explore to what extent the use of an inadequate primary effect parameter might explain why a large percentage of placebo-controlled trials using SSRIs were deemed negative when analyzed using the conventional statistics mostly applied by the drug companies and requested and approved by the authorities, that is, intention to treatment-based ANCOVA.23, 25

In conclusion, we suggest that many SSRI trials that have been deemed negative, and often never published, do provide scientifically valid support for the tested drug being antidepressant, the negative outcome being caused by the use of an insensitive measure of improvement. Our observation that, in spite of the many methodological shortcomings marring antidepressant trials, 29 out of 32 comparisons in this analysis (including those with suboptimal dosage) did pick up an antidepressant signal from the tested SSRI suggests the antidepressant effect of these drugs to be highly consistent across trials.