In their article ‘Is cognitive functioning impaired in methamphetamine users? A critical review,’ Hart et al (2011) present a provocative view of the literature on methamphetamine abuse and cognition to date, parting in their conclusions from previous reviews, which have tended to suggest that methamphetamine abuse is associated with significant cognitive abnormalities. In contrast, Hart et al suggest that such deficits have, in fact, only been documented in a minority of cases, and that the assumptions held by the field may have resulted from poorly worded or conflated interpretations of the evidence rather than consistent or convincing findings, to the potential detriment of treatment and policy decisions.

The concerns expressed by Hart et al are certainly well justified, and the article is a much-needed wake-up call to the field, as the authors flag a number of important methodology and data interpretation problems plaguing current research. Among their most poignant criticisms is the language conventionally used in interpreting and discussing findings, as studies comparing methamphetamine-abusing and control subjects often interpret any statistical difference in scores as clinically significant, resulting in overstated conclusions that often overreach the actual data. That is, studies tend to confound the definitions of statistical significance and clinically significant impairment, and assume that differences in cognitive test scores reflect a meaningful difference in real-life outcomes. In articulating this perspective and exposing the tendency of the field to report more severe deficits than are warranted by the data, the authors encourage more critical reading of previous literature and raise the bar for future studies—a service to the field as it moves forward.

At the same time, we find it curious that Hart et al do not offer an operational definition of what they consider clinically significant impairment, and we question their suggestion that the majority of studies to date are of limited clinical value because test scores have not exceeded a particular normative cut-off. The authors appear to adhere to an ‘abnormality-detection’ definition, where impairment consists of a score which deviates to some degree from normative data. However, they seem to imply that this ‘abnormality-detection’ definition is synonymous with an ‘everyday import’ definition of impairment, where cognitive scores have everyday implications for real-world functioning. It should be clear that these perspectives are not the same, as a score may have relevance for everyday function and not meet a categorical cut-off for abnormality, and a score may be unrelated to everyday function even if it exceeds a particular cut-off (eg, see Silverberg and Millis (2009) for evidence that demographically normed scores can weaken real-world functional relationships).

Even within the ‘abnormality-detection’ definition of impairment, Hart et al do not define the degree to which a score needs to differ from demographic norms to be considered impaired. Although the authors mention a paper by Kalechstein et al (2003), which used stringent impairment criteria (>2 SD below normative data), they do not cover an article by Rippeth et al (2004), which used a more liberal definition of impairment (the average of all demographically normed scores in a neuropsychological battery needed to fall approximately one-half SD below expectations—a criterion shown to maximize the sensitivity and specificity in detecting subtle cognitive deficits; see Carey et al (2004)). According to this definition, 40% of methamphetamine-dependent participants were classified as impaired, compared with 18% of healthy control subjects, replicating findings of subtle cognitive deficits in methamphetamine dependence, and underscoring the idea that no one cut-off value is universally accepted or optimal.

Most importantly, however, we feel that regardless of a particular cut-off chosen for impairment, it is vital that one not reify this threshold, as cognitive differences can be distressing for patients even if they do not fall some specified number of SD below the mean. For example, in criticizing an article by London et al (2004), which found that methamphetamine-dependent subjects had higher Beck Depression Inventory (BDI) scores than healthy control subjects, Hart et al argue that the clinical importance of this finding is unclear because the BDI scores do not reach a threshold for clinical depression (which may have reflected inclusion/exclusion criteria for study participation rather than patterns in the general methamphetamine-abusing population). Even so, the methamphetamine-dependent subjects in this study endorsed an average of nine depressive items, whereas healthy control subjects endorsed an average of 1. We find it difficult to believe that this difference has no relation to emotional functioning, regardless of a particular cut-off. Interestingly, Hart et al do not apply a given cut-off rule in concluding that acute administration of MA enhances cognitive functioning (they do not discuss improvement in terms of SD units from normative data). If the authors hold that improved scores may have clinical relevance on a continuous, rather than a categorical basis, we would agree.

Finally, given the authors' interest in real-world functioning, we find it noteworthy that they did not mention an article by Henry et al (2010), which examined measures of everyday function in abstinent methamphetamine-dependent individuals. These authors found that methamphetamine-dependent subjects performed worse on almost all indices of daily function (comprehension, finances, communication, transportation, medication management) than healthy control subjects of comparable age and education, and Wisconsin Card Sorting Test scores correlated with performance in the everyday functional domains. Perhaps one resolution to improving studies in the future is to encourage the consistent inclusion of measures of daily functioning and disability. Given the number of cognitive tests that measure specific cognitive domains but lack standardized norms, an assessment of their relationship to measures of daily function may yield useful information about clinical relevance from an ‘everyday import’ perspective.

Despite our criticisms, we agree with the conclusion by Hart et al that the field has generally over-interpreted the severity of cognitive problems in methamphetamine dependence. Our hope moving forward is that future studies will clarify their operational definitions of clinically significant impairment, and adopt appropriate strategies for data interpretation and communication of conclusions.