Main

In 1948, a serum tonic factor released from platelets during blood clotting was first isolated, purified, and identified as a monoamine named serotonin by Rapport, Green, and Page (Rapport et al. 1948). This is of personal relevance, because Dr. Maurice Rapport was the first Chief of the Department of Neuroscience at New York State Psychiatric Institute, and I have the honor of being the third Chief of Neuroscience at New York State Psychiatric Institute. A strong tradition of serotonin research in this Department has been uninterrupted since the foundation of this department (Kolb and Roizin 1993). As described elsewhere in this volume, serotonin was subsequently found to be the same as enteramine, an amine found in the small intestine and, subsequently, the highest concentrations of serotonin were found to be in the brainstem, with lesser concentrations throughout the brain. It became clear that serotonin was really a neurotransmitter, in addition to playing a role in intestinal motility and platelet aggregation.

In 1969, several investigators proposed the indoleamine hypothesis of depression (Coppen 1969; Lapin and Oxenkrug 1969; Coppen et al. 1972; Charney et al. 1981; Meltzer and Lowy 1987). This hypothesis proposed that the vulnerability to either depression or mania was related to decreased serotonergic activity, attributable to either less serotonin release or to fewer serotonin receptors or impaired serotonin receptor-mediated signal transduction. Over the last 30 years, a variety of studies of the serotonergic system have reinforced its role in major depression and identified additional associations with suicidal behavior, impulsive aggression, eating disorders, obsessive-compulsive disorder, anxiety disorders, and alcoholism. The serotonergic system also plays a role in the regulation of a variety of basic biological functions including sleep, appetite, circadian rhythm, and cognitive function.

NEUROANATOMY OF THE SEROTONERGIC SYSTEM

The neuroanatomy of the serotonergic system in the human brain has been well described by Törk (1990). The reader is directed to that paper for its excellent figures. The largest nucleus, the dorsal raphe, is divided into five or six subgroups, depending upon the histological methods employed. There are other major nuclei, including the median raphe nucleus, which projects to all parts of the brain. Two major ascending systems are described. One involves thin axons, which are the most numerous, have varicosities that are fusiform in shape, and are sometimes known as D fibers. The second system has thicker fibers with fewer varicosities, known as M fibers. There are also very fine fibers with large, round or oval varicosities. The M fibers and the D fibers provide a dual system of innervation, and the M fiber system includes basket axons that envelop target neurons in the cortex. This extensive system of innervation by the serotonergic system of cortical and subcortical target areas permits the serotonergic system to affect many brain functions and provides the anatomical basis for explaining how serotonin can influence so many diverse brain functions. At the same time, it also provides an anatomical basis for the role identified for the serotonergic system in many psychiatric disorders. Within the raphe nuclei, subgroups of serotonergic neurons have co-transmitters or peptides, including substance P and thyrotropin-releasing hormone. The role of substance P in the serotonergic system is of particular interest, given recent evidence that substance P antagonists may have antidepressant properties (Kramer et al. 1998).

SEROTONIN RECEPTOR SUBTYPES

The serotonergic system has over a dozen serotonin receptor subtypes (Saudou and Hen 1994). The basis of the classification of these receptor subtypes depends upon a number of criteria, including genetic, the transducer to which the receptors are coupled, and pharmacological properties. Not all of these receptor subtypes have identified physiological roles in the brain, and selective agonists and antagonists have not been identified for all the receptor subtypes. Nevertheless, the existence of so many receptor subtypes for a single transmitter permits a great diversity of signaling so that the same neurotransmitter can produce very different effects on different neurons and on different parts of the same neuron. A discussion of the various subtypes of these receptors is beyond the scope of this paper, but those receptor subtypes that have been particularly involved in the pathogenesis of depression and in suicide are discussed.

5-HT1A Receptors

The 5-HT1A receptor has been hypothesized to have a role in depression and anxiety based on evidence that 5-HT1A agonists may have anxiolytic and antidepressant properties (Baldwin and Rudge 1995). 5-HT1A receptors exist as autoreceptors on the soma and dendrites of serotonergic neurons (Sotelo et al. 1990); whereas, they are largely postsynaptic receptors in the serotonin system's terminal fields, on target neurons in the cortex and subcortical regions (Pazos et al. 1987). Thus, reduced function of 5-HT1A autoreceptors would result in increased serotonin transmission; whereas, decreased function at the target neurons would result in decreased 5-HT1A-mediated neurotransmission effects (Blier et al. 1990). This has led to a number of hypotheses with regard to the action of antidepressants and drugs effective for obsessive-compulsive disorder. Blier and colleagues have reported that chronic treatment with antidepressants produces a reduction in electrophysiological responsivity of the somatodendritic 5-HT1A autoreceptor and no reduction in 5-HT1A receptor responses in the hippocampus (Blier et al. 1990). They propose that the progressive desensitization of the autoreceptor leads to a progressive increase in serotonin activity and, thereby, mediates an antidepressant effect. Autoradiography studies have suggested that there may be downregulation of somatodendritic autoreceptors and no effect on hippocampal postsynaptic 5-HT1A receptors with chronic antidepressant administration. Interestingly, electroconvulsive shock (ECS) produces an increase in 5-HT1A receptors in parts of the hippocampus (Hayakawa et al. 1994), and, thereby, may mediate an antidepressant effect. 5-HT1A receptor responses in orbital prefrontal cortex are reported to be decreased by selective serotonin reuptake inhibitors (SSRIs) with long-term administration in rodents, and Blier and colleagues have proposed that this is the mechanism of action that underlies the therapeutic effect of SSRIs in Obsessive-compulsive disorder (see elsewhere in this volume).

5-HT1B Receptors

The 5-HT1B receptor is reported to have a role in the regulation of impulsive-aggressive and sexual behaviors, cocaine intake, and alcohol intake, based on the observation of an increase in all of these behaviors in 5-HT1B knockout mice (Saudou et al. 1994; Ramboz et al. 1996; Rocha et al. 1997). Very few studies have been carried out of 5-HT1B receptors in humans, and the development of therapeutic tools that affect this receptor type is in progress. The 5-HT1B receptor is very closely related to the 5-HT1D receptor. 5-HT1D receptor agonists have been successfully used in the treatment of migraine (Perry and Markham 1998). Sumatriptan, a 5-HT1D agonist, presumably activates 5-HT1D autoreceptors at serotonin nerve terminals, reducing serotonin release, and this may account for its antimigrainous action. 5-HT1D receptors function as autoreceptors both at the level of the nerve terminals as well as autoreceptors on the soma of serotonin neurons. The augmentation or acceleration of antidepressant effects is being proposed for drugs such as pindolol, which are 5-HT1A autoreceptor antagonists. 5-HT1D antagonists are another class of potential augmentators of antidepressant action.

5-HT2A Receptors

A role for the 5-HT2A receptor has been proposed in the pathogenesis of major depression. Some studies have suggested that tricyclic antidepressants may downregulate the number of 5-HT2A receptors in rat cerebral cortex, and a course of ECS in rodents has been reported to increase the number of 5-HT2A receptors. This effect of ECS may be a mechanism for enhancing 5-HT2A receptor-mediated signaling in the brain and has, therefore, been proposed as a mechanism of antidepressant action of electroconvulsive therapy (Mann 1998a). Whether this effect actually occurs in human brain is unknown and we will have to await the conduct of neuroreceptor studies using such techniques as positron emission tomography (PET) or single-photon emission computer tomography (SPECT). Of note, some studies have reported higher levels of 5-HT2A receptors in the brain of suicides and on the platelets of individuals with major depression who have attempted suicide (see Mann 1998b for a review). Thus, there is an association between increased numbers of 5-HT2A receptors and suicidal behavior, but the implications of the increase in receptor number is uncertain. Lesioning the serotonergic system in rodents does not produce a consistent upregulation of 5-HT2A receptors and, therefore, it is not certain that the increased number of such receptors reflects reduced serotonin release. Our group (Mann et al. 1992b) found that the increased number of platelet 5-HT2A receptors was actually associated with decreased signal transduction in platelets of depressed patients with a history of suicidal behavior. Thus, it remains to be determined whether the increased number of 5-HT2A receptors in the prefrontal cortex of suicides is, in fact, offset by impaired signal transduction, or perhaps is secondary to signal transduction defects downstream from the receptor.

5-HT3 Receptors

The 5-HT3 receptor subgroup, originally identified as the “M” receptors by Gaddum and Picarelli (1957), is found in many peripheral tissues. Within the CNS, this subpopulation has been shown to regulate the release of dopamine from rat striatal slices (Costall et al. 1987; Bladina et al. 1988) and the release of acetylcholine. 5-HT3 antagonists, such as ondansetron, are valuable in controlling nausea associated with chemotherapy (Cubeddu et al. 1990), and the 5-HT3 receptor population is concentrated in relation to the nucleus of the tractus solitarius and the emetic center in the brainstem. 5-HT3 receptors are distinguished from other serotonin receptors in that they are an ion channel ligated receptor (Costall 1993). Preliminary studies have not linked this receptor population to mood disorders or suicidal behavior.

SEROTONERGIC SYSTEM AND MAJOR DEPRESSION

Reduced serotonergic function has been related to the syndrome of a major depressive disorder and, in addition, to the components of psychopathology, as discussed above. The methods for characterizing serotonergic function and measuring it are mostly limited to cerebrospinal fluid, neuroendocrine challenge tests, platelet serotonin protein related assays, and functional brain imaging of the serotonergic system.

Cerebrospinal Fluid 5-Hydroxyindoleacetic Acid in Depression

Some of the earliest studies of the serotonergic system in the brain of patients employed the method of assaying the breakdown product of serotonin, 5-hydroxyindoleacetic acid (5-HIAA) (Mann et al. 1989). The underlying assumption is that 5-HIAA is related to neuronal activity. Several studies, but not all, have found reduced levels of CSF 5-HIAA in depressed patients (Dencker et al. 1966; Mendels et al. 1972; van Praag and de Haan 1979; Åsberg et al. 1984; Mårtensson et al. 1989). The degree of reduction of CSF 5-HIAA generally does not correlate with severity of depression. Many antidepressant medications, particularly serotonin reuptake inhibitors and monoamine oxidase inhibitors (MAOIs), reduce CSF 5-HIAA (Mårtensson et al. 1989) because of feedback inhibition resulting from increased intrasynaptic concentrations of serotonin. Thus, low levels of CSF 5-HIAA in treated patients do not indicate reduced serotonergic activity. Levels of CSF 5-HIAA are lower in depressed patients with a history of serious suicidal behavior, as compared to depressed patients with no history of suicide attempts (see Mann et al. 1996b for a review). Gibbons and Davis (1986) analyzed CSF 5-HIAA data in depressed patients from five research groups and found that the levels had a bimodal distribution. The low CSF 5-HIAA group were not distinguished by more severe depression, but by a history of serious suicide attempts. Given that CSF 5-HIAA correlates with levels of 5-HIAA in prefrontal cortex in postmortem studies, it has been proposed that CSF 5-HIAA is a reasonable index of prefrontal serotonin turnover.

Serotonin Neuroendocrine Challenge Tests

The most widely used challenge agent to study the serotoneric system has been fenfluramine, which causes the release of serotonin and blocks its reuptake. Serotonin, in turn, causes the release of corticotrophin-releasing hormone and an unidentified prolactin-releasing factor from the hypothalamus. These, in turn, cause the release of ACTH and prolactin from the pituitary into the bloodstream. ACTH causes the release of cortisol from the adrenal cortex into the blood. The usual indices of serotonin release are prolactin and cortisol release. The latter is complicated by the noise attributable to its marked diurnal variation. Studies with fenfluramine have been reviewed in detail elsewhere (Mann et al. 1995), but generally report a reduction in prolactin responses in depressed patients. We have found that patients who have had a past history of depression, but have been in remission for at least a year and are off psychotropic medications, have a blunted prolactin response that is comparable in degree to acutely depressed patients requiring electroconvulsive therapy (Flory et al. 1998). Thus, it seems that impaired serotonergic activity, as demonstrated by the fenfluramine challenge test, is present to a comparable degree when patients are in remission as when they are acutely depressed. We have hypothesized that impaired serotonergic function is a biochemical trait that underlies the vulnerability for recurrent episodes of major depression. Other types of neuroendocrine challenges also indicate impaired serotonergic function. For example, intravenously administered tryptophan and intravenous clomipramine also produce blunted neuroendocrine responses in depressed patients (Heninger et al. 1984; Golden et al. 1992). One exception to this pattern has been a report of increased corticol responses to 5-hydroxytryptophan (Meltzer et al. 1984). There have been a number of studies using 5-HT1A agonists, such as ipsapirone and buspirone, as well as the 5-HT2A/2C agonist M-chlorophenylpiperazine. The studies that used serotonin receptor agonists have reported conflicting results, sometimes demonstrating impaired 5-HT1A function and other studies not replicating these findings (Lesch et al. 1990).

A related approach used to study the serotonergic system involves depletion of serotonin by either inhibition of tryptophan hydroxylase with parachlorophenylalanine (PCPA) (Shopsin et al. 1976), or diet (Delgado et al. 1994) to produce acute tryptophan depletion. These studies have shown that depression recurs in a few hours after serotonin or tryptophan depletion in remitted, medication-free, depressed patients. Similarly, patients who have responded to serotonin reuptake inhibitors have a transient return of depressed symptoms after acute serotonin or tryptophan depletion. Thus, the antidepressant action of these medications requires a continuous enhancement of serotonergic activity.

Platelet Serotonin-Related Studies

These studies have been very consistent in reporting reduced platelet serotonin uptake and less consistently reporting reduced serotonin transporter binding in medication-free, depressed patients, particularly in those with melancholia (Meltzer et al. 1981; Kaplan and Mann 1982). Other platelet indices have generated less consistent results. For example, monoamine oxidase activity has been reported to be low in bipolar patients and elevated in unipolar patients (Mann 1979), serotonin platelet content does not seem to show any consistent alteration (Mann et al. 1992a), and platelet 5-HT2A binding is increased, but apparently this effect is largely confined to individuals attempting suicide (McBride et al. 1994). A few studies have reported impaired serotonin 5-HT2A signal transduction in the platelet. The limitation of platelet studies is that they do not reflect the complex situation that may involve the same serotonin-related proteins in the brain. Although some genetic effects may be detectable in the platelet, the local regional brain combination of primary pathogenic effects and compensatory adjustments and modulations by other neurotransmitter systems cannot be detected in the platelet.

Postmortem Studies

Postmortem studies of the serotonergic system in depressed patients have been largely confined to studies of suicide. Because there is an association of serotonergic abnormalities with suicide risk, it is difficult to determine what effects are attributable to the presence of a major depression as opposed to a vulnerability to suicidal behavior. Nevertheless, Ferrier and colleagues (Ferrier et al. 1986) reported a study in depressed patients dying from causes other than suicide and found a trend for a significant increase in 5-HT2A receptor binding and a decrease in 5-HIAA, particularly in those patients with evidence of a major depression a month before death, as compared to controls. McKeith and colleagues (McKeith et al. 1987) replicated these findings using autoradiography, in terms of the 5-HT2A receptor. Stockmeier and colleagues (Stockmeier et al. 1998) reported an increase in somatodendritic 5-HT1A binding in the brainstem of suicides with major depression, but did not have a nondepressed, suicide control group to distinguish effects of suicide from those of major depression. Thus, there are data suggesting that not all of the changes reported in suicides may necessarily reflect the risk for suicide. Some of the neurochemical changes may, in fact, be attributable to the presence of major depression. Clearly, further studies of depressed patients dying from causes other than suicide would also be of considerable importance.

Functional Brain-Imaging Studies of the Serotonergic System

Only a few studies have used such modalities as positron emission tomography (PET) or SPECT to study the serotonergic system in depression. Ågren and colleagues (Ågren et al. 1991) have demonstrated reduced 5-hydroxytryptophan uptake in the brain of depressed patients. One study (D'haenen et al. 1992) found an increased number of ketanserin binding sites as demonstrated by 123I-ketanserin, using SPECT. We (Mann et al. 1996a) have reported a blunted regional glucose metabolic response to challenge with fenfluramine in depressed patients. Recently, Malison and colleagues have reported lower β-CIT binding sites in the brainstem of depressed patients (Malison et al. 1997). These studies avoid many of the pitfalls of postmortem studies and permit correlation with symptomatology at the time of the scan.

A SEROTONIN TRAIT

Depue and Spoont (1986) drew on animal and human data to describe a model where behavioral facilitation and inhibition systems were responsive to the serotonergic system, and the influence of the serotonergic system on these behavioral inhibition and facilitatory systems was not specific to any particular response system. This interesting formulation fits with data that we and others have reported linking serious suicidal behavior to a deficiency in the serotonergic system (see Mann 1998b for a review). Serotonergic function correlates with past and future serious suicidal behavior, and we have suggested that it is related to the diathesis or propensity for suicidal acts. Similarly, studies by Virkkunen and others (Virkkunen et al. 1994; Virkkunen et al. 1995; Ebert and Ebmeier 1996) showing a relationship between past and future impulsive, violent behavior and impaired serotonergic function, indicate the serotonergic activity may also modulate the threshold for externally directed aggression. Rodent studies have demonstrated that serotonin depletion results in increased aggression, and serotonin augmentation results in decreased aggression (Mann 1995). The 5-HT1B knockout mouse demonstrates increased impulsive aggression and sexual behavior. Consistent with a biological trait, it has been demonstrated that genetic factors account for a significant part of the variance in serotonergic function, and early maternal deprivation resets serotonergic activity, in an enduring fashion, at a lower level (Higley et al. 1993). This may explain why there is a link between maternal deprivation and child abuse with adult aggression and suicidal behavior. Studies of the serotonergic system in humans and nonhuman primates have demonstrated considerable stability over time in mature adults. There are correlations between CSF 5-HIAA and prolactin responses to fenfluramine with such behavioral traits as aggression and impulsivity (Brown et al. 1979; Brown et al. 1982; Coccaro et al. 1989; Coccaro et al. 1994). Thus, there is an impressive body of literature indicating that the serotonergic system is a relatively stable biochemical trait that correlates with behavioral traits, including impulsivity.

CONCLUSIONS

The history of the study of the serotonergic system in psychiatric disorders reflects our increasingly sophisticated measures and conceptualization of the relationship between neurobiology and behavior. Early studies of the serotonergic system were confined to blood and urine, and then later CSF and neuroendocrine challenges. Now there are direct studies of postmortem brain tissue and functional brain imaging studies. At another level, the serotonergic system was originally examined in relation to the syndrome of acute depression. More recently, we have recognized that this system is more of a biochemical trait and have identified its relationship to the vulnerability for recurrent depression and such traits as impulsivity. The very complexity of the serotonergic system means that it has the potential for a role in a variety of psychopathological conditions and, conversely, manipulating the serotonergic system has the potential for developing treatments for a wide range of psychopathological conditions. Thus, although a great deal of work has been done with regard to the serotonergic system, its complexity, in terms of anatomy, numbers of receptors, and its role in neurodevelopment, means that we again stand at a threshold of a new set of research opportunities.