Introduction

Interest in the role of the 5-HT receptors in the control of anxiety first arose when clinical trials revealed an anxiolytic-like effect of the non-selective 5-HT2 antagonist ritanserin in humans1. Since this time a number of preclinical trials have been carried out, the majority of which have employed non-selective ligands. However, an increasing awareness of the complexity of the 5-HT receptor family2 coupled with a number of equivocal findings for non-selective 5-HT agents has led to great importance being placed upon the need to test the role of specific 5-HT receptor subtypes in regulating anxiety.

Although preliminary studies employing animal models of anxiety have, in general, demonstrated that a reduction in 5-HT neurotransmission plays a role in the effects of novel non-benzodiazepine anxiolytics3, equivocal results for the effects of 5-HT subreceptor ligands have been reported, particularly in studies using the elevated plus-maze paradigm (EPM)4. Inconsistent behavioral profiling of drugs that modulate the 5-HT system led earlier investigators to query the utility of the EPM paradigm5; specifically, the predictive validity of the EPM appears to be limited to benzodiazepine-related compounds6.

The elevated zero-maze (EZM) paradigm represents an important improvement over the EPM in that it removes any ambiguity in the interpretation of time spent on the hub of the EPM and permits continuous exploration7,8,9. The present paper will compare data obtained in our laboratory using 5-HT2, 5-HT3 and 5-HT4 receptor ligands tested using the EZM paradigm with the results from other studies that have utilized similar receptor ligands in a variety of preclinical models of anxiety. Specifically, we tested the effects of the 5-HT2C antagonist RS-102221, the 5-HT2C agonist MK-212, the 5-HT3 antagonist Y-25130, the novel 5-HT3 agonist SR 57227A, the selective 5-HT4 antagonist RS 39604 and the high-affinity 5-HT4 partial agonist RS 67333.

The roles of 5-HT23, 5-HT310 and 5-HT4 receptors11 have been extensively reviewed in a variety of animal models of anxiety. In addition, comprehensive reviews4,12,13,14 have examined, inter alia, strategies for experimental modeling of anxiety, the validity of rodent models of anxiety and discussed the progression of such paradigms.

Results

Combined descriptive statistics for the study variables including time spent on open arms, open arm entries, head dips, SAP, risk assessment and rearing duration/frequency are presented in Table 1 along with the correlations between aggregated study variables. Tables 2,3,4 respectively include the specific results for the three experiments. Finally, Table 5 includes a summary of the direction of statistically significant findings across all three experiments.

Table 1 Correlations between study variables and descriptive statistics for combined experiments
Full size table
Table 2 Results from Experiment One
Full size table
Table 3 Results from Experiment Two
Full size table
Table 4 Results from Experiment Three
Full size table
Table 5 Summary of significant findings across three experimental studies relative to saline controls
Full size table

Experiment 1: 5-HT2 receptor ligands

Analyses revealed significant variation in SAP, head dips and risk assessment behaviors across the various doses of diazepam. Conversely, diazepam did not lead to significant variation in time spent on the open arms of the maze, number of entries into the open arms and duration and frequency of rearing behaviors. Follow-up Mann-Whitney U tests revealed an increased number of head dips for the highest two doses (U = 22.0, p < 0.05; U = 2.0, p < 0.01). All three doses of diazepam were associated with reductions in SAP (U = 8.5, p < 0.001; U = 16.0, p < 0.01; U = 5.5, p < 0.001) and risk assessment (U = 13.0, p < 0.01; U = 18.0, p < 0.05; U = 13.0, p < 0.01).

RS 102221 dose conditions produced significant variation with respect to time spent on open arms, number of open arm entries, SAP, head dips and risk assessment. Follow-up post-hoc analyses revealed that all doses of RS 102221 were associated with reduced SAP (U = 9.5, p < 0.01; U = 3.0, p < 0.001; U = 2.0, p < 0.001) while only the highest dose of 2.0 mg/kg was associated with increased time on the open arms (U = 22.5, p < 0.05) and head dips (U = 10, p < 0.01). Analyses revealed a significant reduction in risk assessment at both the 1.0 mg/kg (U = 17.0, p < 0.05) and 2.0 mg/kg (U = 13.0, p < 0.01) doses.

Significant variation was observed across MK-212 dose conditions for rearing duration and rearing frequency. All doses were linked to decreased rearing duration (U = 20.0, p < 0.05; U = 23.0, p < 0.05; U = 5.0, p < 0.001) with the 0.25 mg/kg and 0.75 mg/kg doses being also linked to decreased rearing frequency (U = 15.0, p < 0.01; U = 2.0, p < 0.001).

Experiment 2: 5-HT3 receptor ligands

Analyses revealed that the highest dose of diazepam (0.75 mg/kg) significantly decreased the duration of risk assessment behavior (U = 16.0, p < 0.01). The highest two doses of diazepam (0.5 mg/kg and 0.75 mg/kg) were associated with decreased SAP (U = 22.0, p < 0.05; U = 17.5, p < 0.05) and the 0.5 mg/kg dose was also associated with increased head dips (U = 23.5, p < 0.05).

While no significant differences were observed for the 0.1 mg/kg and 3.0 mg/kg doses of Y-25130, a number of effects were observed for the moderate 1.0 mg/kg dose. Specifically, the 1.0 mg/kg Y-25130 dose was associated with significantly increased time spent in the open arms (U = 17.0, p < 0.05) and a greater number of open arm entries (U = 17.5, p < 0.05). Additionally, this dose was associated with decreased SAP (U = 23.0, p < 0.05), decreased risk assessment (U = 16.0, p < 0.01) and increased head dips (U = 23.0, p < 0.05).

Mann-Whitney U analysis revealed that the moderate dose of SR 57227A (1.0 mg/kg) was associated with a significant increase in risk assessment (U = 16.0, p < 0.01). No other statistically significant effects were observed across the 0.1 mg/kg and 3.0 mg/kg doses of SR 57227A.

Experiment 3: 5-HT4 receptor ligands

Post-hoc analysis revealed that both the 0.5 and 0.75 mg/kg doses of diazepam significantly increased time spent on open arms (U = 11.0, p < 0.01; U = 13.0, p < 0.01), number of open arm entries (U = 23.5, p < 0.001; U = 23.5, p < 0.05) and number of head dips (U = 9.0, p < 0.01; U = 19.0, p < 0.05), while significantly decreasing duration of risk assessment behavior (U = 8.0, p < 0.01; U = 18.0, p < 0.05). Additionally, the 0.75 mg/kg dose was linked to decreased rearing duration (U = 21.5, p < 0.05).

Mann-Whitney U analysis revealed that all doses of RS 39604 were associated with significant increases in time spent on open arms (U = 1.0, p < 0.01; U = 8.0, p < 0.01; U = 6.0, p < 0.01) an increased number of open arm entries (U = 14.3, p < 0.01; U = 15.0, p < 0.01; U = 6.0, p < 0.01), increased head dips (U = 5.0, p < 0.01; U = 13.5, p < 0.01; U = 12.5, p < 0.01) and significantly decreased risk assessment (U = 3.0, p < 0.01; U = 5.0, p < 0.01; U = 5.0, p < 0.01).

Analyses revealed that all doses of RS 67333 were associated with an increased number of open arm entries (U = 9.0, p < 0.01; U = 23.0, p < 0.05; U = 23.5, p < 0.05) and decreased risk assessment duration (U = 3.0, p < 0.01; U = 6.0, p < 0.01; U = 20.5, p < 0.01). Furthermore, both the 0.01 mg/kg and 0.1 mg/kg doses of RS 67333 were linked to increased head dips (U = 10.5, p < 0.01; U = 12.0, p < 0.01) and time spent on the open arms of the maze (U = 13.0, p < 0.01; U = 11.0, p < 0.01).

Discussion

While both the 0.5 and 1.0 mg/kg doses of the 5-HT2C antagonist RS 102221 failed to significantly modify spatio-temporal measures in the current experiment, the 2.0 mg/kg dose of this drug significantly increased time spent on the open arms of the EZM. This pattern of effects suggests a possible anxiolytic role for higher doses of RS 102221. Consistent with this interpretation, an ethological measure indicative of reduced anxiety (i.e., head dips) was significantly increased only at the 2.0 mg/kg dose. While one earlier experiment failed to find any anxiolytic effects for the 5-HT2A/2C antagonist ketanserin3, our outcome is consistent with an earlier report of an anxiolytic-like effect of the 5-HT2B/2C antagonist SB 200646A27 and other reports of anxiolytic-like action of 5-HT2C blockade in animal models of anxiety28.

An examination of traditional measures of anxiety revealed a general lack of effect for the selective 5-HT2C agonist MK-212. However, MK-212 was found to significantly reduce the duration of rearing at all doses and to significantly reduce the frequency of rearing at both the lowest and highest doses tested. Factor analysis has found rearing to load on a ‘motor activity’ factor in the EPM29 suggesting the possibility that the general lack of effects observed after MK-212 administration may have been due to behavioral inhibition/sedation. Overall, these findings are in agreement with a number of previous studies which also report a general lack of effect of 5-HT2 agonism in animal models of anxiety3,30,31.

The finding that the 1.0 mg/kg dose of Y-25130 increased both time spent on the open arms and total arm entries is in agreement with previous studies examining the effects of 5-HT3 antagonism on traditional measures in both the EPM and EZM32,33 and is compatible with an anxiolytic-like profile comparable to that obtained with diazepam. Consistent evidence for this interpretation was also observed from ethological measures with the 1.0 mg/kg dose leading to decreased SAP and risk assessment in conjunction with increased head dips. This latter finding is also in agreement with previous studies including work with the EPM34, social interaction test35 and light-dark test17, in suggesting an anxiolytic effect of 5-HT3 antagonists in animals. However, it should be noted that these effects were not observed for the smaller (0.1 mg/kg) and larger (3.0 mg/kg) doses suggesting the possibility of a hormetic dose-response curve.

While the 5-HT3 agonist SR 57227A was found to significantly modify most anxiety-related behaviors as measured in the EZM, examination of the effects of specific doses revealed a general lack of significant effects when compared to the saline control condition. The 0.1 mg/kg dose of this drug failed to modify any behavioral measures, although the general pattern of results, which include non-significant increases in behavior associated with both anxiogenesis and anxiolysis, would suggest non-specific effects.

The 5-HT4 antagonist RS 39604 significantly increased both time spent on the open arms and total arm entries in a manner consistent with anxiolysis. These findings are in agreement with those of Silvestre and colleagues11 as well as Kennett and colleagues28,36 who previously reported an anxiolytic-like effect of the 5-HT4 antagonists GR 113808, SB 204070, SB 204070A and SB 207266A on traditional measures in the EPM. With respect to ethological measures, RS 39604 was found to increase head dips and reduce risk assessment, but failed to modify SAP. Overall, RS 39604 was found to modify less anxiety-related behaviors than diazepam in the current experiment. This finding is again in agreement with the work of Silvestre et al.11 and Kennett et al.28,36, both of whom reported that the effects of 5-HT4 antagonists were smaller than those of either diazepam or chlordiazepoxide in the EPM, leading both authors to conclude that 5-HT4 antagonists have modest anxiolytic-like activity when compared with the benzodiazepines.

The 5-HT4 agonist RS 67333 was found to increase both time on the open arm of the EZM and to increase the number of entries onto open arms, a profile consistent with anxiolysis. This interpretation is further supported by ethological measures (i.e., all doses resulted in a reduction in risk assessment with the 0.01 mg/kg and 0.1 mg/kg doses were associated with an increase in head dips). While this finding is in agreement with previous findings in suggesting that 5-HT4 agonism is associated with a decrease in anxiety-induced behavioral inhibition37, the result is surprising in light of the finding that 5-HT4antagonism led to a similar anxiolytic profile. One possible explanation for this seeming paradox is a non-rectilinear relation between 5-HT4 activity and anxiety. An inverted-U shaped anxiolytic profile would be sensitive to 5-HT4 disruptions whether they were in the form of increased or decreased 5-HT4 activation. However, further research will be necessary to develop this possibility.

In conclusion, results from this study indicate a potential role for 5-HT2 and 5-HT3 antagonists and an apparently paradoxical role for both 5-HT4 agonists and 5-HT4 antagonists in producing anxiolytic-like effects on rats tested in the EZM paradigm. Such results from the use of the EZM can be compared to data from the EPM that is considered comparable following diazepam challenge, although the more direct measure of time spent in the open may represent an advantage of the elevated zero-maze7.

The results for the 5-HT2, 5-HT3 and 5-HT4 antagonists employed in our experiments are consistent with the original hypothesis for the role of serotonin in the pathogenesis of anxiety that was predicated on an association between a reduction in 5-HT turnover and the anxiolytic effects of benzodiazepines suggesting that a reduction of 5-HT neurotransmission results in an anxiolytic-like effect, whereas increased activity produces an anxiogenic-like effect38. Notwithstanding this conclusion, the behavioral pharmacology of 5-HT receptor ligands is often more inconsistent than the effects of standard anxiolytics and not all results are explainable in terms of the classic hypothesis, as evinced in our third experiment that demonstrated an anxiolytic-like effect with a compound that increased serotonergic activity.

Moreover, it should be noted standard anxiolytics (e.g., diazepam) sometimes produce counter-intuitive results as well such as the finding in Experiment 2 (see Table 3) that the low dose of diazepam was associated with less time spent on the open arms of the EZM than the saline control condition. Given that data for these groups were collected over a time span of 18 months, we postulated that such variations may, in part, be due to circadian rhythm effects. Benzodiazepines, such as diazepam, are thought to act by potentiating the action of the neurotransmitter γ-aminobutyric acid, which in turn, has been linked to regulation of the sleep-wake cycle39. Hence, the variations in responses to diazepam may be the result of the benzodiazepine acting at varying levels of GABA that mediate the generation of circadian rhythms.

A dual role for 5-HT in the mediation of different types of fear has been posited by Graeff and Zangrossi40. Specifically, serotonin may either enhance or reduce anxiety-like behavior depending upon the receptor subtype involved. Indeed, the fact that selective serotonin reuptake inhibitors are efficacious in the treatment of generalized anxiety disorder and in panic disorder indicates that there are conditions in which increased 5-HT activity can reduce anxiety.

Methods

Animals

All work was licensed by the Home Office via the Northern Ireland Department of Health, Social Services and Public Safety in accordance with the UK Animals (Scientific Procedures) Act 1986 and was in full compliance with Queen's University Belfast's Policy on the Use of Animals in Research and Teaching. In each of the three experiments, one hundred male Sprague-Dawley rats were randomly assigned as follows: saline control (n = 10), diazepam (n = 30), 5-HT antagonist (n = 30) and 5-HT agonist (n = 30). Animals were supplied from a breeding stock at Queen's University and weighing between 280 and 350 grams were group-housed (5 per cage; cage size 51 × 39 × 19 cm) in a temperature-controlled environment 22 ± 1°C under a 12-hour reverse light-dark cycle (lights off at 0800 hours) for four weeks prior to testing and prior to experimentation has been handled only for routine husbandry. Food and water were available ad libitum. All animals were both drug and experimentally naive.

Apparatus

The design of the maze employed was based on that originally proposed by Shepherd et al.9 and consists of a black Perspex annular platform, 105 cm in diameter and 10 cm wide, elevated 65 cm above the ground. The maze was divided into four equal quadrants, two of which were enclosed with black Perspex walls, 27 cm in height. The walls were on both the inner and outer sides of the platform so as to provide the ‘closed’ areas of the maze. The ‘open’ arms had no walls, but did have a 1 cm lip which acts as a tactile aid to animals when on the open areas of the maze. Lighting was provided by two 60 watt red ceiling lights at a height of 2 m positioned directly over the maze and one 60 watt red bulb positioned closer to the maze. Two cameras poised at 45-degree angles were positioned 1 m above the ground and 1 m from either side of the maze. The cameras were connected to a multiplexer in an adjacent laboratory in order to prevent distractions and provided continuous recording of behavior for subsequent analysis.

Drugs

All of the active compounds–MK-212 (0.25–0.75 mg/kg), RS 102221 (0.5–2.0 mg/kg), SR 57227A (0.1–3.0 mg/kg), Y-25130 (0.1–3.0 mg/kg), RS 67333 (0.01–0.5 mg/kg), RS 39604 (0.01–1.0 mg/kg) and the positive comparator, diazepam (0.25–0.75 mg/kg), were obtained from Tocrais Cookson UK. The dose ranges for the drugs were chosen from the experimental pharmacology literature examining the relation between 5-HT and anxiety3,4,15,16,17,18,19,20. All compounds were prepared as solutions or suspensions in physiological vehicle. Saline alone was administered to control animals and diazepam was used as a positive control. All solutions were freshly prepared on the day of testing and administered intraperitoneally (1 ml/kg) 30 minutes prior to testing.

Procedure

All testing was conducted between 0800 and 1400 hours. On the day of testing one animal was selected and transferred in a covered box (to maintain darkness) measuring 32 × 16 × 12.5 cm to a room adjacent to the experimental room. Here a subcutaneous (s.c) injection was administered under red light and the animal was placed in a cage measuring 40.5 × 28 × 12 cm containing only sawdust bedding (i.e., no food and water). The animal remained in this staging cage for 30 minutes in order to allow adequate absorption of drug, after which time the animal was transferred to the experimental room (again in a darkened box) and placed on the zero-maze at a junction between an open and closed arm, facing into the closed area. Each animal was placed on an alternative side of the maze so as to avoid any bias. All testing was conducted under red-light. After 5 minutes, the animal was removed from the maze and the maze was washed using an ethanol/water solution so as to avoid olfactory cues transferring from one experimental session to the next. Once testing was complete, computer-based event recording and ethological analysis software (Hindsight ver. 1.5) was used to register the relevant experimental variables. Videotapes were scored blindly by a highly trained observer (intra-observer reliability > 0.8). Behavioral parameters comprised traditional spatio-temporal, locomotion, maintenance and ethological measures21,22,23, which are outlined below.

Measures

Spatio-temporal measures

Traditional spatio-temporal measures included (a) time on open arms and (b) frequency of open arm entries. The animal was judged to enter into the open arm area of the maze when all four paws crossed the open arm threshold. After crossing into the open arm area, the animal was allowed to have one paw in the closed area of the maze and still be considered in the open area; however, if two paws crossed the threshold, then they were judged to have re-entered the closed area of the maze9,24.

Ethological measures

Stretched Attend Posture was defined as the frequency with which the animal makes a forward elongation of the head and shoulders followed by a retraction to the original position when the animal is on an open arm23. Head dips measured the frequency of the animal protruding its head over the ledge of an open arm and down towards the floor25. Finally, Risk Assessment was scored as the frequency of an animal exiting a closed arm of the maze with forepaws and head only and investigating the surroundings. Risk assessment was scored independently of stretch attend posture, which often accompanied risk assessment21.

Locomotion and maintenance measures

Rearing Frequency was defined as the number of vertical movements against the side or end of walls of the closed arms of the maze or raising up its hind paws sans support in the open area of the maze26. Rearing Duration measured the time each animal was engaged in rearing behavior.

Analysis

Data for each behavioral element were grouped according to treatment and were initially checked for normality and homogeneity of variance. Levene's tests were statistically significant (i.e. assumptions that distributions were normal or variances were equal had to be rejected); therefore, non-parametric tests were conducted (as opposed to data transformation). Data were analyzed using Kruskal-Wallis nonparametric one-way analysis of variance across treatment groups. Where significant variations in the data were identified, post-hoc Mann-Whitney U comparisons with control condition were performed. In all cases, a standard significance level α was set at p = 0.05.