The number of citations to published manuscripts are important for many reasons. For authors, the number of citations to their articles influences the authors Hirsch or H-index, a measure of a scientist’s personal impact and often used in the assessment for promotion and is the highest number of publications of a scientist who received h or more citations each while the other publications have not more than h citations each.1 For journals, the number of citations to published articles is used to calculate the Impact Factor (IF).2 The IF is one attribute of a journal that authors consider when determining to which journal to submit their manuscript. It is also viewed as indicating the importance of the journal. Thus, a journal, desiring the submission of the “hottest” articles and seeking to raise its profile, is interested in having a high IF. Difficulty in finding reviewers for a submitted manuscript may be due to a lack of the readership’s interest in the subject area of the manuscript (outside the scope of the journal) and therefore might indicate a future lack of citations.

Therefore we tested the following hypotheses: (1) poorly cited manuscripts would have a higher number of invited reviewers and (2) other attributes that increase the ability to “find” the article would be associated with a higher number of citations.

Methods

Manuscripts published in 2017 and 2018 in Pediatric Research were collected from the Springer-Nature database.

Three types of manuscripts were included in the analysis: original research articles, review articles, and special articles. The original research articles were categorized as basic science, clinical research, or population study (Table 1). For each article, the following factors were examined: the number of citations a manuscript received in 2019 (obtained from Springer-Nature), total reviewers invited (obtained from ScholarOne database), types of peer reviewer responses received (from ScholarOne), title features such as title length, inclusion of medical subject headings (MeSH) keywords and additional terms using MeSH on Demand (https://meshb.nlm.nih.gov/MeSHonDemand).

Table 1 Characteristics of 482 manuscripts reviewed.
Full size table

The ScholarOne database indicated five different responses from invited reviewers, some indicating an active response from the reviewer (Accept, Decline, Unavailable) and others indicating the invitation had timed out in ScholarOne (No response). Unassigned indicated that a reviewer was uninvited due to the review no longer being needed.

Data including the number of words in the titles and the number of MeSH terms and additional terms were obtained by locating the manuscripts on the ScholarOne database. Titles were collected from the manuscripts’ cover pages. The abstract was cut and pasted into MeSH on Demand. The number of MeSH terms and additional terms that were found in the abstract of the manuscript were counted.

This is an exploratory paper with the objective of examining any relationships between non-scientific features of manuscripts published in print between 2017 and 2018 and the number of citations in 2019. These features include those related to the acceptance history of the manuscript, such as total number of reviewers contacted and number of reviewers declining to review as described above, and features of the manuscript itself, such as length of title and number of MeSH terms in the abstract. While we present no formal hypothesis, we have an interest, in particular, in the number of times reviewers decline to review and the related number of reviewers invited, and their possible relationship to the subsequent number of citations.

Manuscripts with complete data were retained for analysis. We took these approaches: examining whether there is a numerical cut point (e.g. ≤3, >3) in the number of citations that is discriminatory for the other variables and a subset analysis comparing those manuscripts with no citations with those with five or more, assuming if any differences existed, we would be most likely to detect them in groups formed by the extremes of the observed values.

Continuous variables are described with medians and ranges and subsequently divided into ordinal categories described with frequencies and percents. These categories were determined for each variable individually based on its distribution. Nominal variables are described with frequency and percents. Wilcoxon rank-sum tests were used for two-group comparisons of continuous variables and chi squared analysis and the Cochran–Armitage test for trend for two-group comparisons of nominal variables. All analyses were carried out using SAS v9.2 (The SAS Institute, Cary, NC). Level of significance was set at 0.05.

Results

Five hundred and one manuscripts were identified and 482 were included in the analysis (96.2%). The 19 excluded manuscripts were not in print by the end of 2019.

Characteristics of the manuscripts are described in Table 1. Manuscripts received a median of two citations with a range from 0 to 21 citations. Manuscripts had a median of two reviewers agreeing to review as is expected given that invitations are usually sent until two reviewers agree to review. Exceptions may occur when an associate editor does the review him or herself. There were as many as 18 reviewers invited (median of 5) with as many as 10 declining (median of 1).

Figure 1 shows the number of invited reviewers for each article published in 2017–2018 versus the number of citations in 2019. While this shows that the range of citations narrows with increasing number of invited reviewers, there was no obvious cut point in the distribution that indicates articles with fewer citations. Statistical analysis of varying cut points also did not reveal a value useful for distinguishing articles with more citations vs. reviewers invited or reviewers who declined (data not shown). One hundred and nineteen manuscripts had no citations and 90 had ≥5 (Table 1). There were no differences between these two groups in reviewers invited or who declined. However, the number of words in the title and number of additional terms from Mesh on Demand showed promising results.

Fig. 1: Relation of number of invited reviewers to number of citations.
figure 1

Scattergram of number of citations in 2019 for articles published in 2017 and 2018 versus number of reviewers invited.

Full size image

Figure 2 shows the relationship of the number of words in the title of manuscripts, dichotomized at the overall median of 14 words, with no citations and those with 5 or more. Of the manuscripts with ≤14 words in the title, 50.9% had 0 citations and 49.1% had ≥5. Of those with >14 words, 62.8% had 0 citations while 37.2% had ≥5 (p = 0.12). Despite not reaching statistical significance, it appears that a greater number of words in the title may negatively impact the likelihood of citation.

Fig. 2: Number of words in the titles of articles receiving no citations and articles receiving 5 or more citations.
figure 2

Comparison for 0 citations to ≥5 citations for manuscripts with ⋜14 words in the title, and for manuscripts with greater than 14 words in the title.

Full size image

The number of MeSH terms in the abstracts between those articles with no citations and those with ≥5 citations was not significantly different. However, having any additional terms listed in Mesh on Demand significantly increased the probability of that article having five or more citations (Fig. 3). Of the manuscripts with no additional terms, 65.2% had 0 citations and 34.8% had ≥5. Of those with at least one additional term, 48.6% had no citations and 51.3% had ≥5 (p = 0.02).

Fig. 3: Number of words in the titles of articles receiving no citations and articles receiving 5 or more citations.
figure 3

Comparison for 0 citations to ≥5 citations for manuscripts with no additional terms compared to manuscripts with at least one additional term on Mesh on Demand.

Full size image

Discussion

Our study showed that the number of reviewers invited before two reviewers agreed to review the manuscript was not an early indicator of citations. This is an unfortunate finding as an early indicator of no citations would allow editors to reject such a paper without review and save both editors and reviewers the burden of peer review. However, we were able to find two attributes which were associated with a higher number of citations—the number of words in the title and the number of additional terms listed in Mesh on Demand.

How to write a catchy title has been the subject of much discussion. Some articles discuss using “a hook, keywords and a source” and making the title playful. (https://www.wikihow.com/Find-a-Catchy-Title-for-Your-Paper/Essay) Another source suggests not too short but not too long, and don’t use a question (https://www.aje.com/en/arc/choosing-catchy-title-your-scientific-manuscript/) Based on our data, we have limited titles of manuscripts submitted to Pediatric Research to 14 words or less and have provided authors with a link to resources on writing titles. In addition, editors are asked if they approve of the titles following peer review with the option of requesting that the authors change it based on editorial suggestion.

Surprisingly, we did not find a relationship between the number of MeSH terms in the abstract and the number of citations. However, the presence of any additional terms listed on the search results from Mesh on Demand increased the probability of ≥5 ciations. Upon investigation, Mesh on Demand uses the Medical Text Indexer (MTI) program from the National Library of Medicine. This program ranks MeSH terms and also adds supplementary concepts, which are the additional terms. We had assumed that the “findability” of the article would be reflected by the number of MeSH terms which is how PubMed searches for articles. However, Springer-Nature, which collects data on where viewers of articles come from, provided us their data (Table 2). Surprisingly, most viewers of articles are directed by Google to our site. The search engine for Google is proprietary, but it may be that the additional terms are more “findable” by Google. It is difficult to know exactly how to edit an abstract so that it will contain these additional terms other than trial and error. We will provide this information in our Instructions for Authors and will notify authors of the presence of additional terms in a Mesh on Demand search.

Table 2 Visitor referral.
Full size table

In summary, rather than finding a way to reject papers, we found two ways to assist authors in making their articles more citable. We are making changes in our peer review process to ensure that authors take advantage of these suggestions.