Main

The impact factor (IF) is a commercial attempt to quantify and rank journal quality. While all should recognize that this is an imperfect measure that is affected by overall popularity of a field, the IF does provide an objective measure of the citation rate of the average published article in a specific journal over a 2-year timeframe.1 Specifically, the 2006 IF for any journal is a calculated ratio of the number of 2006 citations of articles published in 2004 and 2005, to the number of citable items (research papers and review articles) published in the same 2 years.

2006 journal IFs will be released by Thomson in June 2007. Undoubtedly editorial offices worldwide are anxiously awaiting the good (or bad) news. For Laboratory Investigation (Lab Invest), this year's IF is especially important since it will be the first IF reflecting the sole efforts of the present editors. There has been a long wait for this day as these editors have been guiding the journal since July 2003.

The purpose of this article is neither to reiterate the history of the IF (reviewed in Garfield2) nor to outline the potential shortcomings of the reference collection and calculation processes used for IF determination (eg, Whitehouse3 and Dong et al4). Numerous editorials have also been published that outline the intrinsic pitfalls of IFs, and these articles suggest that IF should not be used to guide the management of a journal or of scientific careers (well summarized in Seglen5 and Hecht et al;6 most recently discussed in Al-Awqati7). It is also important to note that IFs are not, and were never meant to be, a measure of the significance of any one paper published in a journal with a known IF,8 a determination of whether an author of a published paper deserves a grant, promotion or tenure,9 a method for evaluation of departments or institutions,10, 11 or an indication of which research influences health policy.12 Several methods other than IF have been proposed for ranking scientific literature (outlined in Whitehouse,4 Ball13 and Lehmann et al14) but they have not yet been as widely accepted as IF. Although some have suggested that instant online access to articles will make the IF obsolete, the founder of ISI and the creator of the IF predicted in 2001 that even in this age of online publishing, IFs are here to stay.15 At present, this prophecy appears to be true.

The purpose of this report is to examine whether useful information can be obtained about the evolving IF of a journal before the issuance of an official IF in June of the following calendar year. A key assumption is that IF is not the only measure of journal quality. Rather, IF reflects the trajectory of a journal, and may therefore serve as a reference point for the activities of an editorial team. In the case of Lab Invest, the editors have used IF as one objective measure, but not the sole measure, of their progress in elevating the quality and the utility of work published in the journal. Other measures include service to authors by means of fair and thorough peer review and rapid turnaround, and by tracking the quality and quantity of submitted papers. As IF is used as only one of many measures, editorial decisions are not driven by consideration of the effect of an action on IF. Instead, editorial efforts to elevate the overall quality of the journal are hopefully reflected in the IF. Hence, obtaining information on the evolving IF as early as possible has independent value.

Accordingly, this study reports the collection and analysis of citation data for Lab Invest and other top general pathology journals, with aim of predicting the IFs of these journals as much as a year in advance of the official issuance.

Methods

Data Collection

All data were obtained from the Thomson ISISM Web of ScienceĀ®. Access to the citation information requires a subscription. Predicted IFs were calculated for the following general clinical and investigative pathology journals: Lab Invest, American Journal of Pathology (Am J Pathol), Journal of Pathology (J Pathol), Modern Pathology (Mod Pathol), American Journal of Surgical Pathology (Am J Surg Pathol), and Human Pathology (Hum Pathol). The starting dates for the weekly collection of the citing reference data were as follows: Lab Invest, July 3, 2005; Am J Pathol, July 17, 2005; Modern Pathol, July 24, 2005; J Pathol, Am J Surg Pathol, and Hum Pathol, intermittently throughout 2005 and weekly from July 3, 2006. (These start dates reflect the growing curiosity of the editorial team about this project and its potential value.)

Calculation of Current IF

The ā€˜real-timeā€™ or current IF was calculated weekly as described for the official IF:2 namely, IF for year X=((number of citations in year X for papers published in year X-2)+(number of citations in year X for papers published in year X-1))/(the sum of the number of citable papers in years X-2 and X-1).

The weekly change in IF (Ī”IF) was calculated by subtracting the accrued IF of week Y-1 from week Y.

For example, two weeksā€™ calculations from Lab Invest are shown in Table 1. It is also necessary to know that Lab Invest published 159 citable items in 2004 and 128 in 2005, for a total of 287 over the 2 years. The number of items deemed ā€˜citableā€™ by Thomson ISISM are available on their web site under ā€˜Journal Citation Reportsā€™.

Table 1 Example of one week's calculations of Lab Invest citations. The citation numbers given are cumulative for that given year
Full size table

Calculation of Predicted IF

The predicted IF was calculated in two ways (Table 2) as follows. Method 1: for the journals with weekly IFs available from the previous year, the weekly accrued IF for 2006 was divided by the proportion of the actual previous year's official IF that had been obtained by that week in 2005. For journals without a full set of data for 2005, the data were extrapolated. Method 2: the mean Ī”IF as of that week was multiplied by the remaining weeks in the year and added to the current IF to obtain a predicted ā€˜year endā€™ IF. This value was then divided by the proportion of IF that had been accumulated by the end of the previous calendar year (data for which had been obtained for all six journals of interest).

Table 2 Calculation of predicted Lab Invest IF using methods 1 and 2 for three selected dates (see text)
Full size table

Relative Impact Ratios for Each Year of the IF

The relative impact ratios were determined for year X as follows: the number of citations in year X for the papers published in year X-1 were divided by the number of papers in year X-1, and the number of citations in year X for the papers published in year X-2 were divided by the number of papers in year X-2. The value obtained for year X-1 was then divided by the value obtained for year X-2 (which presumably would have accrued more citations owing to more exposure time). These data provided a ā€˜normalizedā€™ ratio of contribution by the two years being measured for the IF, namely, (year X-1)/(year X-2).

Results

IF Tracking

Table 3 shows a sample of the material obtained from the Thomson ISI Web of ScienceĀ® site over the course of 2006. It reveals how the IFs for three journals, J Pathol, Am J Pathol and Lab Invest developed in ā€˜real timeā€™, as citations were updated each week. There is some variation in the accrual of IF for every journal each week, most likely due the collection process. In other words, citation numbers would depend on which monthly and quarterly journals were analyzed by Thomson ISI and added to the database in any given week. The data in Table 3 do not specifically indicate what the official IFs for 2006 will be. However, it was apparent early that Lab Invest would remain in its current ranked position as a general Pathology journal, and that the higher-ranked J Pathol and Am J Pathol would be very close one to another.

Table 3 ā€˜Real-timeā€™ development of the 2006 IFs for J Pathol, Am J Pathol, and Lab Invest
Full size table

A sample weekly IF summary sheet (Table 4) shows the ā€˜real-timeā€™ IFs of each of six pathology journals on one date, December 22, 2006 (column 3), and compares them with those from the same day the previous year (column 2). Lab Invest accrued an IF of 3.895 as of December 22, 2006, which is 0.477 higher than the value attained on the same day the year before (3.418). Modern Pathol was slightly ahead of the previous (2.746 for 2006 vs 2.701 for 2005) and Am J Pathol was slightly behind (5.033 for 2006 vs 5.149 for 2005). (A complete 2005 data set had not been collected for the three remaining journals, so it was not possible to compare them in this manner.) Weekly data collection was continued after December 2006, but the accrual numbers dropped considerably after the start of new calendar year (6 weeks later) as most of the citing journalsā€™ 2006 issues had already been counted (data not shown).

Table 4 Summary of year-end IF accrual for six major pathology journals, for the calendar years 2005 and 2006
Full size table

The ā€˜Ī”IFā€™ columns (Table 4, columns 4ā€“7) refer to the average weekly positive change in IF, based on accrual of citations. The first and the second halves of the calendar year were analyzed separately, since citations were stronger in the second half of the year for all journals for both years. It can be seen that Lab Invest performed better per week in both halves of calendar year 2006, in comparison with 2005. The average weekly Ī”IF was 0.059 for the first half of 2005 and 0.070 for the first half of 2006; and the Ī”IF was 0.072 for the second half of 2005 and 0.084 for the second half of 2006. This type of improvement in the Ī”IF also was true for Modern Pathol, (0.048 vs 0.050 for the first half of 2005 vs the first half of 2006, and 0.058 vs 0.061 for the second half of 2005 vs the second half of 2006, respectively). In contrast, the Ī”IF values decreased in 2006 for both Am J Pathol and J Pathol, as shown in Table 4.

Predictions for the 2006 Pathology Journal IFs

The ā€˜predicted IF 06ā€™ values are based on comparison of the accrued vs actual IFs for 2005. Lab Invest should experience a substantial rise in IF to about 4.4. The IF for Am J Pathol should stabilize at about 5.7 and that for J Pathol will fall to about the level of Am J Pathol. Whether Am J Pathol or J Pathol will be the number one-ranked journal in the field of pathology for 2006 is not clear from this analysis. Modern Pathol and Hum Pathol should enjoy increased IFs, but it appears that the Am J Surg Pathol IF will drop.

Not reflected in Table 4 is the fact that two methods were used to calculate predicted IFs, and that they converged at the end of the calendar year. The first method took the ā€˜real-timeā€™ IF for the week and divided it by the percentage of the final IF that had actually been accrued by the same week in 2005. The flaw in this method is that if the timeline for accrual of IF changes drastically from year to year, the predicted values will not be accurate. Method 2 was based primarily on the Ī”IF and the values from this method seemed to run quite a bit lower than those for method 1 early in the year. The problem with method 2 is that the weekly Ī”IF varied considerably from the beginning to the end of the year, so that the central tendencies for weekly Ī”IF did not mature until the data set was nearly complete. The validity of either predictive method will only be established when the ā€˜officialā€™ IF for 2006 is issued.

Figure 1 shows the weekly 2006 IF predictions by both methods for Lab Invest, Am J Pathol, and J Pathol throughout the latter half of 2006. Several trends are obvious: (1) from the beginning, Lab Invest was third among the three journals, (2) Am J Pathol and J Pathol are very close to one another, although J Pathol seems to have had an early lead, (3) for all three journals, the predicted IFs stabilized and converged in late October, and (4) it appears to be possible to predict an approximate predicted IF in early July that is close to what would be predicted in December. What remains to be seen is if the predictions made in December are close to the official values that will be issued by the Thomson Corporation.

Figure 1
figure 1

Weekly predicted impact factors for Lab Invest, Am J Pathol, and J Pathol. Impact factors were predicted using method 1 (solid symbols) and method 2 (open symbols) as described in the text. ā€¢ Lab Invest; ā–¾ Am J Pathol; ā–Ŗ J Pathol.

Full size image

Breakdown of Contributions to IF (Normalized IF Ratios) from Each of 2 Years

The IF is made up of citation counts over a 2-year period. It is intuitive that for the IF for year X, year X-2 would contribute more than year X-1, because there is more time for the papers from year X-2 to be cited. But what is the normalized ratio of the contributions of each year to the IF? This question becomes especially important to new editors in the year of the ā€˜hybridā€™ IF, where the first year's contribution to the IF comes from a previous editorial group and the second year's contribution comes from the new editorial group. It is a simple matter to divide any year's IFs into annual contributions, as shown in Table 5, which analyzes Lab Invest for the years 2001ā€“2005. It becomes apparent that the high IF in 2003 was largely due to the strong citation rate of 2001 papers, which show the biggest year X-2 contribution in the table (column 5). Likewise, the low IF in 2004 was largely a result of the poor performance of the 2003 papers; although the 2002 papers were relatively well cited, the IF dropped precipitously (4.418 to 3.702).

Table 5 Normalized yearly impact factor ratios for Lab Invest, 2001ā€“2005
Full size table

Although this is a limited data set, the values for the ā€˜strongā€™ publication year (2001) and the ā€˜weakā€™ publication year (2003) suggest that a strong publication year remains a strong publication year for both years of IF. Likewise, a weak year will probably negatively affect the IF for 2 years.

Normalized impact ratios (year X-1 contribution to IF/year X-2 contribution to IF; column 6) are another way to look at the annual contribution to IF. A value that is higher than the central tendencies could suggest that year X-1 is much better than average and/or that year X-2 is much worse than average. It is necessary to view a chronological series of the normalized IF ratios to determine the cause. For example, with Lab Invest, the high ratio in 2002 (0.878) is due to the exceptional performance of the 2001 papers. The high ratio in 2005 (0.850; 2004/2003) is due to both the underperformance of the 2003 papers and the better-than-average citation rate of the 2004 papers. However, a strong publication year 2004 was not enough to overcome a lackluster 2003, and the official IF only climbed slightly (3.702 to 3.859). As of December 22, 2006, the normalized impact ratio for Lab Invest was 0.843 (2005/2004). Therefore, it appears that a strong publication year 2004 is being followed by an even better 2005, which is reflected in the positive prediction for the 2006 IF and bodes well for the 2007 Lab Invest IF.

This type of analysis can easily be performed for any journal. It is important to note that the normalized yearly IF ratios vary quite a bit from journal to journal (the analysis was also performed for Am J Pathol, J Pathol, and Mod Pathol, data not shown), so the values shown here for Lab Invest should not be considered representative of any other publication.

Discussion

Why go through the trouble of attempting to predict a journal's IF? First, the IF is still a common yardstick by which scientific journals are measured, even though the practice is hotly debated.5, 8, 11, 15 Second, since the official IF is issued only once a year and reflects the papers that were handled by an editorial group up to 4 years earlier, editorial groups are eager for reassurance that their hard work is reflected by an objective measure. This report advances the concept that a weekly evaluation of ā€˜real-timeā€™ IF progress, reviewed monthly or quarterly by editors, can provide valuable feedback. Although the data do not directly address this issue, this feedback may also limit the fatigue that can happen at the end of an editorial group's term (ā€˜running out of gasā€™). For example, the weak IF years Lab Invest endured in 2002 and 2003 were the seventh and eighth years of the previous editorial team. Rather, the excitement provided by regular review of IF ā€˜futuresā€™ may encourage editors to maintain the diligence required to keep editorial ā€˜dwell timeā€™ for papers to a minimum, to exercise the highest level of editorial objectivity for original scientific reports, and to ensure that reviewers are timely.

Few editorial groups have revealed that they take advantage of Thomson ISISM Web of ScienceĀ®. There is one report from the Editor-in-Chief of Cardiovascular Research, who in June 2005 published a figure correctly predicting both a small decline in their soon-to-be issued 2004 IF and the recovery of the IF the following year.16 The enterprising editors of Infection, Genetics and Evolution, a relatively new journal that will not have an official IF until 2009, recently calculated their ā€˜unofficial IFā€™ themselves and published it in an editorial.17 In both these instances, the best interests of the journal and prospective authors are served. Unfortunately, neither editorial group revealed their methodology for predicting IF. In this paper, a detailed methodology is provided and quantitative predictions are made.

As important as IF can be to a journal, it is vital not to let IF considerations drive every aspect of editorsā€™ decisions.18 To paraphrase the one-time editor of Archives of Disease in Childhood, ā€˜Does the journal exist to be read or to be cited?ā€™ (quoted in Smith, 2006).19 For society journals, the editors must also consider their duties to the parent organization.20 Lab Invest is an official journal of the United States and Canadian Academy of Pathology, and therefore needs to keep the interests of the member physician-pathologists in mind. The Lab Invest editors best serve submitting authors, the journal readership, and academy members by holding to the original fundamental operating premise, which is to select for publication high-quality original work that falls within the scope of the journal.

One might assume that abstract views on the journal website, and HTML and pdf downloads would be early indicators of highly cited papers. However, unlike what has been reported in the British Medical Journal,21 for Lab Invest there has been little correlation between the most-cited papers and those that received the most ā€˜web hitsā€™. There also seems to be no relationship between citation rates and cover articles, articles highlighted by cover bullets, articles highlighted by editorial ā€˜Inside Lab Investā€™ entries, and articles featured on the publisher's website, nature.com (data not shown). One could either conclude that the editors are clueless as to what the reading public actually is interested in, or that they have an uncanny ability to publish consistently high-quality papers regardless of whether they are highlighted by covers, editorial bullets, or the like. The editors will, of course, prefer the latter interpretation.

One sometimes overlooked aspect of IF is that the majority of the papers in any given journal do not contribute significantly to the IF. As has been previously described in detail for Nature,22 the Journal of Biological Chemistry, Biochimica Biophysica Acta, and the Biochemical Journal,5 only a small proportion of the papers published in Lab Invest, Mod Pathol, J Pathol, and Am J Pathol provide the greatest proportion of the impact (data not shown). While Lab Invest's executive editorial group has prospectively identified a number of papers that have become ā€˜high impactā€™, these prospective identifications represent only a subset of what are now considered the ā€˜bestā€™ papers the journal has published during our term. Therefore, in addition to being academically questionable, there is no data-based evidence to support editorial policies focused on selecting papers on the basis of their anticipated citation impact.

In contrast to the circumspect use of IF for editorial action, IF is a key factor for submitting authors in determining where to submit papers. In the field of pathology, the most cited papers tend to be published in high-impact disease- or organ-specific journals rather than pathology journals.23 But IF is not the only factor authors consider. Speed and fairness of the review process are also important, and authors may prefer ā€˜lower impactā€™ journals like Lab Invest that have a quick and fair review process, free of undeservedly harsh reviews and uncertainties that engender delays in processing. From an operational standpoint, a priority of this group of Lab Invest editors has always been to serve the authors submitting their manuscripts for review. The journal's average time-to-initial-decision has been in the 11 to 13-day range during this editorship. An initial decision to reject on the basis of failing to meet the scope or quality of the journal is made on approximately 58% of submitted manuscripts, usually within 5 days of submission. For papers selected for full scientific review, an initial decision is almost always made within 30 days; 63% of the papers in this category were eventually accepted for publication. Total ā€˜processing timeā€™ in the editorial office, both front end and back end, is usually less than eight calendar days and decision letters are clear and reviewersā€™ comments constructive. These editorial policies ensure the best outcomes for authors. The authors of rejected papers have the opportunity to proceed quickly with alternate plans (from a survey of rejected manuscripts from January 2005, 90% were eventually published elsewhere before the end of 2006). The acceptance rate for revised resubmitted manuscripts is 93%, which indicates the efficacy of the review process. Accepted papers are then published online in an average of less than 30 days.

The conclusion of this report is that IF can be predicted a year in advance of the official date of issuance by Thomson ISISM. At the very least, the weekly ā€˜ticker tapeā€™ of IF evolution is a source of great entertainment for the editorial team (and for a parent society and the publishers). Far more importantly, early IF data help validate editorial policies for journal management as a whole, and can provide reassurance that the philosophy for journal operations is on track. In the absence of these early data, an editorial team really finds out about its performance only on the eve of retirement.