Background

In 1990, Shirley Glasstone Hughes, dentist, researcher and BDA member, left her legacy to a charitable Trust named 'The British Dental Association Shirley Glasstone Hughes Memorial Prize for Dental Research'. The Trust was established 'for the purpose of providing in each year such prizes or scholarships for Dental Research as the Trustees shall in their absolute discretion decide', as stated in the deed.

Shirley Glasstone was a very active researcher and eleven of her papers published between 1952-19731,2,3,4,5,6,7,8,9,10,11 were identified through PubMed®, as well as a single publication12 by Shirley Glasstone Hughes in 1983. Shirley Glasstone's work in embryology and tooth development is regarded as laying the foundations of bioengineering tooth structures in dentistry.13 Ahead of its time, her work did not receive much recognition until 1996.

Since 1991 the Shirley Glasstone Hughes (SGH) Memorial Fund has provided grants for dental research. To date, the SGH fund has sponsored 41 research projects worth a total of £678,223.49. However, the relevance and importance of this research to primary dental care has never been assessed.

In 2004, the SGH trustees made the decision to evaluate the quality of research output from the Trust spending, and assess its relevance to primary dental care practice. The trustees also wished to learn whether the process and governance of distribution of the Trust money could be improved in any way.

Previous application and selection process

General information about the previous application and selection process for research grants from the SGH fund was obtained through discussions with BDA staff members who had previously been responsible for providing administrative support for the scheme (referred to from here onwards as the 'SGH administrator').

Applications for SGH grants

In December of each year, an article about the SGH fund was published in the BDJ. The article, written by the SGH administrator, would ask for applications for research grants from the SGH fund for the following year. Interested individuals were sent an application pack and completed application forms were returned to the BDA. The deadline for applications was usually April.

The SGH Panel

In May of each year, a meeting of the SGH Panel was organised by the SGH administrator. The panel comprised six to eight nominated members and each panel member (including the chair) served for a total of four years. After each four-year term, new members were nominated to replace 'retiring' panel members. The system was a rotating one, so that the panel was constantly changing. The agenda for each meeting included grant applications received that year and during the meeting the panel would decide which applicants should be awarded grants. This was usually done through a scoring system. Agendas also included receipt of annual reports from current grant recipients and a financial report from the BDA Finance Director.

Correspondence with SGH grant applicants/recipients

After the panel meeting each year, the SGH administrator would notify applicants of the outcome of their applications. Annual reports from current grant recipients would be acknowledged by letter, but no specific feedback would usually be given unless the panel had specific concerns about the project. Each year, cheques were sent to current grant recipients, accompanied by a brief letter from the SGH administrator. Any other correspondence between grant recipients and SGH panel members was usually coordinated by the SGH administrator but was on an informal ad hoc basis.

Aims

The aim of the study was to assess the research projects sponsored by the SGH fund and answer the following question: 'Is the SGH fund being used to sponsor high quality research which is of relevance to primary dental care?'

The project tried to answer this question by evaluating the relevance of the funded research to primary dental care, its scientific quality and impact on the research community and the regard recipients felt for the SGH funding stream.

Methods

Relevance to dental practitioners

An Omnibus Survey was circulated at the beginning of October 2005 to 1,500 BDA members in all fields of practice (the sample excluded students, retired members and members overseas). After one reminder circular, a 51% response rate was achieved (762 replies). Respondents who were not currently working in dentistry or retired were excluded from further analysis, so results are based on a sample size of 744. The survey contained questions on a wide range of topics and included a section on the relevance of research themes. Respondents were asked to indicate the relevance of each of 17 SGH-funded research themes by marking one of five options (very relevant, relevant, neither relevant nor irrelevant, irrelevant and very irrelevant).

Recipient perception of the SGH fund

In order to establish the regard recipients felt for the SGH funding, postal questionnaires were sent to 33 of the 41 grant recipients (no identification details were available for eight recipients). The questionnaire asked recipients to confirm project details and list publications.

Identification of published manuscripts and other media

Authors and research subjects were identified using the archive files from the SGH fund. The application forms for 24% (10) of the 41 projects were missing from BDA archives as well as 44% (18) of the annual reports. In some cases, BDA files contained only the abbreviated names of recipients and in two cases (Britton, 1992 and Mason/Callaghan, 1994) only the surname could be identified. PubMed® and ISI Web of Science® (WOS) search engines were used to find manuscripts that had similar subjects to SGH project titles/descriptions and authors that matched recipient and/or supervisor names. Abstracts were found through ISI Proceedings® and the International Association for Dental Research (IADR) database. Publication results were noted and cross-checked against information obtained from the recipient questionnaires (see above). Reprints of all identified papers were thoroughly searched for publication details including author names, academic affiliation, publication date and funding acknowledgement. Papers that matched project descriptions but did not acknowledge the SGH fund and were not listed by respondents to the questionnaire were excluded from the sample.

Relevance to the scientific community

Both ISI WOS® and Scopus™ publication databases were used for evaluation of relevance to the scientific community (scientometricsFootnote 1). Both databases only include International Standard Serial Number (ISSN) indexed journals and therefore only 53 of the 57 published papers could be analysed. The results are as for 1 March 2006. The impact of these indexed journals on the scientific community was assessed by the ISI impact factor (IF), published in the Journal Citation Reports (JCR). This factor is used to predict the probable number of citations that a paper will receive.14 The IF for a particular year is determined by dividing the number of citations received in that year for articles published in the previous two years by the total number of articles published in that period. For example, the 2004 IF of the British Dental Journal (BDJ) is calculated as the number of citations in 2004 of papers published in 2003 (91) and 2002 (85) divided by the number of publications in 2003 (138) and 2002 (134),† which results in an impact factor of 0.647.

The scientific interest of published manuscripts was evaluated by the number of times they were cited. Citation results were identified from both ISI WOS® and Scopus™ databases for each SGH published paper.

Scientific robustness

The scientific robustness of SGH-funded papers was assessed through a series of questions. All 57 published papers were appraised, including those not indexed on INSS.

Funding and publications by themes

Data relating to the financial support provided by the SGH fund was compared with publication results, time for publication and research themes (as stated by surveyed dentists).

Results

The results of the evaluation of relevance of SGH themes to dental practitioners are listed in Table 1. The themes are ranked in order of decreasing overall relevance.

Table 1 Omnibus Survey results: relevance of research themes funded by the SGH fund as percentage of responses. Results are ranked in order of decreasing overall relevance*
Full size table

The response rate for the postal questionnaire sent to SGH grant recipients was 73% (24 respondents). The results showed that 88% (21/24) of respondents felt that receiving SGH funding had influenced their careers in a positive way. One quarter (6/24) of the respondents acknowledged the role of the grant in helping them achieve their PhDs. A number of respondents felt that the grant had improved their academic career and given them an early interest in research.

A small number of respondents (6/24 or 25%) had experienced some type of difficulty during their projects. These included:

  • Support for statistical analysis

  • Ethical committee approval

  • Limited funding to cover clinical commitment of dentists involved

  • Limited time to prepare clinical and/or multicentre trials.

Improvements were suggested by 7/24 recipients (29%). These included more reliable grant payment, better monitoring of progress through annual reports and better payment of clinical time for examiners.

The majority (78%) of the 41 projects sponsored by the SGH fund had disseminated their findings. SGH fund recipients published 57 manuscripts, of which 53 were in indexed journals. In addition, recipients had disseminated their findings through 66 other dissemination routes, as shown in Table 2.

Table 2 Methods of dissemination (other than papers) used by SGH recipients
Full size table

Most of the SGH published papers (79%) acknowledged the fund. In addition, the SGH fund was acknowledged in three abstracts.

The average time to publish was determined by calculating the difference in time between the publication date and the date when the grant was awarded. The average time for publication of results was about four years (see Table 3). If this fact is taken into consideration, then the number of funded projects that did not produce any kind of publication is reduced from 22% to 17%, as projects commenced after 2001 can be excluded. In addition, some papers were published more than 11 years after receiving the grant.

Table 3 Average time to publish from the date when fund was awarded (mean ± SEM)
Full size table

The first SGH published paper was in 1995, with first citations appearing in the same year. Data from 2005 and 2006 are still to be published by the JCR. Therefore, we limited our IF analysis to the period between 1995 and 2004.

Results of the study of relevance to the scientific community are summarised in Figure 1. The figure shows comparisons between calculated IF values for SGH-funded papers with IF values for the most common journals used for publication by recipients. Thirty-nine of the SGH-funded papers were published in journals listed in the JCR, with an average IF of 1.74 (± 0.27). Note that the BDJ is listed in the JCR and has an average IF (1995-2004) of 0.67 (± 0.04).

Figure 1: Comparison between de facto citations per year of Shirley Glasstone Hughes (SGH)-funded publications and ISI JCR® predicted impact factor (IF).
figure 1

De facto citations from all papers and from selected journals were compared with respective IF. Error bars are standard error of the mean (SEM)

Full size image

SGH-funded papers were cited 624 times. A further analysis of the IF of the BDJ from 2000 to 2004 can be seen in Figure 2. Detailed data on the number of publications prior to 2000 were not available from ISI. On average, SGH papers comprised about 1% of all BDJ published papers each year, but were responsible for an increasing number of citations every year. For instance, in 2004, SGH papers published in the BDJ were responsible for almost 5% of all citations. Furthermore, SGH-funded papers that were published in the BDJ (15) were cited 161 times, with an average of 2.02 (± 0.87) citations per year since publication. Calculating the IF of SGH papers published in the BDJ using the ISI JCR formula shows a two-fold increase on the actual IF (Fig. 3). This implies that papers funded by SGH had a positive effect on the IF of the BDJ during the last five years.

Figure 2
figure 2

Contribution of SGH-funded papers to the British Dental Journal impact factor

Full size image
Figure 3: Calculated impact factor (IF) of SGH papers published in the BDJ compared with the published IF for the BDJ in the JCR.
figure 3

Error bars are SEM. *Significantly greater than JCR BDJ results (paired t-test, p <0.05)

Full size image

The majority (72%) of published papers failed to link their research findings with general dental practice. Conversely, 98% did link their findings with clinical practice. Few papers (25%) had suitable sample selection criteria and for 33% of papers it was impossible to establish if there was any type of bias. Scientific flaws included lack of information about drop-outs (17%), lack of sufficient detail to enable repetition of the experiment (18%), failure to mask or blind subjects and/or observers (17%), lack of baseline measurements (14%) and other minor flaws (see Table 4).

Table 4 Critical appraisal of SGH-funded papers
Full size table

The SGH fund provided on average £39,895 for each of the 17 research themes funded (see Table 5). The most funded theme was periodontics and the least was pain control, with a ratio of more than £34 invested on the former for each pound invested on the latter. However, when we correct this value by the number of projects funded, the most invested theme becomes oral cancer, which received £14 for each pound invested in pain control. Oral cancer papers had the highest IF (3.1) among themes, followed by papers on the delivery of care. Overall the money spent per publication was £13,462 (± £2,017) and the cost per citation was £10,894 (± £5,524). Finally, the last column of Table 5 shows the ratio between pounds awarded and relevance to general dental practitioners.

Table 5 Funding details by research theme
Full size table

Discussion

In order for dental care professionals to practise evidence-based dentistry, research evidence needs to be collected in, and be directly applicable and relevant to, primary care.15,16,17,18 It also needs to be robust and reach quality standards equivalent to those in other areas of research.17,19 Finally, if clinical evidence is to be translated into practical actions which actually benefit patients, it must be disseminated in appropriate media and in an easily accessible form.19,20 The evaluation presented here is therefore important, as it seeks to establish whether or not the only funding stream entirely dedicated to primary dental care research produces clinical evidence which reaches these exacting criteria.

The findings of the work reported here are largely positive. All of the research themes hitherto funded by SGH monies were considered to be at least of some relevance by BDA members. Since the BDA represents some 20,000 primary dental care practitioners, this result is of great importance. However, perhaps the funds should be directed primarily at the research themes thought to be most relevant by the responding practitioners: namely infection control, caries, pain control and oral cancer.

It is also pleasing to note that SGH funding has produced a considerable amount of research which is of sufficient quality to be published in peer-reviewed journals, and is consistently cited by other researchers. An average impact factor of 1.7, although not meeting the impact of top medical research, is very respectable and is higher than that of the British Dental Journal as a whole.14,21,22,23 The fact that SGH-funded research is cited, on average, nearly twice every year after publication suggests that the evidence produced is not only of interest and relevance to general practitioners, but is also worthy of the attention of the dental scientific community as a whole. SGH-funded research raises the scientific quality of the BDA's journal and at the same time offers evidence which can be utilised at the dentist-patient interface in primary care. This having been said, it is also worthy of note that many papers reporting work funded by SGH did not have any direct clinical application and some did not reach the standards of robustness and clarity which should be expected of worthwhile research. In particular, a number of papers were not statistically robust.

It would seem that access to both research training and statistical support would enhance the quality of the SGH research output. Dental research has statistical issues which are peculiar to dentistry,24,25,26,27,28 and therefore statisticians who are unfamiliar with the particular problems of oral health research may not be the most appropriate source of support for primary dental care practitioners.

One matter which was of concern to the investigators was the fact that some of the papers disseminating research which was funded by the SGH Trust did not acknowledge the source of their funding. Whilst SGH funding could never be considered as a potentially biasing source of support, it seems unfortunate that such an important and selfless legacy to the dental profession is not always acknowledged by its beneficiaries. Furthermore, researchers should always reveal the sources of their support so that the reader can make judgements about the independence of the funding.

The distribution of SGH monies and the choice of recipients for the funding are decided upon by a system of peer-review. Whilst this method of determining merit is not without challenges, it is generally considered to be a meritocratic way of deciding upon resource distribution in research. However, the criteria against which grant applicants are judged have not always been made clear in SGH grant allocation. This is an issue which should be acted upon and improvements need to be made in the transparency of the processes involved in SGH grant distribution.

Many other grant awarding bodies partly base their judgements on the experience and publication record of the principal applicant. In the case of SGH funding, the principal applicant is intended to be a primary dental care practitioner who may have little or no research history. Such a system would therefore be inappropriate for the SGH fund.

In conclusion, whilst the research sponsored by the SGH fund is generally scientifically sound and potentially clinically relevant, there are a number of recommendations which can be made as a result of the evaluation reported here.

Recommendations

  1. 1

    The SGH fund should continue to fund research by non-academics

  2. 2

    Greater effort must be made to ensure that SGH-funded work addresses research themes of direct relevance to primary dental care, perhaps by stipulating the areas to be addressed

  3. 3

    Statistical and scientific support for recipients of SGH grants is essential. A quality assurance system to ensure that this support is available to recipients should be put in place

  4. 4

    Recipients should be encouraged to acknowledge the funding source and the role of the BDA in supporting the research

  5. 5

    The degree of support and guidance available to recipients (including links with academic experts) needs to be improved

  6. 6

    Most importantly, governance of the distribution of funds must be more carefully structured and clearly defined

  7. 7

    Finally, the progress of each research project needs to be consistently monitored and evaluated.