How young researchers can re-shape the evaluation of their work

Looking beyond bibliometrics to evaluate success.

31 March 2020

Ivar Pel (www.ivarpel.nl)

From left to right: Inez Koopman, Annemijn Algra, and Rozemarijn Snoek from Young SiT.

PhD students are experiencing a major evaluation gap: they are being assessed on criteria that do not match their own research goals, activities, or the role they wish to play in society.

To promote high-quality research, we need to change the incentives and rewards for career advancement in science.

It’s not easy to shift deeply ingrained practices, but finding an alternative to publication numbers as a measure of scientific quality has finally become a priority for research policy.

In 2018, our thinktank, Young SiT, started a grassroots movement at our institution, the University Medical Center Utrecht (UMC Utrecht) in the Netherlands, to reduce our focus on output metrics and promote better science.

As a result, PhD candidates are no longer being evaluated primarily on bibliometrics, but also on their research competencies and professional development.

Involve young scientists

Young SiT takes its name from the Science in Transition (SiT) movement launched at UMC Utrecht in 2013. Its motto is “Fewer numbers, better science.”

SiT has spurred a broadening of assessment criteria at our institution, for instance, by also weighing the societal impact of a scientist’s work. Dutch universities as a whole have recently embraced a similar change, but they mainly impact senior scientists, allowing little involvement from young researchers.

As young scientists, we want to be involved in shaping the landscape that determines our day-to-day work, since we are the ones who will benefit most (namely, for the rest of our careers).

Young SiT’s goal is to improve and speed up a transition towards more open and responsible research evaluation. Our thinktank consists of 15 young scientists, ranging from PhD candidates to associate professors, all from different disciplines.

Through consensus discussion, we choose themes that we deem both relevant and feasible to tackle, such as responsible research evaluation, public engagement, and open science.

For each theme, we organize a mini-symposium for thinktank members and invited speakers from inside and outside science, such as people working in medical start-ups, to learn from best practices and to brainstorm on how changes can be implemented in young scientists’ daily research practices.

During a mini-symposium on the theme, ‘Career reward systems’, we concluded that young scientists would benefit from an evaluation method that promotes professional diversity and growth, instead of mainly output metrics.

Change the evaluation form

Dutch universities often expect PhD dissertations in medical research to consist of at least four publications. At our institution, the PhD candidate evaluation form, which lists publications, abstracts, and prizes, was the main focus of the annual assessment of PhD candidates by their supervisors.

We devised a new evaluation form, based partly on an existing PhD competence model developed by Dutch university medical centers.

Our new form asks the PhD candidate to describe their two best accomplishments, motivated by personal or societal impact. Accomplishments can range from dealing with setbacks to speaking at a conference or a patient organization event.

The new evaluation form also asks PhD candidates to self-evaluate their professional growth in research-related competencies, using the online Dutch PhD Competence Model tool. This tool focuses on competencies such as responsible conduct of science, communication, and leadership.

PhD candidates can create goals based on these competencies, which can serve as a conversation starter and a framework for their annual evaluation.

From idea to implementation

For the implementation of our new evaluation form, we chose the graduate program, Clinical and Experimental Neuroscience, where two of us are currently enrolled, as a starting point.

In the spring of 2019 we pitched our idea to the director, coordinator, and a representative body, and used their feedback to fine-tune and implement our evaluation form.

As of January 2020, around 200 PhD candidates in the graduate program now use this new evaluation form. This has motivated other graduate programs in the university to implement it also.

The Board of Studies of Utrecht University and PhD Council of the Utrecht Graduate School of Life Sciences are both showing interest in implementing our new PhD candidate evaluation form for all 1,800 of their PhD candidates.

Although a new evaluation form might seem like a small change, it empowers PhD candidates to discuss their professional development as part of their assessment. We believe this shows how science can benefit from grassroots movements like our Young SiT initiative.

There are many more themes and ideas that can be tackled like this, as was recently shown by the journal club initiative, ReproducibiliTea, which has set up journal clubs for researchers around the world to promote local open science communities and facilitate discussions about reproducibility of scientific work.

These initiatives show that we do not have to wait for institutions, funders, or journals to involve us. We, as young researchers, can start transforming science now.

Annemijn Algra and Inez Koopman are affiliated with the Department of Neurology and Neurosurgery at UMC Utrecht in the Netherlands. Rozemarijn Snoek is affiliated with the Department of Clinical Genetics at the UMC Utrecht. They write on behalf of the Young SiT.