How do you decide whether or not somebody is a fully trained researcher? Janet Metcalfe, head of Vitae, a non-profit that supports the professional development of researchers, tells Julie Gould that it's time to be "really brave" and look at how doctoral degrees are examined.
But what role should the thesis play in that assessment? Does it need overhauling, updating, or even scrapping?
Inger Mewburn, who leads research training at the Australian National University in Canberra and who founded of The Thesis Whisperer blog in 2010, suggests science could learn from architecture. Student architects are required to produce a portfolio, creating a "look book" for assessors or potential employers to examine as part as part of a candidate's career narrative. For graduate students in science, this could include papers, journals, articles, presentations, certificates, or even video files.
"The PhD is meant to turn out individual, beautifully crafted, entirely bespoke and unique knowledge creators," she tells Gould. "And we need people like that. We need creative people with really different sorts of talents. We don't want to turn out 'cookie cutter' researchers."
David Bogle, who leads early career researcher development at University College London, tells Gould that UCL's three-pronged mission statement includes impact.
"We want our research to make an impact, and in order to support and reinforce that it is now mandatory to include a one page impact statement at the front saying 'this is the difference it will make in the world,'" he tells Gould. "Any impact — curriculum, society, business, anything. It might not end up making that difference, but we want people to think about it."
What about the pressure to publish? In October 2019 Anne-Marie Coriat, Head of UK and EU Research Landscape at the Wellcome Trust in London, argued in a World View article published in Nature Human Behaviour that PhD merit needs to be defined by more than publications.
She tells Gould that the experience of getting published is a good thing, but making it mandatory is not. "Learning writing skills is a hugely important part of PhD training. Should it be a requirement that all students publish in peer reviewed journals in order to pass the PhD? My answer is absolutely and emphatically no."
Julie Gould hears about assessment and impact in PhD programmes, and what role the thesis should play in demonstrating whether a researcher is fully trained or not.
Hello, I’m Julie Gould and this is Working Scientist, a Nature Careers podcast, and in this episode, I’m exploring the PhD thesis.
Now, from what we’ve gathered, the PhD training system is undergoing change or is still undergoing change, but whatever the PhD training ends up looking like, it will, one way or another, need to be assessed. So, I wrote about this in 2017, asking whether or not the PhD thesis needed updating, overhauling and/or scrapping. Now, when I was at the Revisiting Forms and Forces of Doctoral Education conference in Hanover in September this year, I asked Janet Metcalfe from Vitae to give an overview of what the current assessment of PhD training looks like.
The examination process reviews the research, it looks to see whether or not that researcher has done that research and done it well, and by that deduces whether or not we’ve got a trained researcher. If we step back and look about if we were designing a system from scratch, would this be the way you would want to identify or examine whether or not you have a trained researcher? So, I think we need to be really brave and say, let’s fundamentally look at the way we examine doctoral degrees and look more in terms of how do you assess whether somebody is a trained researcher.
Now, one way to train researchers is to make sure that they are aware of how their work fits into the bigger picture. At the same doctoral education conference, I asked David Bogle, the vice president of research at University College London about how University College London has made it mandatory for all PhD candidates to include a one-page impact statement in their thesis as a way of making sure that they understand how their work fits into the bigger picture and has an impact in any way, whether on society, on their research or just on themselves.
What we really wanted, part of the institutional mission, the current strategy of institution is three lines: leadership, cross-disciplinarity and impact. We want our research to make an impact, so if we want our research to make an impact, we have to be persuading our researchers that they have to think about the impact that they’re going to make. So, in order to support and reinforce that, we introduced that there should be just a one-page impact statement at the front of the thesis. There’s an abstract and then there’s a statement that says this is the difference that it’s going to make in the world. Now, we have various training things about what that might mean, and it’s very varied because it’s not a narrow view of impact. It’s any impact this might have on the curriculum, on society, on business, on anything – what difference is it going to make in the world? And eventually, I see this impact statements improving. I think some people from outside will look at it and thing what’s all this about, but as we move on, people will get better at articulating just in one page, this is the difference this is going to make to the world, and eventually, we get more towards being a bit more predictive about it. At the end of the day, it hasn’t made an impact but this is why I’ve done it and this is what it’s going to do because I was also a bit tired myself of when I examined even engineering PhDs sometimes they duck that question and you think engineering has to make a difference in the world. But everything, all of this research enterprise, we need to be thinking about… it might not end up making that difference, but they need to at least be thinking about it, certainly from the end of the thesis.
So, this was part of a discussion about skills development and labour markets, so what sort of skills are you hoping that people will develop by having to write this impact statement as part of their thesis.
Well, skills development is a much broader thing because doing the thesis, thinking about things, thinking about their own personal development, doing courses, doing skills development, the impact statement itself, I would have said it’s about the difference that the work is going to make. So, it’s not a skill in that sense. I guess it’s just like writing an abstract. How do you do that clearly and crisply? How can you communicate that something which is clear and crisp with evidence of some sort or some at least argument and everybody needs to do that. We need to do that in academia, we need to do that when we write significant statements in journal articles and if you go into the private sector, the board wants a one-page what difference, why should I put money into this.
Another way to assess whether or not someone would make a good researcher is to see if they’ve published any work, and some programmes require their PhD researchers to publish their research during a PhD, others don’t. Some people have a PhD thesis by publication, others don’t. In a commentary that Anne-Marie Coriat from the Wellcome Trust wrote for Nature Human Behaviour which was published in October this year, she argues that having the experience of publishing is a good thing, but making it mandatory might not be.
If asked do I think that students should publish, the answer to that, well do I think it’s a good thing, the short answer is it is not a bad thing and learning writing skills is a hugely important part of PhD training. If asked the question in a slightly different way, do I think it should be a requirement that all students publish in peer-reviewed journals in order to pass a PhD, my answer is absolutely, emphatically, no. To sort of determine that publication in a peer-reviewed journal and often that’s translated into and one with high impact factor or one word summary is then driving pressures on the system which say that you can predict the way that science is going to develop, and I think that’s quite unhealthy in many ways.
But what if, like Janet Metcalfe suggests, the thesis is completely revamped. Throw the old star out the window and start with a blank canvas. What would this new form of assessment look like? Inger Mewburn is an associate professor and the director of research training at the Australian National University and she’s also the author of the blog Thesis Whisperer. Now, she believes in a portfolio system like the one that is used in architecture, which is what she initially trained in.
The way that architecture is assessed is through a portfolio, and what you do is you spend years studying and you make a range of buildings, you draw them, you make models, you take photos and so on, and then just before you graduate, what you’re expected to do is to take a piece of work, maybe redraw some of them, write a bit about them, put them in context and create sort of like a book, sort of like a fashion look book and that’s got writing, that’s got drawings, that’s got photos and that’s got maybe some certificates that you got along the way that you demonstrated that you learnt about fire hazards or something, and you put it all in the form of a book that an employer can look through and the book represents you. It’s your portfolio. So, the portfolio can contain anything but it’s kind of a collection of objects and a narrative that tells people who you are as a designer. And I think I’m attracted to this idea for the PhD because a portfolio idea is quite flexible and what I don’t like about some of the things that have been suggested to fix it like let’s whack a bit more coursework in there, is that it’s sort of at odds to what the enterprise of the PhD is. The PhD is meant to turn out individual, beautifully crafted, entirely bespoke and unique knowledge creators, and we need people like that in this world where increasingly we need creative people with really different sorts of talents. We don’t want to turn out just cookie cutter researchers. We want to turn out a really wide range of different people and a portfolio sort of gives you the chance to say, well, you can include all sorts of objects in it, maybe you’ve done some coursework, maybe you’ve written a paper, maybe you’ve done journal articles or maybe you’ve done presentations, maybe you’ve done a thesis, maybe there’s some video of you in there, and there’s also a narrative about who you are.
Would you include a viva voce style assessment to go with it as well?
I think you absolutely would. I think that a portfolio makes the most sense when the person narrates it and that’s the point of the architectural portfolio. So, you’re meant to put it on a desk in front of either an assessor or an employer and you turn the pages and you talk them through everything you’ve done, why you’ve done it, what you learnt from it and what you would have done differently, so it becomes a conversation. How you set a set of standards or criteria for what something like this looks like is challenging, but I would draw everyone’s attention to the fact that we don’t really have a benchmark or set of criteria or a perfected rubric for the PhDs as we have them at the moment.
Gillian Houston is the Chair of the UK Council for Graduate Education and she thinks this model would be a great step forward, but it would still need to be accompanied by more traditional aspects of the PhD thesis.
So, I feel that that’s very appropriate to architecture, but I also think that those elements would have to be accompanied by a review of the literature or a review of design in your field and how it’s relevant to your work, and I mean an analytical review, so you have to say what you think is good and what is not so good, and a general contextualisation of your work in the wider world, and I also think you would have to have certain academic conclusions. So, I think those are very good developments, but I don’t think it’s like a replacement. I think they have to be integrated into a coherent whole, which is true at the time you graduate.
Now, Janet Metcalfe believes that the future of PhD assessments needs to include an assessment of researcher development as well as of the work that they do.
If you are assessing competencies, somebody would have to review that. So, certainly in the way we’ve been looking at recognition of researchers’ competencies, we’re using skilled assessors, those that are used to dealing with that sort of qualitative information, doing independent reviews by several assessors who then come together in the same way you get academics to do that initial review of the thesis, then a similar process in terms of reviewing the portfolio of evidence around their skills and competencies, and then they come to a joint conclusion as to whether or not that evidence is robust and sufficient in order to identify somebody as a skilled researcher.
Thank you to Janet Metcalfe, David Bogle, Anne-Marie Coriat and Gillian Houston for their contributions to this episode an