Nature’s Take: what’s next for the preprint revolution

Host: Nick Petrić Howe

Welcome to Nature’s Take, a new series from Nature. Over the years, we often get requests for us to dive deeper into topics – in other words, analysis of the topics in science that matter to you. Well, we’ve heard you, and that is what we are hoping to do with Nature’s Take. In each episode, we are going to pull in some of Nature’s finest into one room and take on a big topic and see where the discussion leads us. For the first episode, we’re talking about preprints. Now, these pre-peer-review open-access articles have become much more common during the pandemic – sometimes for better or worse. Nature as well has been increasingly using them in our reporting, and that has even led to emails from people being critical of our use of them. Regardless, they’ve become an integral part of scientific publishing, but they certainly have their pros and cons. To get into this and more, I’ve gathered three extremely knowledgeable guests here with me from across Nature. First up, I've got Ehsan Masood, Nature’s editorials editor.

Ehsan Masood

Hey, hi, Nick.

Host: Nick Petrić Howe

Next, familiar voice from the podcast, Lizzie Gibney.

Lizzie Gibney

Hiya, I’m Lizzie and I write about physics for Nature.

Host: Nick Petrić Howe

And last but not least, manuscript editor and immunologist Zoltan Fehervari.

Zoltan Fehervari

Hi, Nick. Thanks for inviting me.

Host: Nick Petrić Howe

And of course, I’m your host Nick Petrić Howe. Thank you all for joining me and being my guinea pigs in this new show. Now, preprints, as I mentioned, have had a bit of a surge during the pandemic, but they aren't anything new. In physics, for example, they've been around for a long time.

Lizzie Gibney

Yeah, they've been around since I think it was 1991. I did look that up because I thought it was just forever. But it turns out, it's just a really long time ago. And they grew a lot. So, I think it was quite steady for a long time and then it went through the roof. And I would say most physicists probably publish on the arXiv so, in our field, it's very, very common. But I understand that it’s not that common in other fields.

Host: Nick Petrić Howe

And the arXiv, when you say that, that’s the sort of main repository of these preprints.

Lizzie Gibney

Yeah, exactly. It’s massive. There are about 2 million papers there now. Certainly, in physics and maths and computer science where it started all those years ago, it’s very much standard. But since then, we’ve had new repositories across pretty much every discipline. It’s just really becoming mainstream, I’d say.

Host: Nick Petrić Howe

How do you think it has changed?

Ehsan Masood

I mean, what I’ve certainly found is that initially there was almost an expectation that preprints might actually even change science publishing in a huge way, and partly because the facility now exists for the community to be able to publish what they want to do, and then to sort of have this instant back-pocket access to all the sort of top experts to quickly dive in and sort of help with their findings. And then, of course, that then may lead to other things. I don't necessarily know to what extent that may or may not happen. But I think, in terms of the way that we do our jobs as science editors and science reporters, it pretty much has changed things.

Lizzie Gibney

And I think there's a lot less wariness than there used to be. Certainly, as a reporter, I remember when I first started here which was coming up to nine years ago, I think, we needed kind of special permission to write about something from the arXiv, which you can understand – it's not peer-reviewed. But nowadays, obviously, we make it really clear when a paper we're covering is not peer-reviewed. But we don't need to get a special tick box anywhere that lets us do that. It's just standard.

Host: Nick Petrić Howe

And Zoltan, as sort of a person who receives papers from authors trying to get their work out there, how have preprints sort of changed your workflow, or have they?

Zoltan Fehervari

So, speaking from a biological perspective, for whatever reason, biologists have been a lot more reticent about using preprints. The equivalent to arXiv is biorXiv for the biological sciences and, of course, medRxiv as well, and they appeared in 2013. So, they've been around for a while, so they've been on the landscape for a while but, honestly, I knew very few people who'd use them just purely anecdotally. And COVID basically changed everything because many of the publishers, like ourselves, were basically mandating that all COVID stuff needs to go on there. And then I think that just naturally reduced the bar for people working in other areas of biology that weren't necessarily COVID-related, for them to also start putting their stuff onto biorXiv or medRxiv. In terms of how it's changed my job, I'd be lying if I said I used it a hell of a lot. If I get time – I’m lucky if I get time to look at actual papers that are submitted to us – but if I do get time, I dip into it occasionally, but it's such a morass of stuff that it's where do you start in a way. Let's say a paper comes in and is submitted to Nature. Let's say the authors are making some exceptionally grand and bold claims about their paper – dare I say the expression ‘paradigm shift’, blah, blah, blah, ‘This will change the face of science as we know it,’ kind of thing. I say, ‘Alright, let's just have a little peek at biorXiv or medRxiv, see if it's on there and see if there's been much engagement from the community. So, I'll go on and you can comment on it, you can see the number of downloads, you can get all metric stats and so on, on it. So, occasionally, if I see a paper that's on there and it's done really well, it's got a lot of attention, then I think, well, maybe there's something in it.

Host: Nick Petrić Howe

And we should say here that, although in general the number of submissions from biologists to preprints is certainly much lower than, say, physicists, there are some fields within biology that are, and have been for some time, very active on preprint servers. So, as ever, not one size fits all. But thinking again more generally, one thing that you mentioned there is this idea that there’s this sort of pre-peer-review for a preprint, where you see what scientists think before publication. So, that can be taken as one kind of positive. What are some of the other positives of preprints?

Ehsan Masood

I think, particularly if you might be in the sort of what might be regarded as kind of middle-tier or middle-experience researcher, so you're not highly experienced, maybe you're not really early career, and so it's a good place. I mean, ordinarily, before publishing, you would send your material to a sort of brains trust, just to kind of get a bit of a feel for what other people think prior to submission often. And so, it really helps in that sense and you'll discover people you wouldn't have thought existed at that time. So, I think that certainly a sort of a big plus.

Host: Nick Petrić Howe

And Lizzie, I guess with physics, it's basically the done thing to put it on arXiv first. So, what are the sort of positives people have seen since that has been happening since the 1990s?

Lizzie Gibney

Well, you can stake a claim. So, if there's some discovery, you put it on the arXiv, then you were the one who made it first. And often there are a lot of different teams who are racing towards the same kind of findings. It's also good for making rebuttals or challenging findings. People might remember a few years ago this BICEP2 discovery – or they thought at the time it was a discovery – of this like twisted light that revealed gravitational waves from the early Universe. And it turned out that actually it was a dust pattern in the sky that they were seeing instead. And that was a huge announcement in the press absolutely everywhere – in Nature too, of course – and very quickly, within, I think, a couple of weeks papers accumulatively said that the finding was not real. And that was through the arXiv. It was able to respond that quickly, that people were able to put their findings out there. So, I think also this rapid response is something it's really good for.

Host: Nick Petrić Howe

And also, this rapid response, governments around the world were eagerly trying to get any information they could about COVID when this first happened, but that was also not without its dangers. You're probably all familiar with the ivermectin scandal. These were preprint papers that basically went viral, and this sort of information that wasn't accurate, as it turned out, was out in the world. I guess, I'm just wondering like what are the other sort of risks with these preprints?

Ehsan Masood

So, I know that we're, in a sense, we're talking in the context of preprints. But in a way, the idea that a government or a lab can issue its findings prior to sort of formal peer-reviewed publication, and that in its own way has probably existed since the year zero. I think the interesting thing is that the pandemic has sort of coincided with this sort of hyper-viral technological ability to move things quickly. But even if we take previous infectious disease outbreaks prior to where we are now, there's been a requirement, both by policymakers, by ourselves in the press and elsewhere, for fast information. And so, researchers have had to find other ways of getting that information out. I guess it's just part of the difference is the scale. The scale is just so huge. So, again, in that sense, the phenomenon necessarily, I would say, probably isn't kind of like 100% new.

Lizzie Gibney

It puts a lot of responsibility on journalists, and we're okay with that. We have very close ties with the research community and I'm aware that if I report on a preprint paper, then I'm essentially doing my own little version of peer review before I write about it. But in the case of ivermectin, and a lot of publications around the world, that didn't happen. So, that's definitely a big risk of preprints is that some publications, some news outlets, don't distinguish between a preprint and a published paper.

Zoltan Fehervari

That was the question I was going to actually ask you guys. I think things have certainly got better, but I did notice, certainly in the early part of the pandemic, I'd be reading something, and I thought, ‘This is kind of interesting.’ And then I looked for the paper on PubMed, or whatever to find it and couldn’t find it. It turns out it was being reported just from a preprint. Most news outlets now have got better at that and they'll flag it up and they'll put little asterisks on it and they'll say what it means to be a preprint, that it’s not been peer-reviewed. I think in the early days, it was a bit of a wild west thing. In fact, the BBC reported an article that I had under review, but was on a preprint, and didn't say it was on a preprint. And I actually thought the guy who was with us, who was publishing his paper with us and were having it reviewed, had got scooped by another lab because I thought, ‘Blimey, this stuff that the BBC is talking about sounds exactly like the stuff that we've got under peer review,’ and I got really worried. And lo and behold, I found it on biorXiv, but the BBC had reported on it without saying it was a preprint.

Host: Nick Petrić Howe

And that is a very common story throughout the pandemic, right, not least because, as I think Ehsan said earlier, lots of publishers decided to start mandating data and research be released on preprints, even when it was submitted to journals for publishing, including Nature.

Zoltan Fehervari

So, when the pandemic raised its ugly head, we realised, basically, here's this problem of global importance. It was our responsibility as a publisher to disseminate any information that we've got as soon as possible and not hold up stuff behind paywalls and peer review, which can take months and months and months. So, we were highly supportive of that.

Ehsan Masood

It was also just responding to the fact that pretty much anyone in this field was just needing to get their information out fast, and it was a recognition from us and I think from, I would say, pretty much every other publisher, that we need to work with the flow and just understand that that's a phenomenon of behaviour that's happening, and we need to recognise it, appreciate it, accept it, work with it.

Lizzie Gibney

Because the researchers themselves were working at just incredible pace, weren't they? So, it would just seem crazy to then put something to slow the system in the way of getting it out to the world. But I think Nature has been behind preprints for a while and people don't know that. I think like a lot of times, when we come to write about a preprint, people sometimes worry that they won't be able to still publish their paper in somewhere like Nature. And that's just not the case.

Host: Nick Petrić Howe

And so, Nature is pro-preprints, as you've sort of said, but something Zoltan said earlier about reading in the BBC, not necessarily knowing that something was a preprint rather than a conventional paper, it just makes me wonder, how is it best that we actually communicate that with our audience? On the podcast, for example, we'll say, ‘This hasn't been peer-reviewed yet,’ and I think maybe that means something to the Nature audience. But more generally, how is best to communicate that a preprint is something different?

Lizzie Gibney

Yeah, I think, as you say, for our audience, they probably do have a sense of what peer review is. But for a lot of people, peer review, I think, I hope, just means a kind of rubber stamp that this has been vetted by other scientists. I think a lot of people get the idea. And actually, on the flip side, some people don't give too much credence in it, I think. A lot of people assume that if it's peer-reviewed, that's it. That means it's 100% true. There's no disagreement in the community. And actually, if you see the reviews themselves, which now, we and others are making open, you see that the comments are a lot more nuanced than just a big gold star for this paper, this is all correct. So, yes, I think people maybe don't quite get the difference between a preprint and a published paper. But actually, the difference, this might be controversial, is not as big as some people think.

Zoltan Fehervari

I think it’s worth adding, though, that it’s not equivalent to some keyboard warrior knocking out something in their basement. There is actually a level of curation and moderation that goes on, on preprints. Also, other authors on arXiv have to kind of like sign off on you and say that this is guy legit.

Lizzie Gibney

There are moderators.

Zoltan Fehervari

They’re not a total loon. Something similar works in medRxiv and biorXiv. There is a quality check even for a preprint.

Host: Nick Petrić Howe

And so, thinking about ivermectin and other things that have happened with preprints where it's gone kind of wrong, is there a risk that these types of articles may damage trust in science?

Lizzie Gibney

Yeah, I think there probably is because people obviously don't understand the full difference between a preprint and a published paper. But at the same time, there are so many instances where it's been a big blowout publication in Nature, a huge finding, and it's turned out not to be real, and that's been through peer review. So, that does also happen, and I think, to some extent, that damages people’s opinions of science and trust in science. But what we really need to be working on is this idea that science is not this done-and-dusted, certain process, as it's often portrayed to be. It's a constant work in progress, and we're never quite sure. It's all about uncertainty. And so, in an ideal world, we’ll just help, in whatever articles you're writing, whether about preprints, whether about publications, we’ll help to kind of push that message and show science as it really is, rather than this kind of fairy-tale version.

Ehsan Masood

And I think the more we can do to just expand on this idea that we're always trying to approximate to the truth, and we often get to it, but it's a journey. It's a process. We're on the way. And preprints are sort of part of that. I don't think, at the moment, preprints quite get communicated in that way, and I think that's a bit of a stretch to say that's, yeah, that's where it's at. But I think if we could do that somehow, if the narrative could be changed, I think that'd be really powerful.

Host: Nick Petrić Howe

Yeah, the other thing, though, I'm thinking about preprints is, especially during the pandemic, there was so many preprints that it was difficult to find what was good and what was not. And sometimes you might be like, ‘Oh, it's in this journal, I trust that journal.’ or something like that. How difficult is it to sort of sift through them all and find the sort of wheat from the chaff?

Lizzie Gibney

It’s really, really hard. I tend to, I mean, if there is a lab that has a good history in a particular field, you're obviously more likely to trust results coming out of that lab, so that's one way. In general, in terms of sifting down that huge number, I just have loads of alerts set up really. So, the thing I find it really useful for is, say there's an experiment that you hear about, maybe you get a press release saying there's an experiment, they've got a really big goal that sounds quite far-fetched but incredible. So, then you just put all the keywords in and you set up your search and then whenever they finally achieve that, you get pinged. So, that's the way that I do it, is kind of use it as a resource for things that I'm semi-expecting to see already that you're waiting for, and then you can go and get there first. Because of course, the scooping others is not just for scientists, it's also for journalists. But, yeah, it is, if you just – and it’s very rare that I have days this quiet – but I have before just gone to the arXiv and gone like, ‘Oh, what's there today,’ and I can tell you, you don't get very far with that policy.

Host: Nick Petrić Howe

And on the flip side, I guess, the scientific community has long been pushing for sort of open access, more transparency. Is this kind of a move towards that?

Zoltan Fehervari

I would say so, to an extent, although, as I say, it hasn't been fully embraced, particularly in the biological sciences.

Lizzie Gibney

I think it probably is the easiest way of doing it because you've written this paper, it hasn't had any input, you can put that in the repository, and you've adhered to open-access requirements. And the beauty of it as well is then you have this other version that eventually comes out in a journal, and you can see the value add and what's happened to it throughout that process, and you can avoid all of the hoops and things you've got to jump through in order to make your paper open-access just by putting your original paper in a repository. So, I think in that way, it's definitely helped to push open access.

Ehsan Masood

Open access is coming, clearly. It'll be really variable depending on which part of the world we're in, what disciplines, what journals, but the momentum is clearly there. I think in terms of preprint uptake, from different groups of disciplines or different disciplines, I do wonder – and I'd be interested to know what, Lizzie and Zoltan, you think – I do wonder to what extent it's reflected by the level of collegiality and competitiveness.

Zoltan Fehervari

Physicists are a special breed in that in that regard, actually.

Lizzie Gibney

But it's not always collegiality, though, is it? As I say, it's often staking your claim. It’s saying actually, this paper someone else has written is complete rubbish.

Ehsan Masood

And maybe, particularly if you’re sort of relatively new in a field and you haven't got that sort of big name, kudos in lights, and you do want to make sure, it's a bit like sending your IP in an envelope.

Lizzie Gibney

I mean, it has the potential to be really democratising like that, doesn't it? You don't have to have some big lab name behind you. You don't have to be a big professor, which, we'd like to say that that isn't how you end up with your paper in a big journal, but there's so many subtle things that interplay in order to get there and your network in science plays a huge role. And in theory, you could put your paper, it might be ground-breaking, you put on a preprint server and that's the way it gets out there.

Zoltan Fehervari

It has happened in the past, right? There's the famous case with Grigori Perelman. It's kind of a famous story. So, the Poincaré conjecture, and there was this guy Grigori Perelman, rather than eccentric chap by the sounds of it, he'd solved it, put the proof up on arXiv, was not interested in publishing in any official journal and just said, ‘Look, it's on arXiv. It's there. Make of it what you will.’ I think maybe mathematics is a special case because you don't need technology, you don't need lots of money to prove something. You can just work your way through his proof to see if it's correct or not. So, that was never published anywhere. It just exists solely on arXiv. They offered the Fields Medal to him and some other Millennium Prize, which he declined because he’s, as I say, an eccentric chap. But it shows for some people, in some fields, you don't actually need to go the full publication route – arXiv might be sufficient. It’s not going to work in biology or molecular biology or some lab-based science. That's never going to wash, really, I don't think. But for mathematics, it seems eminently feasible.

Host: Nick Petrić Howe

I mean, to throw out then a very controversial question, do we need journals and things like that? Can we just throw everything on the arXiv and then science is open and there for everyone?

Ehsan Masood

I mean certainly within the sort of foundations of open-access publishing, that was one of the ambitions and it remains definitely one of the ambitions for a subsect of the community.

Zoltan Fehervari

I'm not going to be a turkey and vote for Christmas on this one, so I do think we still need the journals. I know the guy who started arXiv, Paul Ginsberg, I think he suggested it as a way to just bypass the whole standard publication route and we just go with arXiv and that's it.

Lizzie Gibney

Yeah, I mean, the arXiv has been around for so long, but physics journals still exist, so there's clearly something happening there. I would say, as someone who reads a lot of papers as a non-expert, you can tell when it's been well edited and when it's in a journal. It's just so much more palatable because, of course, when you're writing a preprint, you're writing directly to your peers. You're probably writing to your rival, so it's very specific, what you put in and the level you're pitching at, which is very, very different than a journal publication.

Zoltan Fehervari

Lizzie, you made me feel good about myself as an editor then. So, when you read a preprint – just to clarify this for the record – you say it's a lot easier to read the final published version as, say, in Nature, as opposed to the preprint. So, that step of peer review and editorial interaction you do think improves the manuscript?

Lizzie Gibney

Yeah, definitely.

Zoltan Fehervari

That’s good to know. I think it’s worth repeating.

Lizzie Gibney

The more general the journal, the more that is the case. If it’s in a specific astronomy journal then maybe it doesn’t change that much. But if it’s going to be published in Nature, then the whole idea is that any other scientist needs to be able to read it and so, yeah, it usually gets quite an overhaul.

Host: Nick Petrić Howe

I’m not sure whether I can understand all physics papers in Nature, but I take your point.

Lizzie Gibney

What you talking about, Nick?

Host: Nick Petrić Howe

So, then, as a reporter or as an editor, are you then a bit more wary when something’s in a preprint?

Lizzie Gibney

I will do my own version of peer review. I'll kind of look at who they reference in the paper, especially if those references are to papers that disagreed with them in the past. I'll go to their authors and I'll say, hey, and the fact that it's on the preprint server already means you can just share it widely and ask as many people as you want for their thoughts on it, and they have as much time as they want to pore over it. So, a huge dollop of caution because I, of course, myself cannot look at that and say, ‘Oh, hey, they've clearly fiddled with their stats here’ or something. I'm not that level of expert. But you’ve just got to try and send it to the right people who would know.

Ehsan Masood

I would just say, so, let’s imagine, I’m going to make a prediction now. And, of course, journalists making predictions, we always get them wrong, but it’s a bit of fun. So, if we imagine a world, let’s say 50 or 100 years from now where there weren’t journals and it was some sort of system where there was total democratisation of publishing, a very flat structure, I would still think that within that system, something would need to emerge where you have, I wouldn’t call it hierarchy but some sort of validation or some way of knowing what is it that I should be reading. Why should I be reading this? Because, Lizzie, as you said, there is just the potential to drown in the material.

Lizzie Gibney

There’s too much.

Ehsan Masood

And so, even if we didn’t have the present sort of system of journal publishing and that all got wiped away for some reason, I think something would need to emerge that would be a sort of triaging-type system or some way of encouraging us or explaining this is what you need to be reading, here’s our recommendations, here’s what we think you should read, that kind of editorial curation. So, maybe it’ll come back and it might be a different business model to the one that exists, but the principle, I think, might still be there. So, predictions – dangerous.

Host: Nick Petrić Howe

Well, predict away. With that in mind, what do we think the sort of future is of preprints? They’ve obviously had a huge surge during the pandemic. Is that going to wane? What’s going to happen?

Zoltan Fehervari

Well, I mean, I think clearly the pace of publishing COVID-related stuff is slowing down. I mean, it was completely unprecedented. I mean, I think up until recently, like HIV was the most studied disease in the world. I think it’s going to get overtaken by COVID very shortly, just in terms of the number of publications on it. Yeah, HIV has been around for 40 years. COVID has been around for, what, two and a half years. So, that is a really weird, upside-down setup. I do think it has changed the landscape, though, so I do think people will be engaging with preprints more and more. So, the bar for people being willing to submit to preprints has come down definitely. So, I think they’re here to stay. And we’ve just talked about a couple of them at the moment, but there are dozens and dozens of preprints catering to every single little niche area and a few other big ones as well. So, I think they’re to stay and they’ll grow, as long as they get funded, right? Because they’re not. You kind of assume they just go on in the background, but actually, I was reading somewhere it costs about US$1 million to run every year, US$1 million to keep arXiv going, and that’s just working on volunteers working for free. All the moderators work for free. But all the overheads for everything, maintaining the servers and what have you, they are expensive for sure.

Host: Nick Petrić Howe

Well, this has been an absolutely fascinating discussion, but I think that’s probably all we’ve got time for. Nature’s Take will return, and our next topic will be the ongoing conflict in Ukraine and where science fits in to that. But for now, all that’s left to say is thank you all for joining me.

Ehsan Masood

Thank you.

Zoltan Fehervari

Thank you very much.

Lizzie Gibney

Thanks, Nick. It’s been a pleasure. I’ve learnt a lot.

Host: Nick Petrić Howe

Thank you all. I’ve been Nick Petrić Howe. See you all next time.