Science Fictions: How Fraud, Bias, Negligence, and Hype Undermine the Search for Truth Stuart Ritchie Metropolitan (2020)
If “you’ve heard of the idea that if you take a larger plate to the buffet, you’ll eat more than you otherwise would have”, writes psychologist Stuart Ritchie, then “you’ve indirectly heard of Professor Brian Wansink”. Wansink, a nutrition psychologist, spent two years as director of the US Department of Agriculture’s Center for Nutrition Policy and Promotion, and published many articles that formed the evidence base for the ‘Smarter Lunchrooms’ movement in US schools. The validity of his research has since been called into question, in what is now one of the most well known such cases in nutrition research. At least 18 of his papers have been retracted, 6 in a single day.
Fraud, bias, negligence and hype are the themes of Science Fictions. Some of the cases Ritchie presents, like Wansink’s, are intriguing and disturbing combinations of all four. His examples of questioned findings run from psychic precognition, psychological priming and the benefits of striking a ‘power pose’ to trachea transplants, the gut microbiome and autism-like characteristics in mice, and arsenic-based lifeforms. All the replication-failure and scientific-misconduct stories you’ve ever heard are here — along with more that you haven’t. Together, these crank up the tension between engaged scientific criticism and maintaining trust in science.
Ritchie opens with the reassuring line that he comes to “praise science, not bury it”. In many ways, the book is a defence of ideals that he thinks we’ve drifted away from. Central to those are the ‘norms’ of science codified in 1942 by sociologist Robert Merton: universalism, disinterestedness, communality and organized scepticism. Yet Ritchie fails to acknowledge that even in what we might consider paradigmatic breakthroughs, scientists have mostly not followed such norms. This uncritical presentation might unsettle those interested in modern philosophy of science.
Ritchie prefaces his rogues’ gallery by introducing some nuts and bolts, including the structure of the journal article and terms such as desk rejection and peer review. Together with his overview of the replication crisis, this introduction would be useful for undergraduates or general readers.
Cognoscenti can dive straight into the central section. This comprehensive collection of mishaps, misdeeds and tales of caution is the great strength of Ritchie’s offering. There are examples from nutrition and social psychology (of course), but also inorganic chemistry, evolutionary biology, genetics, cancer biology, economics, public health and education, demonstrating the disciplinary breadth of the reproducibility problem. This will help to build bridges for those in metascience hoping to address the causes.
Ritchie’s four themes carve complex, interconnected issues at natural joints, and allow his case studies to shine. At times, I was slightly frustrated that this came at the price of separating issues that would ideally be presented together: P values and statistical power are explained in different chapters, and the discussion of publication bias is elsewhere again. In the end, however, the trade-off is worth it.
He concludes by addressing the problems with incentive structures in scientific culture, and open-science initiatives aimed at fixing them. He clearly and efficiently articulates issues such as the flawed metrics we use to assess research quality — from h-indices to impact factors — and a publish-or-perish culture. And he sets out possible solutions such as preregistration of studies and increased transparency. He ends with a plea to “Make Science Boring Again”, which is funny — despite historical inaccuracy.
Ritchie attempts to recognize science as a social and human enterprise, referencing philosopher Helen Longino, and even stating that “science is a social construct”. So I was puzzled that after introducing philosopher Cordelia Fine’s arguments for including feminist perspectives in science, he doesn’t connect these with Longino’s concept of collective objectivity; instead, he dismisses Fine’s points.
The heart of Longino’s idea is that objectivity in science is a collective enterprise, rather than merely an individual one. To maintain it, the scientific community must be diverse, to help cancel out individual biases. Feminist perspectives are an example of the kind of diversity Longino means to include, important for balancing hidden biases in the status quo. Ritchie seems to find Fine’s arguments incompatible with other strategies for mitigating bias, such as preregistration and blinded trials. I disagree: they are from the same toolbox.
No golden age
I am sympathetic to Ritchie’s argument that researchers have adapted to perverse incentives, publishing practices and inappropriate metrics by engaging in P-hacking, over-fitting and other problematic activities, such as rejecting criticism, neglecting error detection and committing fraud. But occasionally he rests too heavily on the idea that there were once golden days when science was a pure truth-seeking enterprise. For example, he bemoans that we have allowed science to “become so tarnished, and its progress to be so badly stalled”.
He goes on to say that the trouble started “somewhere along the way, between Boyle and modern academia”. Perhaps these are simply rhetorical flourishes, and it might be unfair to infer a particular historical commitment, but I do wonder whether Ritchie means to suggest that there was time when scientists did uphold Merton’s norms. If so, I suspect most historians of science would respectfully disagree.
Towards the end of the book, rightly imploring us to take responsibility for the mess, Ritchie refers to a moment “when the scientific community gave its collective approval to these low-powered studies”. Yet underpowered studies have been the norm since statistical-significance testing entered the social and life sciences after the Second World War. Fraud, bias and negligence (hype less so, perhaps) have been with us all along, too. This ‘just so’ undercurrent doesn’t ruin Science Fictions, but it does hold it back from developing deeper insights into how we got to where we are now, and whether the suite of fixes laid out in the final chapter will really be enough to get us out.
Nature 583, 515-516 (2020)