We're incentivizing bad science
Current research trends resemble the early 21st century’s financial bubble.
5 November 2019
Whatever you might want to say about humans, our behavior is profoundly affected by the incentive structures we encounter. Imagine what might happen if banks that issued home loans no longer made money off the interest, but rather made money by blending the loans into investment bonds that they then sold to investors. There are a limited number of people fortunate enough to afford a home. Once all those people had mortgages, the banks would then become a mortgage-backed–security factory that had run out of raw materials to make its product.
The banks could simply stop making money — or they could start making loans to anyone who applied, regardless of people’s ability to pay. After all, once the loans were sold to investors, the risk was no longer the bank’s. Of course, the rating agencies are designed to alert us to risk, but they get paid to do so by the banks, and angering your only customer base is not good business. Prior to 2008, without the intention of doing so, the system had evolved such that the bankers were specifically incentivized to inflate a massive bubble in the economy, built upon bad loans and unsustainable debt, and make a fortune doing it at no risk to themselves — and this is precisely what they did.
So, let’s imagine what might happen if the rules of professional science evolved such that scientists were incentivized to publish as many papers as they could and if those who published many papers of poor scientific rigor were rewarded over those who published fewer papers of higher rigor? What would happen if scientists weren’t rewarded for the long-term reproducibility and rigor of their findings, but rather became a factory that produced and published highly exciting and innovative new discoveries, and then other scientists and companies spent resources on the follow up studies and took all the risk?
Just as banks in 2008 made money from selling the loans, not holding the loans, the quality of the loan ceased to be meaningful to them. Likewise, once published, the innovators of novel science often move onto the next new innovation, and because of publication bias and the “file drawer effect,” we never hear about it if their findings fail in the hands of others. Of course, reputations for good work affect scientists as much as anyone else, but one or two “real” advances by a researcher will erase any downside to even a litany of other findings that disappeared into the trash pile of time since no one else can reproduce them. Indeed, in a now famous report from Bayer Pharmaceuticals, 65 percent of published scientific findings were not reproducible by Bayer scientists when they tried to use them for drug development.
This is not an issue of scientific fraud or misconduct where scientists invent data or purposefully lie; the data are real and were really observed. However, the fiercely competitive environment leads to a haste to publish and a larger number of less rigorous papers results. Careful and self-critical scientists who spend more time and resources to carry out more rigorous and careful studies may be promoted less often, receive fewer research resources and get less recognition for their work.
Of course, scientific publication is subjected to a high degree of quality control through the peer-review process, which despite the political and societal factors that are ineradicable parts of human interaction, is one of the “crown jewels” of scientific objectivity. However, this is changing. The very laudable goal of “open access journals” is to make sure that the public has free access to the scientific data that its tax dollars are used to generate.
However, open access journals charge the authors of articles a substantial fee to publish, in order to make up for the dollars lost from not requiring subscriptions. So, instead of making more money the more copies of the journal they sell, open access journals make more money as a function of how many articles they accept. Authors are willing to pay more to get their articles published in more prestigious journals. So, the more exciting the findings a journal publishes, the more references, the higher the impact the journal, the more submissions they get, the more money they make.
Self-regulation by scientists of decades and centuries past has created modern science with all its virtues and triumphs. However, much like the bankers of the early 21st century, we risk allowing new incentives to erode our self-regulation and skew our perceptions and behavior; similar to the risky loans underlying mortgage-backed securities, faulty scientific observations can form a bubble and an unstable edifice. As science is ultimately self-correcting, faulty conclusions are remedied with ongoing study, but this takes a great deal of time.
Unless and until leadership is taken at a structural and societal level to alter the incentive structure present, the current environment will continue to encourage and promote wasting of resources, squandering of research efforts and delaying of progress; such waste and delay is something that those suffering diseases for which we have inadequate therapy, and those suffering conditions for which we have inadequate technological remedies, can ill afford and should not be forced to endure.