Marine snails from the US West Coast show signs of shell weakening as a result of ocean acidification. Credit: NOAA

As the oceans’ chemistry is altered by rising levels of atmospheric carbon dioxide, the response of sea-dwellers such as fish, shellfish and corals is a huge unknown that has implications for fisheries and conservationists alike. But the researchers attempting to find an answer are often failing to properly design and report their experiments, according to an analysis of two decades of literature.

Oceans absorb much of the CO2 emitted by human activities such as coal burning. This leads to a variety of chemical changes, such as making waters more acidic, which are referred to as ocean acidification.

The United Nations has warned that ocean acidification could cost the global economy US$1 trillion per year by the end of the century, owing to losses in industries such as fisheries and tourism. Oyster fisheries in the United States are estimated to have already lost millions of dollars as a result of poor harvests, which can be partly blamed on ocean acidification.

The past decade has seen accelerated attempts to predict what these changes in pH will mean for the oceans’ denizens — in particular, through experiments that place organisms in water tanks that mimic future ocean-chemistry scenarios.

Yet according to a survey published last month by marine scientist Christopher Cornwall, who studies ocean acidification at the University of Western Australia in Crawley, and ecologist Catriona Hurd of the University of Tasmania in Hobart, Australia, most reports of such laboratory experiments either used inappropriate methods or did not report their methods properly (C.E.CornwallandC.L.HurdICESJ.Mar.Sci.http://dx.doi.org/10.1093/icesjms/fsv118;2015).

Cornwall says that the “overwhelming evidence” from such studies of the negative effects of ocean acidification still stands. For example, more-acidic waters slow the growth and worsen the health of many species that build structures such as shells from calcium carbonate. But the pair’s discovery that many of the experiments are problematic makes it difficult to assess accurately the magnitude of effects of ocean acidification, and to combine results from individual experiments to build overall predictions for how the ecosystem as a whole will behave, he says.

The survey, published in the journal ICES Journal of Marine Science, was based on a search of the Scopus database of research papers. Cornwall and Hurd analysed 465 studies published between 1993 and 2014 that manipulated seawater chemistry and found that experiments often failed to implement widely accepted measures to ensure quality.

For instance, to ensure robustness, manipulation studies should use multiple arrays of independent ocean-mimicking tanks. And in experiments that compare sea animals under acidified conditions with controls, these tanks should be randomized to remove bias. But the pair found that in several papers, researchers used one main seawater tank to supply multiple, supposedly independent smaller tanks.

Chemical errors

The researchers also found mistakes in basic chemistry: some authors simply added acid to a tank and ignored other chemical changes that result from the absorption of CO2, such as increased levels of carbonates. Although the frequency of these chemistry errors has dropped since the 2010 publication of an international ‘best practice’ guide for ocean-acidification experiments (see go.nature.com/sp5kgn), the researchers found no evidence for improvements in the design of tank arrays.

Truly rigorous designs are logistically complex and expensive.

Bayden Russell, an ocean-acidification researcher at the University of Hong Kong who reviewed drafts of the latest paper, has also noticed that some researchers fail to take into account the complexities of ocean acidification when designing their experiments. “It is these complexities that will drive ecosystem responses to ocean acidification,” he says.

Overall, Cornwall and Hurd found that in only 27 cases could they be certain that an appropriate experimental design had been used, and in 278 cases, the design was clearly inappropriate. The remaining studies had insufficient detail on experimental set-up — a problem in itself, note the researchers.

The pair present a series of recommendations for well-designed experiments and suggest a checklist of details that should be included in papers to allow replication of experiments, including which chemicals were used to manipulate the seawater chemistry and the configuration of tank arrays.

Ove Hoegh-Guldberg, director of the Global Change Institute at the University of Queensland in St Lucia, Australia, suggests that researchers also need to take account of natural variations in temperature and CO2 in experimental set-ups and ensure that experiments that manipulate acidity also simulate the accompanying rise in temperature from global warming, which many do not.

Pressure to publish

Russell thinks that most research groups are now trying to use appropriate designs, but says that there are still problems, which he attributes to a variety of factors. “Unfortunately, truly rigorous designs are logistically complex and expensive in both set-up costs and ongoing maintenance time,” he says. “When superimposed on the increasing pressure to publish rapidly, and in top journals, some researchers or research groups are still attempting to publish what I would consider sub-standard research.”

Jonathan Havenhand, who works on marine invertebrates at the University of Gothenburg in Sweden and who co-authored the 2010 guide, welcomes the latest paper: “Everybody should know the stuff that’s in Cornwall and Hurd’s paper. It’s good that they wrote it. It’s disappointing that they had to.”

Havenhand suspects the paper will be highly cited. “Whether people are going to be happy to be citing it, I don’t know.”