Skip to main content

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

SCIENCE OF SCIENCE

Methods matter

Adv. Methods Pract. Psychol. Sci. 1, 337–356 (2018)

Every scientific result is based on an analytic choice made by the researcher (and approved by peers) to answer the research question at hand. However, such analytic choices are subjective. How dependent are results on the analyses?

figurea

Sally Elford/Ikon Images/Getty

R. Silberzahn, of the University of Sussex Business School, and colleagues answered this question by having 29 teams (comprising 61 researchers) analyse the same data to answer the same research question, namely whether soccer players with a darker skin tone are more likely than those with lighter skin to receive a red card from referees. Researchers chose analytic strategies ranging from simple linear regressions to complex multilevel regressions and Bayesian approaches. These approaches were subjected to an internal peer review that occurred before the analyses were run. The results of these distinct analyses, however, did depend on the methods: while 20 teams found that there was a relationship between darker skin and receiving a red card, nine teams found a non-significant relationship. This variability in the main conclusion was not associated with either the expertise of the researchers or the evaluation of the soundness of the methodological approaches that occurred during the internal peer-review process.

Taken together, these results show that analytic choices are not just a means to an end but rather that they themselves are partially responsible for the final conclusions.

Author information

Affiliations

Authors

Corresponding author

Correspondence to Mary Elizabeth Sutherland.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Sutherland, M.E. Methods matter. Nat Hum Behav 2, 802 (2018). https://doi.org/10.1038/s41562-018-0462-9

Download citation

Search

Quick links

Nature Briefing

Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.

Get the most important science stories of the day, free in your inbox. Sign up for Nature Briefing