Every scientific result is based on an analytic choice made by the researcher (and approved by peers) to answer the research question at hand. However, such analytic choices are subjective. How dependent are results on the analyses?
R. Silberzahn, of the University of Sussex Business School, and colleagues answered this question by having 29 teams (comprising 61 researchers) analyse the same data to answer the same research question, namely whether soccer players with a darker skin tone are more likely than those with lighter skin to receive a red card from referees. Researchers chose analytic strategies ranging from simple linear regressions to complex multilevel regressions and Bayesian approaches. These approaches were subjected to an internal peer review that occurred before the analyses were run. The results of these distinct analyses, however, did depend on the methods: while 20 teams found that there was a relationship between darker skin and receiving a red card, nine teams found a non-significant relationship. This variability in the main conclusion was not associated with either the expertise of the researchers or the evaluation of the soundness of the methodological approaches that occurred during the internal peer-review process.
Taken together, these results show that analytic choices are not just a means to an end but rather that they themselves are partially responsible for the final conclusions.