Discussions to strengthen the quality of statistical analyses are a welcome demonstration of scientists’ willingness to confront uncomfortable knowledge (J. Leek et al. Nature 551, 557–559; 2017). Just as science in general is not a truth machine, statistics is not a device for automatically bootstrapping certainty out of data sets.
All users of statistical techniques, as well as those in other mathematical fields such as modelling and algorithms, need an effective societal commitment to the maintenance of quality and integrity in their work. If imposed alone, technical or administrative solutions will only breed manipulation and evasion.
There may be methodological issues as well. For example, we are only now discovering that the universally accepted standard tests, notably significance and P values, are simplistic and misleading. It might be that improved tests, such as those involving power calculations, are just too sophisticated for otherwise competent researchers. If so, then the conduct of empirical science will need substantial modification.
Nature 553, 281 (2018)