Reproducibility: A Crisis or A Revolution?
First published at my LinkedIn page: https://www.linkedin.com/pulse/reproducibility-crisis-revolution-fatma-bet%25C3%25BCl-din%25C3%25A7aslan
Imagine a paper showing the efficacy of a drug candidate offering treatment to a deadly disease. Then, imagine this being just a mistake in the analysis. You do not want to imagine such scenario, do you?
However, mistakes can happen anywhere including science. There might be various reasons for wrong conclusions. For example, it was claimed that neutrinos are faster than light at 2011 at CERN despite its contradictory nature with the well-known and formulated axiom of Albert Einstein. Well, contradiction is part of science as long as there is a strong scientific evidence. In this case, it was a simple human error due to incorrect connection of a fiber cable rather than a new discovery [3,4].
Self-Deception
Most scientists are skeptics at different levels by the nature of doing science. Yet, when it comes to their own research, there might be some cognitive fallacies, institutional pressure, and liability of hypothesis [5]. Due to the possibility of biases and random or systematic errors, reproducibility of the analysis/data/paper is one of the most important parts of a high-quality research.
I am not sure when the discussions on the reproducibility have started. The first attempts might be on p-value, statistical errors [1] and self-deception [2] of researchers. In fact, this is a big concern in many disciplines including social and behavioral sciences.
Reproducibility Challenge, If Not A Crisis
When Nature surveyed 1,500 scientists regarding reproducibility crisis in 2016, it revealed that 70% researchers failed to reproduce each other’s experiment [9]. Interestingly, they also admit failing attempts of reproducing their own findings (50%). These led to release of scientific manifesto on the reproducibility together with awareness on transparency and open science [6,7].
On the contrary, it was claimed that the term “reproducibility crisis” is misguided and unsupported [10]. The reasons were the negligible low frequency of scientific misconduct and questionable research practices compared to whole scientific literature, no data supporting any increase on this matter, and their unknown impact on current literature. Also, the author added that this does not help promoting better scientific practices. Meanwhile, the term called “credibility revolution” has been added to literature replacing the replication crisis to emphasize productivity, creativity, and progress [11].
Lower Expectations on “Hype”
Interestingly, a recent finding showed that the credibility (let’s consider citations here) of non-replicable publications are more than replicable ones in life sciences, economics, and psychology [12]. In addition, the citation gap between non-replicable vs. replicable in Nature/Science has been increasing enormously since 2015. One of the reasons why non-replicable papers were still published in high-ranking journals was proposed as trade-off between lower expected reliability and being interesting. This means, if paper is interesting and innovative enough, journals might not pay enough attention to robustness of the work.
While several journals, including Science and Nature, have been updating their guidelines, checklists or adding statisticians to reviewer editors in the last decade to prevent non-credible works to get high credit, it seems the gap is still high in 2018.
Nevertheless, whether it is a crisis or a revolution after awareness, we need to find solutions for better practices. Let’s discuss these in the next session.
Manifesto on Reproducible Science
Although, the focus of the solutions is “replicating”, there is more than replicating the data itself [13]. First, it would be nice to understand the factors behind. Considering scribbling important steps on paper towels and not saving on a computer or as a video, running experiments without controls, using outdated reagents, not having a proper version control or comments for the codes, and guessing the details long time after an experiment, “How experiments were done” and “How/where data were stored” under the “quality control” might be important contributors towards solving reproducibility crisis [8].
Turning Crisis into Revolution
There are different solutions has been proposed for different aspects of the reproducibility crisis. Open science gained huge attention due to requiring clear explanation, experimental and analysis-wise transparency [2]. Improvement in data analysis such as blind analysis or randomness when it is applicable might be another solution. Some labs tend to discuss less while research in progress. However, routine discussion of research and methods applied can provide careful progress [16]. Developing high standards for the training and reporting, as well as institutional enforcement for the reproducible research with relieving the pressure, encouraging sloppy science, might contribute to the purpose [16].
Recently, some journals have started to publish method focused papers in a separate and detailed method paper format. Yet, Cell has started to publish STAR protocols with detailed and technical peer-review process separately from the original research articles [14]. In this way, main focus will be given to publish “usable” and “robust” protocols that every lab can easily replicate without confusion.
References:
1. Nuzzo, R. Scientific method: Statistical errors. Nature 506, 150–152 (2014). https://doi-org.libproxy1.nus.edu.sg/10.1038/506150a
2. Nuzzo, R. How scientists fool themselves — and how they can stop. Nature 526, 182–185 (2015). https://doi-org.libproxy1.nus.edu.sg/10.1038/526182a
3. Brumfiel, G. Neutrinos not faster than light. Nature (2012). https://doi-org.libproxy1.nus.edu.sg/10.1038/nature.2012.10249
4. https://ed.ted.com/lessons/is-there-a-reproducibility-crisis-in-science-matt-anticole
5. Yanai, I., Lercher, M. A hypothesis is a liability. Genome Biol 21, 231 (2020). https://doi-org.libproxy1.nus.edu.sg/10.1186/s13059-020-02133-w
6. Open Science Collaboration, Estimating the reproducibility of psychological science. Science 349, aac4716 (2015).
7. Munafò, M., Nosek, B., Bishop, D. et al. A manifesto for reproducible science. Nat Hum Behav 1, 0021 (2017). https://doi-org.libproxy1.nus.edu.sg/10.1038/s41562-016-0021
8. Baker, M. How quality control could save your science. Nature 529, 456–458 (2016). https://doi-org.libproxy1.nus.edu.sg/10.1038/529456a
9. Baker, M. 1,500 scientists lift the lid on reproducibility. Nature 533, 452–454 (2016). https://doi-org.libproxy1.nus.edu.sg/10.1038/533452a
10. Fanelli D. Opinion: Is science really facing a reproducibility crisis, and do we need it to?. Proc Natl Acad Sci USA 2018; 115: 2628–2631. DOI: 10.1073/pnas.1708272114
11. Vazire, S. (2018). Implications of the Credibility Revolution for Productivity, Creativity, and Progress. Perspectives on Psychological Science, 13(4), 411–417. https://doi.org/10.1177/1745691617751884
12. Serra-Garcia, M., Gneezy, U. (2021). Nonreplicable publications are cited more than replicable ones. Science Advances, 7(21), eabd1705. https://doi-org.libproxy1.nus.edu.sg/10.1126/sciadv.abd1705.
13. https://www.wired.com/2017/04/want-fix-sciences-replication-crisis-replicate/
14. https://star-protocols.cell.com/protocols/664
15. If you want to be part of the process: https://www-cell-com.libproxy1.nus.edu.sg/star-protocols/reviewers
16. Begley, C., Buchan, A. & Dirnagl, U. Robust research: Institutions must do their part for reproducibility. Nature 525, 25–27 (2015). https://doi-org.libproxy1.nus.edu.sg/10.1038/525025a
