Reproducibility in Scientific Research: Ensuring Reliable Findings
Since I was a little boy, like many Bengalis of my generation, I have been obsessed with Satyajit Ray’s tales about the mythical scientist Professor Shonku. Among his other magical inventions are “Miracurall,” a drug that cures all illnesses except the common cold; “Annihillin,” a pistol that can exterminate any living thing; “Shonkoplane,” a small hovercraft built on anti-gravity technology; and “Omniscope,” which combined the telescope, microscope, and X-ray-scope. Evidently, Prof. Shonku was a brilliant scientist and inventor.
Or was he?
Reproducible research
The fact that none of Shonku’s powerful and useful inventions could be produced in a factory and that only he was capable of manufacturing them was a genuinely disheartening feature of his innovations. Later, after being exposed to the scientific community, I understood that Prof. Shonku couldn’t be considered a ‘scientist’ in the strictest sense of the word for this precise reason. The reproducibility of research is the essence of scientific truth and inventions.
In his 1934 book The Logic of Scientific Discovery, the Austrian-British philosopher Karl Popper wrote: “Non-reproducible single occurrences are of no significance to science.” This said, in some fields, especially observational sciences, where inferences are drawn from events and processes beyond the observer’s control, irreproducible one-time events can still be a significant source of scientific information, so reproducibility is not a critical requirement.
Consider the 1994 collision of Comet Shoemaker-Levy with Jupiter. It offered a wealth of knowledge on the dynamics of the Jovian atmosphere as well as preliminary proof of the danger posed by meteorite and comet impacts. One may recall the famous observation made by Stephen Jay Gould in his brilliant 1989 book Wonderful Life: The Burgess Shale and the Nature of History, that if one were to “rewind the tape of life,” the consequences would surely be different, with the likelihood that nothing resembling us would exist.
“We’re all biased”
However, scientists working in most disciplines do not have that kind of leverage, for sure. In fact, reproducibility – or the lack thereof – has become a very pressing issue in more recent years.
In a 2011 study, researchers evaluated 67 medical research projects and found that just 6% were fully repeatable whereas 65% showed inconsistencies when evaluated again. An article in Nature on October 12, 2023, reported that 246 researchers examined a common pool of ecological data but came to significantly different conclusions. The effort echoes a 2015 attempt to replicate 100 research findings in psychology, but managed to do so for less than half.
In 2019, the British Journal of Anaesthesia conducted a novel study to address the “over-interpretation, spin, and subjective bias” of researchers. One paper had disregarded the potential link between higher anaesthetic doses and earlier deaths among elderly patients. However, by analyzing the same data in another 2019 paper in the same journal, different researchers found different death rates. The new paper also argued that there weren’t enough trial participants present to reach that conclusion, or any conclusion at all, about mortality.
The purpose of such an analysis – publishing two articles based on the same experimental data – was to broaden the scope of replication attempts beyond just techniques and findings. The lead author of the original paper, Frederick Sieber, commended the methodology saying, “We’re all biased and this gives a second pair of eyes.”
Affirming the method
Replicating other people’s scientific experiments appears messy. But could trying to replicate one’s own findings be chaotic as well? According to one intriguing paper published in 2016, more than 70% of researchers have failed to replicate the experiments of other scientists, and more than half have attempted and failed to replicate their own experiments. The analysis was based on an online survey of 1,576 researchers conducted by Nature.
The Oxford English Dictionary’s definition of “reproducibility” is “the extent to which consistent results are obtained when produced repeatedly.” It is thus a fundamental tenet of science and an affirmation of the scientific method. In theory, researchers should be able to replicate experiments, get the same outcomes, and draw the same conclusions, thus helping to validate and strengthen the original work. Reproducibility is significant not because it checks for the ‘correctness’ of outcomes but because it ensures the transparency of exactly what was done in a particular area of study.
Axiomatically, the inability to reproduce a study could have a variety of causes. The main factors are likely to be pressure to publish and selective reporting. Other factors include inadequate lab replication, poor management, low statistical power, reagent variability, or the use of specialized techniques that are challenging to replicate.
Our responsibility
In this milieu, how can we improve the reproducibility of research?
Some obvious solutions include more robust experimental design, better statistics, robust sharing of data, materials, software, and other tools, the use of authenticated biomaterials, publishing negative data, and better mentorship. All of these, however, are difficult to guarantee in this age of “publish or perish” – where a researcher’s mere survival in the academic setting depends on their performance in publishing.
Funding organizations and publishers can also do more to enhance reproducibility. Researchers are increasingly being advised to publish their data alongside their papers and to make public the full context of their analyses. The ‘many analysts’ method – which essentially employs many pairs of eyes in which different researchers are given the same data and the same study questions – was pioneered by psychologists and social scientists in the middle 2010s.
All this said, today, it seems that we simply can’t depend on any one outcome or one study to tell us the complete story because of the pervasive reproducibility issue. We are more acutely experiencing this awful state. Maybe we will have to understand that it is our responsibility to ensure reproducibility in our research – more so to avoid risking becoming a fictitious scientist like Prof. Shonku.
Atanu Biswas is Professor of Statistics, Indian Statistical Institute, Kolkata.
Fun Fact: Satyajit Ray, the renowned filmmaker and writer, created the mythical scientist Professor Shonku in his stories, capturing the imagination of many Bengalis and intertwining science and fiction.
Mutiple Choice Questions
1. Which of the following is NOT one of the inventions of Professor Shonku?
a) Miracurall
b) Annihillin
c) Shonkoplane
d) Omniscope
Explanation: The passage states that “Among his other magical inventions are ‘Miracurall,'” which implies that Miracurall is indeed one of Professor Shonku’s inventions.
2. According to the passage, why couldn’t Professor Shonku be considered a ‘scientist’ in the strictest sense?
a) He lacked the knowledge and expertise of a scientist.
b) His inventions were not reproducible by others.
c) He didn’t publish his research findings.
d) He only focused on one specific scientific field.
Explanation: The passage states that “none of Shonku’s powerful and useful inventions could be produced in a factory and that only he was capable of manufacturing them,” which indicates that his inventions were not reproducible by others.
3. In which book did Karl Popper write about the importance of reproducibility in scientific research?
a) The Logic of Scientific Discovery
b) Wonderful Life: The Burgess Shale and the Nature of History
c) The Oxford English Dictionary
d) The Miracurall Inventions
Explanation: The passage states that “In his 1934 book The Logic of Scientific Discovery, the Austrian-British philosopher Karl Popper wrote: ‘Non-reproducible single occurrences are of no significance to science.'”
4. What significant event provided scientific knowledge about the dynamics of the Jovian atmosphere?
a) The collision of Comet Shoemaker-Levy with Jupiter
b) The publication of the book Wonderful Life: The Burgess Shale and the Nature of History
c) The replication of 100 research findings in psychology
d) The creation of the Shonkoplane by Professor Shonku
Explanation: The passage states that “Consider the 1994 collision of Comet Shoemaker-Levy with Jupiter. It offered a wealth of knowledge on the dynamics of the Jovian atmosphere.”
5. What percentage of medical research projects were found to be fully repeatable in a 2011 study?
a) 6%
b) 65%
c) 100%
d) 50%
Explanation: The passage states that “researchers evaluated 67 medical research projects and found that just 6% were fully repeatable.”
6. What method was used to address the “over-interpretation, spin, and subjective bias” of researchers in a study conducted by the British Journal of Anaesthesia?
a) The replication of experiments by different researchers
b) The use of authenticated biomaterials
c) The publication of negative data
d) The creation of novel scientific inventions
Explanation: The passage states that “The purpose of such an analysis – publishing two articles based on the same experimental data – was to broaden the scope of replication attempts beyond just techniques and findings.”
7. According to the passage, what percentage of researchers have failed to replicate the experiments of other scientists?
a) More than 70%
b) More than 50%
c) Less than 50%
d) Less than 30%
Explanation: The passage states that “more than 70% of researchers have failed to replicate the experiments of other scientists.”
8. What is the definition of “reproducibility” according to the Oxford English Dictionary?
a) The extent to which consistent results are obtained when produced repeatedly.
b) The ability to publish research findings.
c) The replication of experiments using specialised techniques.
d) The creation of scientific inventions.
Explanation: The passage states that “The Oxford English Dictionary’s definition of ‘reproducibility’ is ‘the extent to which consistent results are obtained when produced repeatedly.'”
9. Which of the following is NOT mentioned as a potential cause for the inability to reproduce a study?
a) Pressure to publish and selective reporting
b) Inadequate lab replication
c) High statistical power
d) Reagent variability
Explanation: The passage states that “The main factors are likely to be pressure to publish and selective reporting. Other factors include inadequate lab replication, poor management, low statistical power, reagent variability, or the use of specialised techniques that are challenging to replicate.”
10. What do researchers need to do in order to enhance reproducibility according to the passage?
a) Share data and materials, publish negative data, and improve experimental design
b) Get authenticated biomaterials and better mentorship
c) Depend on one outcome or one study to tell the complete story
d) Replicate experiments using the same study questions
Explanation: The passage suggests that researchers should do more to enhance reproducibility, such as sharing data and materials, publishing negative data, improving experimental design, and seeking better mentorship.
Brief Summary | UPSC – IAS
The article discusses the issue of reproducibility in scientific research. The author reflects on the fictional character Professor Shonku, who was a brilliant scientist and inventor but whose inventions could not be replicated by anyone else. The author argues that reproducibility is essential for scientific truth and inventions. They highlight examples of studies in various fields that have failed to be fully repeated, demonstrating the lack of reproducibility in scientific research. The article suggests potential solutions to improve reproducibility, such as robust experimental design, data sharing, and publishing negative results. Ultimately, the author emphasizes the responsibility of researchers to ensure reproducibility in their work.
