Professor Nosek: The relevant terms have not been used consistently in the past, largely because there has not been enough attention to their importance. But, there is a slowly emerging consensus. Reproducibility refers to using the same analysis on the same data to see if the original finding recurs. Robustness refers to using different analyses on the same data to test whether the original finding is sensitive to different choices in analysis strategy. And, replicability refers to testing the same question with new data to see if the original finding recurs.

From left to right: Prof. Brian Nosek, Prof. Christine Mummery, Dr Leonardo Scarabelli, Prof. Vitaly Podzorov.

Confusingly, reproducibility is also used to refer to all three activities. I am hoping that something like “repeatability” becomes the general term referring to all of them.

Professor Mummery: There are multiple ways of interpreting this; one is the way it is widely used in the laboratory, the other is more formal. In the laboratory, we use it to mean that if the same researcher carries out an experiment (say) three times, or that another researcher in the same lab carries out the same experiment and, in both cases (across operators), the outcome and conclusions are the same, then we usually refer to it being reproducible. If a researcher in another lab carries out the same experiment using the same protocol or equivalent (or the same) materials, then we usually say the outcome is robust (across labs). More formally, especially in cohort studies or other large datasets, reproducibility is defined a bit more narrowly: same analysis of the same data gives the same outcome. Robustness is when the same dataset but a different analysis method gives the same outcome.

Dr Scarabelli: The possibility of reproducing an experiment and its results, or the derived conclusions (starting from the same dataset) by a team that was not involved in the original work as close as possible to the original work itself.

Professor Podzorov: Reproducibility in science involves the ability of independent research groups to obtain the same main result as that reported in the original publication in question by following their published procedure, recipe or protocol. Such confirmational studies should occur within a reasonable time frame and be well-documented in peer-reviewed journals. The term ‘result’ encompasses more than observations of natural phenomena or behavior. It may include specific numerical values associated with performance or functionalities, such as charge carrier mobilities, critical temperatures, power conversion efficiencies, transcription rates and precision in polymerase chain reaction research, or toxicity-efficacy window in drug development—the metrics the community continually seeks to improve.

Professor Nosek: Transparency is important because science is a show-me enterprise, not a trust-me enterprise. Confidence in scientific claims is rooted in being able to interrogate the evidence for the claims and how that evidence was generated. Without transparency, the self-corrective processes of science are hampered.

Reproducibility is important because the social process of interrogating evidence and claims presumes that the evidence is accurately provided. If the findings are reported erroneously, then there is little value in discussing them. Robustness is important to provide context for how dependent the findings are on the decisions about how to analyze the data to know whether to include those decisions in debate about the evidence and claims. And, replicability is important to know if the claims are regularities that can be observed again. If they are not, then the value of the claims for constructing theories and explanations is limited.

Professor Mummery: We can learn a lot when some experiments turn out not to be reproducible: is it something to do with the method, reagents, apparatus used, is it the operator, or, in some cases, the biological variability of the system. It can help identify parameters requiring essential control or it can tell us something about intrinsic (say biological) variability we might not understand. We may not be able to figure it out, but transparency means others will have access to that information and will also be able to give their own interpretation to why things may not be reproducible. Great discoveries have been made through careful analysis of discrepancies that would likely not have been possible without transparency. A second paper, reporting a novel finding and confirming it, should be as important as the first in this context.

Dr Scarabelli: Simply put, I believe that reproducibility is what makes up the vast majority of the impact of a research discovery, as it allows others to build on the results and move the field forward. Without transparency or reproducibility, the discovery itself, no matter how potentially groundbreaking, is ultimately useless for the big picture of scientific advancement. In cases where repeating the experimental conditions is almost impossible due to technical limitations, reproducibility is the ability to reach the same conclusions starting from the same raw dataset. Here the treatment of the data (and its description in the manuscript) plays the main role in the reproducibility of a result.

Professor Podzorov: Reproducibility is one of the most distinctive and fundamental attributes of true science. It acts as a filter, separating reliable findings from less robust ones by establishing the value and the intrinsic nature of the former and thus serving as a reliable guide for further research. Reproducibility potentially saves significant time and resources in subsequent scientific endeavors. Efforts to validate or challenge original claims, including identifying and flagging researchers or groups systematically publishing unreliable results, are essential for the development of science. In the long term, such efforts collectively benefit the research community by helping to optimize resource allocation. Transparency in scientific research and publications involves sufficiently detailed disclosure of technical and methodological aspects, along with the availability of raw data, necessary for a scientific work to be reproduced by others.

Professor Nosek: I would frame it more positively. The last decade exposed challenges in transparency, reproducibility, and replicability. The research community’s strong response to that is evidence that the scientific enterprise cares deeply about credibility and addressing weaknesses in the scientific process. Some of the challenges are very deeply rooted and will take a lot of time and effort to reform. But, that reform project is underway.

Professor Mummery: At the level of the individual researcher, possibly the need to be first in order to be “counted”, because so much hangs on being the first to make a discovery. Transparency is essential for the self-correcting mechanism in science to work as it should. Few papers actually mention lack of reproducibility or robustness since this can be seen as incompetence; however, the inability to reproduce certain experiments across research groups is widely discussed at conferences, and those who do not participate in those discussions are dis-advantaged and may waste considerable time trying to reproduce the irreproducible. The driver of the current reproducibility crisis could be perceived as the need to report perfect results in publications. A few dramatic retractions might give the impression of “crisis,” but rather an overstatement at this point, and many see this as self-correction at work.

Dr Scarabelli: The origin and driving force of this crisis originates in a downward spiral that the scientific community enters, that is no longer sustainable and is beginning to affect the community as a whole.

We are witnessing a constant increase in the separation between the market value of scientific publication and the incentive based on the bibliometric evaluation of researchers that supports it, on the one hand, and the indispensable necessity for the progress of science that consists in sharing knowledge in the most controlled and reproducible way possible, on the other hand. In other words, these two aspects are more and more chronically in contrast with one another. Ultimately, this led to a scientific discovery that is forcing researchers to publish “as quick as possible” and not “as good as possible”. In my opinion the core of the problem is to be found in the way we evaluate a scientific output, which affects the funding cycle and publication system, originating the downward spiral that led to the crisis we are witnessing.

Professor Podzorov: This crisis is primarily fueled by the desire for more attractive or rapid publications, with authors frequently engaging in practices inconsistent with academic integrity standards, or unintentionally making errors due to a lack of expertise or insufficient time and effort invested in their research. In my view, the most significant factor is the overreliance on scientometrics in the evaluation and reward of scientists. High scientometric indices help landing jobs, getting promotions, gaining competitive edge in grant applications, or receiving awards and prizes. The commercialization of academic publishing and the lack of financial or institutional accountability of individual researchers for generating irreproducible research also contribute to this crisis. Together with virtually non-existent financial support for identifying and flagging unreliable publications, these factors trigger and sustain an uncontrollable snowball effect, fostering an environment where the sole purpose of publishing a paper is to advance one’s academic career rather than make a meaningful and lasting contribution to science.

Professor Nosek: Authors can embrace transparency across the research lifecycle. Exposing how research was planned and conducted, and sharing the underlying data, materials, and code, provides others the opportunity to understand how the research was conducted and interrogate the strength of the evidence. Moreover, it makes it easier for others to challenge or build on the work. Acting transparently is embracing intellectual humility and prioritizing getting it right over being right.

Professor Mummery: Encourage editors to accept inclusion of a section in a paper on what was tried and did not work. Would save many people with the same idea a lot of time and provide transparency in thought processes, from first hypothesis to planned experiments and their outcome.

Dr Scarabelli: Undoubtedly, we can all take advantage of the digital tools that are now ubiquitous and endeavor to ensure that one’s research contributes as much as possible to the advancement of the field by including any detail that might help others reproduce the results presented. However, the real challenge is in the development of a “reproducibility culture”, where the attention is once again given to what a result is teaching us and how it can impact the field or the community. All these aspects are very commonly discussed within group meetings and research groups, but very rarely emerge during conferences or papers. Ultimately, this is a factor that seriously compromises reproducibility.

Professor Podzorov: Individual researchers should proactively promote reproducible and transparent science within their respective fields. This begins with fostering a culture of critical thinking, especially when reading original research papers. Beyond titles, abstracts, and stated claims, researchers need to delve deeper and critically evaluate the content. I would say that it is essential for active researchers, including PhD students and postdocs, to allocate a portion of their time to identifying, analyzing, and reporting cases of bad science. For example, PhD students and postdocs could be encouraged to identify scientific and technical errors, inconsistencies or misconduct in papers in their fields. Financial support for such an effort should come from the respective academic programs. Such work, culminating with critical reviews published in journals or other open sources, must be considered as a requirement by degree-granting academic institutions and funding agencies.

Professor Nosek: Reviewers can ask for greater transparency of the underlying content of the research so that they need not depend exclusively on the reporting of the research. Also, reviewers can embrace that the most important part of good science is in the asking of questions and pursuing rigorous methods to test them. An overemphasis on novel, positive, and tidy results creates all the wrong incentives for encouraging authors to conduct rigorous, transparent research and just report what they learned.

Professor Mummery: Take into account what is truly necessary for the revision of a paper and consider what is feasible within the time frame usually set by the journal. Show appreciation of transparency, including during the revision process.

Dr Scarabelli: I think the reviewers have very little chances to make a dent in this crisis system. Of course, all reviewers should critically read the experimental sections, putting themselves in the situation of having to reproduce that experiment. However, in most of the cases, the only real way to test reproducibility would be to repeat the experiments yourself, which is practically impossible, logistically, experimentally, and financially, without considering the amount of time that should be sacrificed for this to happen.

Professor Podzorov: One obvious problem in my field is the frequent occurrence of reviews that are perfunctory, lacking depth, rigor or rational criticism. First, reviewers whose expertise is not well aligned with the core aspects of manuscripts must decline review assignments. Second, if a reviewer is unable to dedicate the necessary time and energy to conduct a thorough in-depth review, they should decline the assignment instead of resorting to a superficial report.

Professor Nosek: Editors and publishers can adopt transparency policies following the TOP Guidelines (https://cos.io/top/) to incentivize or require authors to be more open in their research. They can also adopt more open policies for the review process so that readers can see the scholarly discussion about the merits of the research. Finally, they can adopt publishing models like Registered Reports (https://cos.io/rr/) in which commitments to publish findings are made prior to knowing the research outcomes so that the reward of publication is based on asking important questions and conducting rigorous research.

Professor Mummery: Consider if there is sufficient evidence to support the hypothesis without a large number of experiments that are not all relevant. Identify key experiments necessary. In my specific field of stem cell research, they might consider using the ISSCR Guidelines on Stem Cell Research, particularly the checklist for reporting results, to benchmark data in papers (International Society for Stem Cell Research; https://www.isscr.org/guidelines). What is needed to draw robust conclusions when using stem cell models, what controls are needed and what QA/QC should be carried out with the starting materials.

Dr Scarabelli: “For the betterment of mankind and the progress of science” should be more than just a motto, but a commitment. Editors and publishers are the first line of defense to try and stop this spiral and steer the system in a positive direction. It is becoming more and more challenging to remain up-to-date with the sheer amount of papers published, the majority of which fail to meet reproducibility standards, to the point that a substantial re-optimization of a published protocol is considered a given in many scientific fields. Ultimately, the focus must return to “publish less, but publish better”, while the number of published manuscripts is constantly increasing, with an ever-increasing burden on the peer-review process (which rests mainly on the shoulders of the researchers themselves). This is a very complicated problem for which there is no simple solution. It might be interesting to explore a hybrid system where a certain number of professional reviewers (employed by the journal) participate in the revision process, although the clear risk here is that the additional costs could (once again) be passed on to the researchers. Artificial intelligence tools could potentially assist the review process (e.g., cross-checking references or novelty claims against the state of the art), but this option should be considered with extreme caution and subject to rigorous scrutiny.

Professor Podzorov: At the stage of peer review, editors would need to be more careful in selecting reviewers, with the potential conflicts of interest thoroughly examined, making sure that the selected reviewers are able to provide a high-quality, in-depth technical evaluation of manuscripts. Superficial reviews should be discarded. Post publication, editors need to promptly evaluate any reports of potentially flawed papers, engage an independent team of experts to investigate the matter and prioritize corrective actions if deemed necessary. I would also propose that publishers incentivize whistle-blowers with various forms of rewards if a reported paper is proven to be significantly flawed.

Professor Nosek: Funders can likewise adopt TOP-aligned transparency policies for their grantees to share all research outputs, regardless of whether the research ultimately leads to a publication or not. Moreover, funders can partner with journals through Registered Reports to create a single review process for proposed research that, when accepted, the journal commits to publish the findings and the funder provides the funding to conduct the research (see, for example, https://www.cos.io/consciousness). Finally, funders can do more experimentation with funding models to incentivize different research strategies and discover ways to accelerate discovery.

Professor Mummery: Create new/supplementary funding to allow papers to be properly revised. Encourage all research outcomes to be openly available, even if not published, to avoid unnecessary duplication of work.

Dr Scarabelli: Funders and sponsors can help solve this crisis by identifying new criteria for the evaluation of the research portfolio of a candidate or applicant that do not rely exclusively on the number of publications, magazine, or publisher group. In the last years there has been a significant effort in this direction, at least in the European Union, with projects such as CoARA (and others). I hope that this process will ultimately put an end to this crisis and set the editorial system back on track.

Professor Podzorov: Allocate dedicated funding to efforts on identifying and reporting on irreproducible or erroneous research. Upon receiving a report on a potentially problematic paper, funding agencies should prioritize conducting an investigation by an independent team of experts. If concerns are substantiated, the financial support provided to responsible authors should be terminated. Importantly, a mandatory refund of the budget allocated for the compromised work must become a common practice. A suspension of eligibility for funding applications must follow. I am convinced that the proposed financial incentives and deterrents are crucial and necessary for addressing the escalating reproducibility crisis in science.