What Are Reproducibility of Results?

Reproducibility of results is the property of a scientific or engineering finding whereby a researcher, using the same input data, computational methods, code, and conditions of analysis as the original study, obtains results consistent with those originally reported. The National Academies of Sciences, Engineering, and Medicine use this definition to distinguish reproducibility from the related but distinct concept of replicability, which requires independent data collection. Reproducibility in this sense is a computational or procedural property: it asks whether the original results can be recovered by rerunning the original workflow, not whether the underlying phenomenon holds under new experimental conditions.

Reproducibility of results is foundational to scientific integrity and engineering practice. It is the minimum standard of transparency that allows independent verification of a claim, enables error detection, and forms the starting point from which broader replication and generalization can proceed. Failures of reproducibility, ranging from undisclosed preprocessing steps to software bugs discovered only after publication, have drawn significant attention in fields including machine learning, computational biology, and materials characterization.

Computational Reproducibility

In computational disciplines, reproducibility depends on capturing the entire software environment used to produce a result: the code, library versions, data, random seeds, and hardware configuration. A model trained on one version of a deep learning framework may produce different numerical outputs on an updated version, even if the algorithm is nominally identical, due to floating-point rounding differences or changed default behaviors. Containerization tools such as Docker and reproducible package managers allow researchers to archive the full software environment alongside their results. Workflow management systems designed for scientific computing, including Snakemake and Nextflow, record the full directed acyclic graph of processing steps so that any output can be traced to its inputs and the exact operations applied. The ACM's artifact review and badging program assigns separate badges for reviewed, available, and reproduced artifacts, providing a formal mechanism for the community to certify that published computational results are reproducible by independent evaluators.

Experimental Reproducibility

In laboratory and field sciences, reproducibility asks whether the same procedure, applied to the same materials or data, yields the same measurement. Calibration drift, batch-to-batch material variability, operator technique differences, and instrument firmware changes can all produce discrepancies that have nothing to do with the correctness of the underlying scientific claim. Reproducibility in this setting depends on protocols written at sufficient detail that a trained practitioner who was not present during the original experiment can reproduce the exact conditions. Metrology, the science of measurement, addresses this systematically through traceability chains that link local measurements to national or international standards, ensuring that a resistance measurement made in one laboratory on one instrument agrees with measurements made elsewhere. The National Institute of Standards and Technology maintains standards and reference materials specifically to enable this kind of inter-laboratory reproducibility.

Policies, Infrastructure, and Open Science

Increasing awareness of reproducibility failures has driven policy changes at funding agencies, journals, and professional societies. IEEE, ACM, and Nature journals have introduced requirements for data availability statements, code deposition, and methods reporting standards that exceed historical practice. Preprint servers and data repositories, including arXiv, Zenodo, and the IEEE DataPort, allow researchers to make data and code publicly available at the time of submission rather than upon request. Pre-registration of analysis plans before data collection reduces the space for post-hoc adjustment of methods to fit results. Research on reproducibility challenges in web-scale measurement studies illustrates how the problem manifests even in observational research where no physical experiment is repeated, because the state of the measured system evolves between the original and the attempted reproduction.

Applications

Reproducibility of results has applications in a range of fields, including:

• Machine learning model development, where benchmark comparisons require consistent evaluation conditions
• Clinical and regulatory engineering, where device testing must yield consistent results across test laboratories
• Materials science, where synthesis and characterization protocols must be documented for independent verification
• Environmental monitoring, where sensor networks must produce traceable, consistent measurements over time
• Software testing and continuous integration, where deterministic builds underpin reliable regression detection