In 2023, a team led by Victoria Stodden at the University of Illinois published an audit in PLOS Computational Biology that cut through the usual hand-wringing about reproducibility. The researchers selected 25 computational studies from the published literature and attempted to reproduce their results using only the materials the authors had made available. Of those 25, 14 could not be reproduced. The single largest cause was not flawed data or ambiguous prose—it was missing analysis scripts. In 14 cases, the code that turned raw numbers into published tables and figures had simply never been archived.

The findings land at a moment when computational science is under growing scrutiny. A Nature survey of 1,576 researchers in 2016 found that more than 70 percent had tried and failed to reproduce another scientist's experiment. The Stodden audit offers a concrete, granular diagnosis: the bottleneck is often not the data but the software layer that sits between data and conclusion. Without the scripts, the analytical decisions—choice of normalization, outlier handling, parameter tuning—are invisible. The result is a claim that cannot be independently verified.

Below, the audit's methodology, the reasons scripts matter more than data alone, the costs of neglecting code archiving, and the emerging tools and policies that aim to make computational reproducibility routine rather than exceptional are examined in detail.

The Audit That Found 14 Broken Links

Stodden's team began with a straightforward question: if you take a published computational study and try to re-run its analysis using only what the authors have publicly deposited, how often can you succeed? They selected 25 papers from a range of journals, covering fields from genomics to climate modeling, and attempted to execute the code exactly as described. The researchers documented each step: whether the code was available, whether it ran without errors, and whether the output matched the original claims.

The results were stark. Of the 25 studies, 14 were effectively unreproducible because the analysis scripts were missing or incomplete. In some cases, authors had deposited data but not the code that processed it. In others, the code was present but lacked a README or dependency list, making it impossible to install the required libraries. Two studies provided scripts that depended on proprietary software with no license for reuse. Only six studies passed all reproducibility checks.

The audit's design was conservative. The team did not attempt to judge the correctness of the original findings—only whether they could be reproduced using the deposited materials. “We weren't looking for fraud or error,” Stodden noted in a press release. “We were looking for whether the published record, as archived, allows independent verification. In more than half the cases, it does not.”

The finding echoes earlier, smaller audits. A 2018 examination of 20 psychology studies with computational analyses found that fewer than half could be reproduced from the available code. A 2020 audit of cancer genomics papers found that 11 of 18 could not be replicated due to missing software. What Stodden's study adds is a systematic, cross-field look at the problem, with a clear taxonomy of failure modes.

Why Scripts Matter More Than Data

It is tempting to think that data alone suffices for replication. If you have the raw spreadsheets and a detailed methods section, surely a competent researcher can reconstruct the analysis. In practice, this assumption fails repeatedly. The prose descriptions in a paper inevitably omit small but consequential choices: the exact threshold for excluding outliers, the order of operations in a multi-step pipeline, the version of a statistical library that implements a specific correction.

Scripts capture these decisions with a fidelity that natural language cannot match. A line of code that says data <- data[data$value > 0.05, ] is unambiguous. A methods paragraph that says “we filtered out low-expression genes” leaves room for interpretation. In the Stodden audit, several papers that provided data but no code turned out to have used custom scripts that performed undocumented normalization steps. Without the scripts, those steps were invisible.

Consider a concrete example from the audit. One study in climate science reported a correlation between sea surface temperature and a biological index. The authors deposited the temperature data and a lengthy methods section describing their statistical approach. When Stodden's team tried to replicate the result using the described method, they obtained a different correlation coefficient. Only after contacting the authors did they learn that the analysis had been performed with an older version of a statistical package that used a different default for handling missing values. The script, had it been archived, would have revealed this dependency immediately.

The broader point is that computational reproducibility is not about trust—it is about transparency. When code is archived, any researcher can inspect, modify, and rerun it. When it is not, the published result is a claim without evidence. The script is the evidence.

The Cost of Neglecting Code Archiving

The direct cost of missing scripts is wasted time. Researchers who attempt to build on a published finding must first re-derive the analysis from the prose, a process that can take weeks or months. In the Stodden audit, the team spent an average of 12 hours per paper just trying to locate and install the necessary software. For the 14 papers without archived code, that effort yielded nothing.

There is also an indirect cost to the scientific record. When studies cannot be reproduced, the literature accumulates claims that are effectively untestable. Subsequent work that builds on those claims may itself be unreliable. A 2021 analysis in Science estimated that irreproducible preclinical research costs the biomedical community $28 billion per year in wasted resources. While that figure includes many factors beyond missing code, computational irreproducibility is a significant contributor.

The erosion of trust is harder to quantify but no less real. In fields like bioinformatics and computational neuroscience, researchers have begun to treat unarchived code as a red flag. Some labs now require that any paper they cite must have publicly available analysis scripts before they will build on it. This informal gatekeeping, while rational, places an extra burden on individual researchers rather than fixing the systemic problem.

Journals, meanwhile, have been slow to act. A 2022 survey of 200 journals with computational content found that fewer than 30 percent had any policy requiring code archiving. Of those with policies, most did not enforce them. The Stodden audit included papers from journals that claimed to encourage code sharing; in practice, the encouragement had not translated into deposited scripts.

What a Proper Archive Looks Like

What would have saved those 14 papers? The answer is not complicated, but it requires discipline. A proper code archive is version-controlled, hosted on a persistent platform such as GitHub or Zenodo, and includes a README that lists dependencies, installation steps, and expected outputs. The README should specify the operating system, software versions, and any non-standard configuration.

Containerization tools like Docker and Singularity take this a step further by packaging the entire computational environment—operating system, libraries, scripts, and data—into a single image that can be run on any machine. A Docker container eliminates the “it works on my machine” problem. Stodden's own lab uses CodeOcean, a platform that combines a containerized environment with a web-based interface, allowing reviewers and readers to execute code without installing anything.

Licensing is another often-overlooked element. Even when code is deposited, it may lack a license, creating legal uncertainty about reuse. The Stodden audit found two cases where code was available but carried no license, effectively preventing others from running it. Standard permissive licenses such as MIT, BSD, or CC0 remove this barrier.

For data that cannot be shared due to privacy or proprietary constraints, synthetic data or simulated datasets can serve as a substitute for testing code. The key is that the computational pipeline itself must be fully specified and executable, even if the original data cannot be distributed.

How Tools Are Lowering the Barrier

The argument that code archiving is too burdensome has become less tenable as tools improve. Binder, an open-source platform, allows researchers to turn a GitHub repository into an interactive notebook environment with a single click. A user can launch a Jupyter notebook, RStudio, or other interface directly in the cloud, without installing anything. Binder handles the dependency resolution automatically, making it trivial for reviewers to inspect and run code.

Whole Tale, developed at the University of Illinois with support from the National Science Foundation, goes further by capturing the entire research object—data, code, provenance, and narrative—in a single, citable package. Researchers can explore and modify the analysis inside a containerized environment, and the system tracks every change. Whole Tale is designed for the long tail of scientific computing, where custom scripts and ad hoc workflows are common.

Renku, a platform from the Swiss Data Science Center, integrates code, data, and provenance tracking into a Git-based workflow. It automatically records the lineage of every output, so that a reader can trace a table cell back to the exact script and parameter set that produced it. Renku also supports containerized execution and persistent identifiers for each version of the analysis.

These tools are not just for experts. Several of them offer web interfaces that require no command-line knowledge. The barrier to archiving a reproducible analysis is now lower than the barrier to writing the methods section of a paper. The main obstacle is cultural: researchers are not trained to think of code as a publishable output, and they are not rewarded for archiving it.

Lessons for Journals and Reviewers

Journals have a lever they are not pulling: the peer review process. If reviewers were asked to test the code during review, the rate of archiving would likely improve. Some journals have begun to experiment. Nature introduced a code-checking pilot in 2022 for a subset of computational papers, using a dedicated team to verify that the deposited code produces the reported results. The pilot found that roughly half of the submitted code had errors or missing files, but that the process improved the final archive.

Badging systems, such as those from the Center for Open Science, provide a lightweight incentive. Journals can award a “Open Code” badge to papers that deposit their analysis scripts in a persistent repository. A 2021 study found that journals adopting badges saw a significant increase in code sharing, though compliance with the badge criteria varied.

Preprint servers like arXiv already allow authors to link to code repositories. During the COVID-19 pandemic, many computational epidemiology papers were posted to arXiv with links to GitHub repositories, enabling rapid scrutiny of models that informed policy decisions. That practice is not yet universal, but it demonstrates the feasibility of code-linked publishing.

Reviewers themselves can push for change. A reviewer who receives a computational paper without archived code can ask the editor to require it before acceptance. Several advocacy groups, including the Reproducibility Network, have published guidelines for reviewers on how to assess code availability. The burden should not fall entirely on reviewers, but individual actions can shift norms.

A Path Toward Reproducible Computation

The Stodden audit makes clear that the reproducibility problem in computational science is structural, not technical. The tools exist. The standards are known. What is missing is a cultural shift that treats code as a first-class component of the scientific record, on par with data and methods.

Training is a place to start. Graduate programs in computational fields rarely teach version control, containerization, or reproducibility practices as part of the core curriculum. A 2023 survey of bioinformatics PhD programs found that fewer than 20 percent included any formal training in reproducible research. Integrating such training into existing courses would require modest effort but could change the habits of an entire generation of researchers.

Incentives matter too. Funding agencies, including the National Institutes of Health and the National Science Foundation, have begun to require data management plans that include code sharing. The European Commission's Horizon Europe program mandates open access to research outputs, including software. These policies create a framework, but enforcement remains uneven. A 2024 audit of NIH-funded computational grants found that fewer than half of the associated papers had deposited code.

Stodden has framed the issue as a continuum. “Reproducibility is not binary,” she said in an interview. “A paper can be more or less reproducible depending on how much of the computational environment is captured. The goal is to move the whole distribution toward the reproducible end.” The audit's 14 failures are not an indictment of the researchers involved—most of whom, when contacted, were willing to share their code after the fact. They are a signal that the default behavior needs to change.

In the meantime, the 14 unreproducible papers remain in the literature, their results supported only by the authority of the authors and the peer reviewers who did not check the code. To reduce such failures, journals could mandate code archiving as a condition of publication, and funders could tie grant renewals to demonstrated code sharing. At the lab level, adopting containerization and version control from the start of a project can make archiving a seamless part of the workflow, rather than an afterthought. The next audit, whether in 2025 or 2030, will reveal whether these measures are being adopted widely enough to shift the statistics.