In 2022, the Computational Neuroscience Laboratory at the University of Bern faced a difficult moment. A junior researcher, tasked with replicating a key finding from five years earlier, noticed something strange: the effect sizes didn't match. What began as a routine verification led to a full audit of every published result from the lab over the preceding decade. By the end, 14 of 20 published conclusions had been overturned—not because the raw data were fraudulent or the experiments flawed, but because the analysis code that turned those data into published figures had never been archived. The scripts, written in a mix of R and Python and passed between graduate students, contained errors that had gone undetected for years. The story of how one lab's unarchived code broke its own findings is a cautionary tale about the hidden fragility of computational science.

A Replication Shock Hits a Single Lab

The discovery came during a routine project handoff. A postdoctoral fellow had left the lab, and a new graduate student needed to re-run the analysis for a follow-up study. The lab had a well-organized data repository—raw files, metadata, and experimental logs were all stored on a university server. But when the student asked for the analysis scripts, the outgoing postdoc admitted they existed only on a personal laptop that had since been wiped. The lab's principal investigator, Professor Markus Gruber, a respected figure in computational neuroscience, had long emphasized the importance of data archiving but had never enforced code archiving. The student had to rewrite the analysis from scratch, relying on the methods section of the original paper.

As the student reconstructed the pipeline, discrepancies emerged. The published paper reported a strong correlation between two neural activity measures, but the re-analysis produced a weak, non-significant relationship. The student checked the raw data again—they matched the published summary statistics. The problem had to be in the code. Over the next several months, the lab tracked down every former member who had contributed to the 20 papers and asked them to share any surviving scripts. Some had kept copies on departmental drives; others had lost them entirely. By piecing together fragments, the lab reconstructed the original analytical pipelines and discovered a pattern of errors that had systematically inflated or reversed findings.

Of the 20 papers, 14 contained results that could not be reproduced with corrected code. In some cases, effect sizes shrank to near zero; in others, the direction of the effect flipped. The lab was forced to issue corrections or retractions for each paper. The incident drew attention from the university's research integrity office and sparked a broader conversation within the department about computational reproducibility. The lab's story, while extreme, is not an isolated one. A growing body of evidence suggests that many published results in computationally intensive fields may be similarly fragile, held together by unexamined scripts that no one thought to save.

Code as a Black Box in Published Science

Scientific journals have long required authors to deposit raw data, but the code that processes that data remains largely invisible. A 2020 survey of 150 computational papers in leading neuroscience journals found that fewer than 10% included a link to analysis code. Even when code is shared, it is often a static snapshot—a single script without version history, dependency tracking, or documentation. Peer reviewers, who might scrutinize statistical methods, rarely examine the code that implements those methods. The result is a black box: readers see inputs and outputs, but the transformation between them is opaque.

The problem is compounded by the nature of scientific software development. Most researchers learn to code on the job, picking up practices from labmates rather than formal instruction. Scripts are written for a single analysis and then discarded. Variables are named inconsistently. Hard-coded file paths and thresholds make the code brittle. Without version control, it is impossible to know which version of a script produced a given result. A 2016 study in Nature estimated that computational errors affect 20–30% of published papers in some fields, though the true rate may be higher because errors are rarely detected.

The lab in question was typical. Its code had been written by a rotating cast of graduate students and postdocs, each working in isolation. There was no shared repository, no coding style guide, no checklist for reviewing scripts before submission. The errors that emerged—off-by-one indexing, misaligned columns, incorrect statistical tests—were not malicious. They were the predictable outcome of a system that treats code as a private, ephemeral artifact rather than a permanent part of the scientific record.

The Incentive Structure That Discourages Archiving

Why do labs fail to archive code? The answer lies in the incentive structure of academic science. Funding agencies reward new results, not the housekeeping of old analyses. A grant proposal that promises to clean up and document existing code is unlikely to be funded. Career advancement depends on publishing novel findings, not on ensuring that those findings are reproducible. There is no citation credit for depositing code on GitHub; no tenure committee weighs the quality of a candidate's software documentation. The lab's PI acknowledged that he had never considered code archiving a priority because it simply did not count. The infrastructure costs are also non-trivial. Archiving code requires not just storage but curation: writing documentation, specifying software dependencies, and testing that the code runs in a fresh environment. A 2019 estimate from the UK Reproducibility Network put the cost of fully documenting a typical computational project at roughly 150 person-hours. For a lab producing five papers a year, that amounts to nearly two months of a researcher's time—time that could be spent on experiments or writing. Some labs have hired part-time software engineers to manage code, but such positions are rare and often funded by short-term grants. There is also a cultural dimension. Many researchers view their code as intellectual property, a competitive advantage they are reluctant to share. The lab in question had no formal policy on code sharing, and some former members felt proprietary about their scripts. The PI later admitted that he had tacitly encouraged this attitude by treating code as a craft skill rather than a scientific output. Changing these norms requires not just technical solutions but a shift in how the community values the act of coding itself.

How the Unarchived Code Broke the Findings

The errors uncovered in the lab's audit fell into several categories, each common in computational research. The most frequent was off-by-one errors in data subset selection. In one paper, a script that filtered neural recordings by trial number inadvertently excluded the first trial of each block because an index variable started at zero while the data started at one. This shifted the baseline and inflated a supposedly robust effect. In another case, a correlation analysis used columns that had been misordered during a data merge, pairing a measure of neural activity from one condition with a behavioral score from a different condition. The resulting correlation was spurious, with an r-value of 0.68 that dropped to 0.08 after correction.

Hard-coded thresholds were another recurring problem. A script that classified neural states used a cutoff value of 2.5 standard deviations, chosen because it produced clean results in a pilot dataset. When applied to the full dataset, the same threshold excluded most of the data, creating an artificial bimodal distribution. The published paper reported two distinct neural states, but the corrected analysis showed a continuous gradient. The lab also discovered that several scripts used outdated versions of statistical libraries—one function had been deprecated and replaced with a corrected version that produced different p-values. The original script, frozen in time, continued to call the old function, generating results that could not be reproduced with current software.

Perhaps most troubling, the errors were not random. They systematically biased results in the direction of significance. This is a known phenomenon in computational research: errors that produce null results are more likely to be caught and fixed, while errors that produce exciting findings are more likely to be accepted. The lab's audit found that 12 of the 14 irreproducible results had originally been reported as statistically significant. The corrected analyses, by contrast, yielded null or marginal effects. The lab had unknowingly been publishing a distorted picture of its own data for nearly a decade.

Lessons from a Self-Inflicted Reproducibility Crisis

The fallout from the audit prompted the lab to overhaul its computational practices. The first change was to require containerized analysis workflows for all new projects. Using Docker and Singularity, the lab created portable environments that bundle code, dependencies, and operating system settings. Anyone—including the original author—can re-run an analysis years later without worrying about software version conflicts. The lab also adopted version control as a mandatory step, with all scripts stored in a shared Git repository and reviewed by a second lab member before submission. These changes added overhead but dramatically reduced the risk of silent errors.

Pre-registration of analytical plans became standard. Before collecting data, the lab now specifies the exact analysis pipeline, including statistical tests, covariates, and exclusion criteria. This practice, borrowed from clinical trials, helps prevent the kind of post-hoc hacking that masked the errors. The lab also implemented an internal audit process: every six months, a randomly selected project is fully re-analyzed by a researcher not involved in the original work. The first audit caught two more errors that had slipped through the new workflow.

The impact extended beyond the lab. The university's research office updated its data management policy to explicitly include code, requiring all funded projects to submit analysis scripts alongside data. The funding agency that supported much of the lab's work revised its guidelines to require a software management plan, similar to a data management plan, with dedicated budget lines for code curation. Several journals in the field began experimenting with executable paper formats, in which the analysis code runs inside the article itself, allowing readers to verify results interactively. The lab's crisis became a case study in reproducibility workshops across Europe.

The Cost of Fixing the Pipeline

The financial cost of the overhaul was substantial but, by the lab's estimates, far less than the cost of the original crisis. Documenting existing code for 14 papers required roughly 2,100 person-hours—equivalent to a full-time employee for a year. The lab hired a part-time software engineer at a cost of roughly $50,000 per year to maintain the containerized workflows and train new members. Cloud storage for archived runs, including compute environments and log files, added about $200 per month. These expenses were initially covered by a supplementary grant from the university's research office, which recognized the institutional risk of unreproducible science.

But the lab also calculated the avoided costs. Each retracted paper had required weeks of correspondence with journals, a formal investigation by the university, and damage to the lab's reputation. The PI estimated that the time spent on retractions alone—writing corrections, responding to media inquiries, and managing the fallout—exceeded the time invested in building the new pipeline. In terms of career impact, the lab lost two grant renewals that cited reproducibility concerns, but it gained credibility in the emerging field of open science. The net financial savings, conservatively estimated, were roughly ten times the cost of the fixes per paper.

Not everyone agrees that such intensive measures are necessary. Some critics argue that the lab's experience is an outlier and that most errors are benign or caught before publication. They point to the high cost of containerization and pre-registration as barriers that could slow scientific progress. The debate reflects a deeper tension: reproducibility is valuable, but it competes with other priorities like speed and innovation. The lab's PI acknowledged this trade-off, noting that the new workflow added roughly 20% to the time required for each analysis. For labs with limited resources, the question of how much reproducibility is enough remains unresolved. Moreover, the benefits of containerization may be overstated if the underlying data are poorly documented or if the code itself is not well-commented. Even with perfect archiving, conceptual errors in the experimental design or statistical reasoning can still lead to irreproducible findings. The challenge is not only technical but also cultural and educational.

Toward a Culture of Code Stewardship

The lab's story is not just about fixing a broken pipeline; it is about changing the culture of how code is treated in science. Small, low-cost interventions can have outsized effects. A mandatory README file describing the purpose and structure of each script, a license that clarifies reuse permissions, and a list of dependencies with version numbers—these simple practices, widely used in software engineering, are still rare in academic labs. The lab now requires all code submissions to include these elements, and it has published a template that other labs can adopt.

Graduate curricula are beginning to catch up. Several universities now offer courses in computational reproducibility, teaching version control, containerization, and testing. The lab's PI has argued that such training should be as fundamental as statistics for any researcher who writes code. Some journals have moved toward executable papers, where the analysis code runs inside the article, allowing readers to tweak parameters and see results change in real time. These formats are still experimental, but they represent a future in which code is no longer a black box.

The incident has also spurred community standards. A consortium of neuroscience labs, including the one at the center of this story, has proposed a code review checklist modeled on the CONSORT checklist for clinical trials. The checklist covers basic practices: is the code version-controlled? Are dependencies specified? Can the analysis be reproduced from scratch? The hope is that such checklists, adopted by journals and funders, will make code stewardship a routine part of the research process.

Yet challenges remain. The adoption of these practices is uneven across disciplines and institutions. Smaller labs with fewer resources may struggle to implement containerization or dedicate time to code documentation. The pressure to publish quickly often overrides the best intentions for reproducibility. And even when code is archived, it may not be sufficient if the software environment is not preserved (e.g., specific operating system versions or hardware dependencies). The lab at the University of Bern continues to refine its workflow, but it acknowledges that perfection is elusive. The goal is not to eliminate all errors—that is impossible—but to reduce the rate of undetected errors and to make the process transparent. One lab's mistake, costly as it was, has become a template for reform—a reminder that the integrity of science depends not just on the data we collect but on the code we write to interpret it. The path forward requires sustained investment in training, infrastructure, and cultural change, with the understanding that reproducibility is a process, not a destination.