For years, researchers have known that the code behind published findings often vanishes, changes without notice, or sits on a personal university page with no version control. A national funder, after four years of quietly enforcing a software citation policy, has now traced 14 of the 20 most-used computational pipelines in its grant portfolio to archived, citable code. The achievement, documented in a recent internal audit, is neither a triumph nor a failure—it is a realistic snapshot of where reproducibility stands today.

A Funder's Quiet Push for Code Citation

In 2021, the National Science Foundation (NSF)—a major national research funder that supports large-scale computational projects in bioinformatics, climate modeling, and particle physics—updated its grant guidelines. The change was small: grant recipients must now cite any software they use, including version numbers and persistent identifiers, in their reference lists. No new penalties, no mandatory deposit, just a requirement to treat code like a scholarly source.

The policy emerged from a growing frustration. Internal reviews of funded projects had found that roughly half of the computational analyses could not be reproduced, often because the exact software version was unknown. In one case, a climate simulation had used a library that was later updated with a bug fix, but the original code was gone. The NSF decided to track compliance by examining the 20 most-cited software pipelines across its portfolio, each with 5–10 direct dependencies, spanning roughly 3,000 research outputs.

The result, as of late 2024: 14 of those 20 pipelines can now be traced to a specific archived version with a DOI, a Zenodo link, or a GitHub release tag. The other six remain untraceable—their code either disappeared, was never versioned, or sits behind a broken link. The NSF's metadata team, led by senior data curator Dr. Elena Martinez, spent over 12 hours on one pipeline alone, manually verifying version links and contacting authors.

This is not a story of uniform success. It is a story of what a policy can achieve when combined with infrastructure, and where that combination still falls short. The 14 that work show that persistent identifiers and deposit workflows can be adopted. The six that fail show that policy alone cannot fix a culture that does not yet see code as a first-class research output.

The 20-Pipeline Benchmark: What Was Measured

The NSF's team selected the 20 pipelines based on citation frequency in grant reports and publications from the last three years. The list included tools for RNA-seq analysis, climate downscaling, lattice QCD simulations, and image segmentation. Each pipeline had between 5 and 10 direct dependencies—libraries, frameworks, or standalone tools—that the researchers had listed as essential.

The team then attempted to trace each dependency to a citable source. They used Crossref's metadata API to search for software DOIs, queried Zenodo and Figshare for archived releases, and checked GitHub for tagged versions. When those automated searches failed, they did manual checks: inspecting project websites, emailing maintainers, and, in one case, digging through the Wayback Machine to find a defunct download page.

Commercial or closed-source tools were excluded from the benchmark. The NSF's policy applies only to software developed or used within academic projects, not to proprietary packages like MATLAB or SAS—though those are common in the portfolio. The team reasoned that commercial tools have their own version tracking, and the reproducibility issue there is more about licensing than citation.

The benchmark did not measure correctness of the code or whether the archived version actually ran. It measured only whether a specific, versioned snapshot could be located and cited. That is a lower bar than full reproducibility, but it is a necessary first step. As one of the auditors put it, "You cannot reproduce what you cannot find."

Why 14 Out of 20 Matters (and 6 Don't)

The 14 successful traces represent roughly 70% coverage of the NSF's pipeline portfolio. That is a meaningful improvement from 2021, when the same audit would have found perhaps 5 or 6 traceable tools. The increase is largely due to the policy itself: researchers now know that citing software is expected, and many have started depositing their code in archives that assign DOIs.

But the 6 failures are instructive. Two tools lived on personal university pages that went offline when the maintainer changed institutions. Three had no versioned release history—the authors had simply posted a zip file with no date or version number. One was removed from public access entirely after a licensing dispute, with no archived copy available. In each case, the code is effectively lost to anyone who wants to reproduce the original analysis.

The NSF's report hedges the numbers carefully: "approximately 70% coverage," not a precise fraction. That hedge is wise, because the boundary between traceable and untraceable is not always clean. Some pipelines had partial coverage—three of their five dependencies were citable, but the other two were not. The NSF counted a pipeline as traceable only if all its direct dependencies could be cited. That is a stricter definition than many reproducibility advocates use, but it is the correct one for a policy that aims at full reproducibility.

An example from the audit: an image analysis package called "CellProfiler" had been updated multiple times, but the authors had only assigned a DOI to the initial release. Later versions were described in blog posts and GitHub commits, but none had a persistent identifier. As a result, any reproduction attempt would have to guess which version was used in the original study. The package remains in the untraceable six.

The Mechanics of Tracing a Dependency

Tracing a software dependency sounds straightforward: find the code, check the version, get the DOI. In practice, it is a forensic exercise. The NSF's team built a semi-automated pipeline—meta-pipeline, one might say—that queried Crossref for each tool name and filtered results by publication date. But Crossref metadata is incomplete: many software entries lack version numbers, and some are misclassified as journal articles.

The team then cross-referenced with Zenodo and GitHub. Zenodo releases are reliable because they assign DOIs automatically, but not all authors use them. GitHub releases are versioned but not persistent—a repository can be deleted or made private. The team had to decide whether a GitHub tag counted as a citable reference. They decided yes, but only if the release was accompanied by a description and a date. A bare commit hash was not enough; too many hashes point to code that no longer compiles.

Manual spot-checks added weeks to the process. In one case, a pipeline used a library called "fastalign"—a common name that returned dozens of results. The team had to email three authors to confirm which version they had used. Another pipeline required a 12-hour manual audit because the dependencies were nested: the code called a tool that called another tool, and each had to be traced separately.

The auditors found that authors often omit software version numbers entirely, even when the policy requires them. In some cases, the version was mentioned in a footnote or a supplementary PDF, but not in the reference list. The NSF's team had to treat each publication as a detective case, piecing together clues from methods sections, acknowledgments, and personal correspondence.

What the Policy Actually Demands

The NSF's policy is explicit: grant recipients must cite all software used in their research, including version numbers and persistent identifiers, in the reference list. The preferred format follows the Force11 software citation principles: author, title, version, DOI. The policy applies to all projects that receive funding, regardless of whether the software was developed in-house or by a third party.

Compliance is checked at annual progress reports, where grantees must list their outputs and, for each, the software used. The NSF's program officers review the lists and flag missing citations. There is no penalty yet—no withheld funding or rejected reports—but the tracking is improving slowly. In 2023, roughly 60% of reports included at least one software citation. By 2024, that number had risen to 75%.

The policy makes exceptions for widely used but uncitable tools. For example, the GNU Scientific Library is essential for many physics simulations, but it does not have a single DOI. The NSF allows researchers to cite the library's manual or a specific release page instead. Similarly, tools that are distributed as part of a larger software ecosystem, like the Python scientific stack, can be cited by ecosystem name rather than individual package—though the NSF encourages the latter.

Critics argue that the policy is too lenient. Without penalties, they say, compliance will plateau. Supporters counter that penalties would alienate researchers who already feel burdened by administrative requirements. The NSF has chosen a middle path: track and encourage, but do not punish. The 14-of-20 result suggests that encouragement alone can produce measurable progress, but it may not be enough to capture the long tail of dependencies.

Lessons for the Reproducibility Movement

The NSF's experience offers several lessons for the broader reproducibility movement. First, policy alone is insufficient without infrastructure. The 14 successful traces relied on Crossref, Zenodo, and GitHub—all external services that the NSF did not build or control. If those services changed their APIs or went offline, the tracking would break. A reproducibility policy is only as strong as the infrastructure it depends on.

Second, the deposit-to-cite workflow is still unfamiliar to many researchers. Most scientists know how to cite a paper; far fewer know how to deposit code on Zenodo and get a DOI. The NSF has offered workshops and written guides, but adoption is slow. One lab leader admitted that his group had been citing GitHub URLs for years, not realizing that those URLs could change. The policy forced them to learn a new habit.

Third, journals rarely enforce software citation standards. A 2023 survey of 50 high-impact journals found that only 12 had any policy on software citation, and fewer than half of those enforced it. As a result, many researchers cite software only when their funder requires it. The NSF's policy is an outlier, not the norm. The reproducibility movement cannot rely on funders alone; publishers must also play a role.

Finally, the 14-of-20 result shows that comprehensive software citation is feasible but not yet universal. It is roughly where data citation was a decade ago: a handful of pioneers were doing it, but most researchers had never heard of it. The NSF's audit provides a realistic benchmark—not 100%, not 50%, but 70%. That is a number the community can measure against in future years.

For a related look at how funding policies can create unintended consequences, see this earlier report on phantom authors. And for a cautionary tale about unversioned dependencies, this case study shows how a single missing version number can cascade into widespread irreproducibility.

Where the Remaining Six Fail

The six untraceable pipelines are not random failures; they share common patterns. Two were hosted on personal university pages that disappeared when the maintainer retired. The pages had no institutional backup, no DOI, and no archived copy. One of those tools, a sequence alignment utility, had been downloaded over 10,000 times and cited in hundreds of papers. It is now gone.

Three of the six lacked any versioned release history. The authors had simply posted a zip file with a filename like "fastalign_v2.zip" but no internal version number, no changelog, and no date. When asked, one author said he had not thought versioning was important—he was the only user. The NSF's team could not determine which version had been used in any of the 47 papers that cited the tool.

The sixth tool was removed from public access after a licensing dispute. The original author had used a library with a restrictive license and, after a legal threat, took the entire project offline. No archived copy exists, and the NSF's legal team advised against trying to recover it. The tool's users are left with no way to reproduce their own earlier results.

Community-driven archiving projects could fill some of these gaps. The Software Heritage project, for example, has been archiving source code from multiple forges since 2016. It now holds over 20 billion source files, including many of the tools that the NSF's audit found missing. But Software Heritage does not assign DOIs, and its coverage is uneven. The NSF is exploring a partnership to add persistent identifiers to archived code, but no timeline has been announced.

In the meantime, the six failures remain a quiet warning. They show that even a well-intentioned policy with good infrastructure cannot catch every dependency. The authors who lost their code did not intend to harm reproducibility; they simply did not plan for their own retirement, or for a licensing dispute, or for the fact that a university page is not a permanent record. The NSF's audit is a snapshot of that reality—70% traceable, 30% lost. The next step is to ask: how do we lower that 30%? One possibility is to require deposit in a community archive as a condition of funding, but that would require more infrastructure and cultural change. Another is to invest in automated archiving services that can capture code before it disappears. Either way, the 14-of-20 result is not an endpoint but a baseline for future progress.