Coding Agents Rebuilt Two Bio-AI Tools From the Paper for $4 to $20

TL;DR

UK AISI and SecureBio ran ReproBAIT, giving four coding agents a published paper and an internet-free sandbox and asking them to rebuild the tool it described. On the two simpler targets, a closed-source molecular design tool and ProteinMPNN, agents matched or beat the originals for $4 to $20 per successful run, with Claude Opus 5 hitting 96.5% sequence-recovery fidelity on ProteinMPNN. No agent rebuilt RFdiffusion3 inside the budget. The researchers conclude that withholding code does not by itself assure control over a tool’s capabilities, and point to training-data governance as the stronger lever. The large caveat: agents were handed the original training data, which a real attacker working around managed access would not have.

Handing the agents the entire paper instead of just the abstract made no statistically significant difference to their success rate, the paper reports. That is the result worth sitting with. If text detail were the bottleneck, the full methods section should have helped, and it did not. What the agents needed was data and compute.

The evaluation is called ReproBAIT, run by AISI with SecureBio, and it went up on arXiv 12 days ago. Four agents were tested: Claude Opus 5, GPT-5.6 Sol, Kimi K3 and Qwen3.8-2.4T. Three biological AI tools of rising complexity were the targets: a closed-source molecular design tool the authors call CMD, ProteinMPNN, and RFdiffusion3. Each agent worked in a sandbox with no internet, on a single GPU, with a 12-hour budget.

What the agents actually produced

On CMD and ProteinMPNN, the reimplementations matched or exceeded the published performance of the originals. Opus 5 reached 96.5% sequence-recovery fidelity against ProteinMPNN. The cost of a successful run landed between $4 and $20.

RFdiffusion3 held. No agent rebuilt it within the time and compute allowed, and the paper notes that larger rebuilds would cost “considerably more.” Whether that is a durable complexity moat or a snapshot of this generation of agents on a single GPU is the open question, and it matters more than the two wins: RFdiffusion3 is the most capable of the three targets, which makes it the one managed access is most plausibly protecting.

SecureBio’s companion post is blunt about the limit of the finding, saying the results “do not show that agents can recreate entire BAITs.” Partial reconstruction of a core capability is what was measured.

The memorization problem

Two of the three targets are open source, so their code and weights plausibly sat in the agents’ pretraining data. The authors ran memorization checks and found evidence of it in roughly 28% of attempts, at a level they describe as “suggestive.” For ProteinMPNN the evidence was moderate; for RFdiffusion3, little.

That muddies the headline. A reimplementation built partly from recall of the public repo is a weaker demonstration than one built from the paper alone, and the strongest memorization signal sits on the tool with the cleanest success. The signal is weakest on the tool nobody managed to rebuild, which at least means recall was not what was blocking RFdiffusion3.

Where the $4 figure overstates the case

Superpower Daily’s summary collects the methodological objections, and they are real ones. Agents were given each tool’s original training data. Someone trying to route around a managed-access regime would be missing exactly that. Writing the model code is the cheap half of the job; assembling an equivalent corpus is where the money goes. The reported scores are best-of-three runs rather than averages, which inflates apparent reliability. The targets are small next to the largest released biological AI tools.

Their sharpest line is that the question in the title remains unanswered: whether withholding code meaningfully limits replication is still untested. ReproBAIT tested whether a paper plus data plus a GPU is enough. It did not test the counterfactual an access policy is actually built around.

Code secrecy, data governance, and the case against both

AISI and SecureBio land on a reasonable conclusion: keeping a tool’s code private does not, on its own, give its developers control over the capability, and data governance looks like the more effective place to apply pressure. The arXiv paper puts it as a recommendation that developers weigh publication risk against managed-access strategies rather than treating the latter as sufficient.

The step from there to “so govern the data instead” deserves more resistance than it usually gets. A 2025 report from the Engineering Biology Research Consortium, Risk Mitigation for Biological Design Tools, argues that managed access is not an obviously correct mitigation for these tools even before you ask whether agents can bypass it. Without a clear threat model, it is unclear what should be gated, who runs the gate, and who gets through it. The controls cost money and fall on legitimate developers first. Those objections transfer intact to data gating, which is a heavier lift than code gating and concentrates access among whoever already holds the corpora.

This is the second AISI result this year where the interesting number is a price rather than a capability score. Its open-weight cyber gap analysis in July put the capability lag at four to seven months and the cost per reliably solved task at $12.50 for Opus 4.5 against $0.28 for DeepSeek V4-Pro. ReproBAIT finds a reimplementation cost in the tens of dollars. Capability scores tell you whether a thing is possible; the price tells you how many attempts an attacker can afford. AISI’s eight technical priorities for frontier labs, set out on 12 October, already argued that evaluation methods are falling behind capability growth. ReproBAIT reads like evidence for the narrower claim that access controls are being outrun too, at least for the tools simple enough to rebuild in 12 hours.

Sources: AISI: Can AI coding agents rebuild biological AI tools from a paper? • ReproBAIT (arXiv:2610.19810) • SecureBio: ReproBAIT benchmark • EBRC: Risk Mitigation for Biological Design Tools

Coverage: Superpower Daily: Coding agents rebuilt parts of biological AI tools for $4–$20