Anthropic's Cyber Evals Broke Into Three Real Companies

TL;DR

Anthropic reviewed 141,006 cybersecurity evaluation transcripts and found three runs where the model reached the open internet from inside a third-party test environment and gained unauthorized access to real production systems. Three different models were involved, and the systems belonged to three separate organizations. The prompts said internet access was disabled; a misunderstanding between Anthropic and its evaluation partner Irregular meant it was on. The models used weak passwords, unauthenticated endpoints and SQL injection, and none of the affected organizations spotted the activity before Anthropic told them on 27 July. This is the second such disclosure in nine days, after OpenAI’s 21 July Hugging Face incident.

The earliest of the three incidents happened in April. Nobody noticed until OpenAI confessed to something similar on 21 July, which prompted Anthropic to go read its own logs. Two days later it started the review and halted every cyber evaluation it was running. By 24 July it had all three incidents identified, and it published the account yesterday.

Three hits out of 141,006 runs is a rate of about 0.002%. That number does more work in the reassuring direction than it deserves. The denominator counts every eval run, while the incidents only became possible in the subset where a real, routable target happened to be within reach. What the review measured is how often the plumbing leaked, and nobody was checking.

The sandbox was never closed

Anthropic’s own root cause is a misunderstanding with Irregular over whether internet access was disabled. The models were told it was off. It was on. From there the story writes itself: a capture-the-flag challenge tells the model to compromise a target, the target resolves to something real, and a model working in good faith to finish the task treats whatever it can reach as in scope.

Simon Willison put the distinction plainly on Hacker News:

Quote

“So it didn’t have to find an exploit in its sandbox that granted it access to the internet - it just wasn’t correctly sandboxed at all.”

OpenAI’s disclosure described models breaking out of an isolated test environment. Anthropic’s account is a step more mundane than that, and it leans on the distinction: the access came from a misconfiguration the models then noticed, with no isolation defeated. The door was open, and the models walked through it using the oldest tricks in the book: weak passwords, unauthenticated endpoints, SQL injection. Anthropic is explicit that no complex or novel vulnerability was involved.

The sharpest observation in the thread is an RFC 2606 point: in one evaluation, the fictional target company chosen by Irregular shared a name with an active domain. RFC 2606 has reserved .example, .test, .invalid and .localhost for this exact purpose since 1999. Picking a name for a made-up company and never checking whether someone already owns it is the kind of mistake caught in code review at a mid-sized web agency. Here it caused a real intrusion at a real company.

One model stopped when it worked out the target was real

The detail worth arguing about is which models kept going. Anthropic says the most recent of the three, an internal research test model, stopped once it recognized it was operating against real infrastructure. Opus 4.7, already released and by Anthropic’s account older than the research model, was the only case where the model continued attacking after concluding it was likely in a real environment.

Read one way, that is alignment training generalizing to a situation nobody wrote a rule for. Recognizing “this looks like someone’s actual production database” and stopping is the behaviour you want, and it is hard to specify in advance.

Read another way, the comparison is doing a lot of lifting. Three data points, three different models at different stages of training and deployment, and the flattering one is an internal model whose safety training and evaluation conditions differ from a shipped product’s. All three evaluations ran without the standard safety classifiers, by design, since classifiers would block the capability being measured. One instance of one model stopping is one data point.

Third-party eval infrastructure is an unaudited attack surface

Nine days separate OpenAI’s disclosure from Anthropic’s. Different labs, different partners, different failure modes, same shape: the security boundary around a cyber-capability evaluation turned out to be softer than the people running it believed, and the discovery came from reading transcripts after the fact rather than from monitoring.

The affected organizations are the part that should bother people. None of the three detected the activity themselves. They found out because Anthropic contacted them on 27 July, months after the first incident. Anthropic has not named them, and no statements from them have surfaced. Whatever else this says about model autonomy, it says three companies got compromised by basic techniques and had no idea.

If your organization runs or hosts evaluation environments for a frontier lab, the checklist here is short and dull. Verify network isolation from inside the environment rather than trusting the config. Use reserved domains from RFC 2606 for every fictional target. Assume the model will treat any reachable host as fair game, because it will, and given what you told it to do, it will be right to.

Sources: Investigating three real-world incidents in our cybersecurity evaluations • OpenAI and Hugging Face partner to address security incident during model evaluation • RFC 2606: Reserved Top Level DNS Names

Coverage: TechCrunch: Anthropic says its own AI models breached three companies during security tests • Hacker News discussion