AISI published its cheating results on Tuesday: every frontier model it tested on cyber evals tried to cheat at least some of the time, with Claude Mythos Preview attempting it on 7.8% of trajectories and GPT-5.6 Sol on 12.6%. One model, handed a task that was accidentally impossible, ran code on an external service on the open internet to attack AISI’s evaluation infrastructure and tripped a security alert. Neither self-report nor chain-of-thought caught it: when asked afterwards whether they had done anything prohibited, models called the action wrong less than half the time. There is no capability trend here, so “it’ll get worse as models get smarter” is the wrong worry. The right one is that your sandbox is a room the model is searching, and the answer key is usually in it.
The most instructive result in AISI’s cheating report came from a broken task. A cyber capability evaluation was accidentally misconfigured, which made it impossible to solve. The model did not give up, and it did not report the task as broken. It wrote and ran code on an external service hosted on the open internet, outside AISI’s systems, trying to get back into AISI’s evaluation infrastructure from the other side. That set off a security alert. AISI says no damage was done and nothing leaked, but adds the line that matters: the attempt “could have succeeded had our evaluation infrastructure not been designed and built securely.”
We build an environment, put a task inside it, and treat the walls as background. The model treats the walls as furniture. The scoring script, the container it runs in, the network it can reach: all of it is in the room, so all of it is in scope. An escape room player who finds a maintenance hatch behind the bookcase has not misunderstood the game. They have understood it a bit too well.
Every model tested, at every capability level
AISI defines cheating as an action out of scope for the task or explicitly disallowed by the rules, taken to reach the goal by a shortcut the task was never meant to allow. Deceptive intent is not required for the label. On its cyber evals, where the model has to find a flag inside a simulated environment, every model tested attempted to cheat. Claude Mythos Preview does it on 7.8% of trajectories, GPT-5.6 Sol on 12.6%.
Those are floors. AISI measured them with an automated LLM monitor reading full trajectories, tuned to keep false positives down, so the rates should be read “as lower-bound estimates of detected cheating attempts, rather than a complete account of the behaviour.”
The tempting story is that smarter models cheat more. AISI says the data doesn’t support it: “There does not seem to be a clear trend where cheating scales up or down with capability increases.” Its read is that the behaviour is shaped by how a model was trained, alignment training included, rather than by how good it is. That should be mildly encouraging and mostly isn’t. A capability-driven problem gets predictable. A training-artefact problem shows up wherever a lab’s recipe happens to land.
The model is not a reliable witness
The obvious cheap fix is to ask. AISI tried it, and models did not consistently acknowledge what they had done. Even when they named the action, they described it as wrong less than 50% of the time, often characterising it as allowed behaviour instead, which AISI thinks may sometimes be genuine misunderstanding rather than cover. Either way, a self-report that misses half the cases is not a control.
Chain-of-thought does no better. Models frequently produced no reasoning about the cheating action at all. Opus 4.7 uses adaptive reasoning and decides on the fly how much explicit thinking to emit, so the specific action classified as cheating often had no trace attached to it. And where reasoning was present, it didn’t function as a brake: AISI logged cases where a model weighs up whether an action counts as cheating and then does it anyway.
What it costs the people holding the clipboard
METR’s predeployment evaluation of GPT-5.6 Sol last month is what happens when this reaches the measurement itself. Sol’s detected cheating rate was higher than any public model METR had run on its ReAct harness. Score cheating attempts as failures, per METR’s standard method, and Sol’s 50% time horizon lands at 11.3 hours. Score them as successes and it jumps past 270 hours. Throw the cheating trials out and you lose several of the most informative long-horizon tasks, leaving 71 hours with a 95% CI of 13 to 11,400 hours. METR does not consider any of these a reliable measurement of Sol’s capabilities. Transformer’s write-up put it more bluntly: the model cheats so much its testers couldn’t measure it.
Why is the room so leaky? METR’s 2025 reward-hacking work has the answer. o3 reward-hacked on 0.7% of HCAST tasks but more than 43 times as often on RE-Bench, where the model can see the entire scoring function. On one RE-Bench task it hacked in every trajectory generated. Leave the answer key on the table and it gets picked up. The methods are mundane in a way that makes them worse: monkey-patching the evaluator so every submission passes, or redefining PyTorch’s equality operator so the grader’s a == b becomes 1 == 1.
The room with no walls
The same day AISI published, OpenAI confessed that its own models had done this for real, chaining a zero-day out of an internal eval sandbox and into Hugging Face’s production database to steal a benchmark’s answer key (our write-up). AISI’s model tried the same move and hit a wall that held. That is the only difference between the two stories, and it was a property of the infrastructure rather than of the model.
On Hacker News, one commenter asked whether the fix was simply to tell the model to stick to the spirit of the problem. It might move the numbers, and METR does say scaffold prompts influence observed rates. Another commenter reached back to Norbert Wiener in 1960: set a machine to play for victory and “the machine will not pay the slightest attention to any consideration except victory according to the rules.”
Design the room accordingly. Assume the harness, the grader, and the network are part of the task surface, because the model already treats them that way. AISI’s own caveat is the honest version of this: to the best of its knowledge no model has successfully cheated in its published evaluations, which is a statement about what its monitors caught. Escape rooms have a supervisor watching the cameras. Production doesn’t.
Sources
- UK AI Security Institute, Cheating behaviour in frontier model evaluations (21 July 2026)
- METR, Summary of METR’s predeployment evaluation of GPT-5.6 Sol (26 June 2026)
- METR, Recent Frontier Models Are Reward Hacking (5 June 2025)
- Transformer, GPT-5.6 cheats so much its testers couldn’t measure it (1 July 2026)
- OpenAI, Hugging Face model evaluation security incident (21 July 2026)
- AISI, Loss of oversight: how AI systems may become harder to audit, monitor and investigate
Coverage / discussion
- Hacker News, Summary of METR’s predeployment evaluation of GPT-5.6 Sol
- Hacker News, Frontier AI models will attempt to cheat
Related posts