--- title: "OpenAI Tripled Its ARC-AGI-3 Score by Fixing Its Own Plumbing" pubDatetime: 2026-07-30T01:04:00.000Z description: "Retained reasoning and compaction took GPT-5.6 Sol from 13.3% to 38.3% on the ARC-AGI-3 public set with 6x fewer output tokens. The engineering lesson is solid. The 38.3% is not comparable to the 30.2% Opus 5 posted five days earlier." tags: [openai, benchmarks, evaluations, arc-agi, context-engineering, ai-agents, anthropic, 2026, 2026-q3, 2026-07] --- > [!tldr] TL;DR > OpenAI [published a post yesterday](https://openai.com/index/how-two-settings-tripled-our-arc-agi-3-scores/) showing that two Responses API settings, retained reasoning and compaction, took GPT-5.6 Sol from 13.3% to 38.3% on the ARC-AGI-3 public set while cutting output tokens 6x. The diagnosis is genuinely good: ARC's harness threw away private reasoning after every game action and rolling-truncated old moves, so the model re-derived each game from scratch every turn. Take the engineering advice. Do not read 38.3% as passing the [ARC-verified 30.2%](https://arcprize.org/results/anthropic-claude-opus-5) Claude Opus 5 posted five days before, because that number is a different task set, measured by a different party, on a harness the other model didn't get. Community harnesses already hit 97%+ on the same public set, which is exactly why lab-run harness scores can't be the scoreboard. The post [went up](https://openai.com/index/how-two-settings-tripled-our-arc-agi-3-scores/) on 29 July, by Ilan Bigio and Ted Sanders. Last Friday, ARC Prize published Claude Opus 5 at [30.2% on ARC-AGI-3](https://arcprize.org/results/anthropic-claude-opus-5), roughly four times GPT-5.6 Sol's 7.8%, which the-decoder [wrote up](https://the-decoder.com/anthropics-opus-5-blows-past-fable-5-and-gpt-5-6-sol-on-the-benchmark-designed-to-measure-real-intelligence/) as blowing past the field on "the benchmark designed to measure real intelligence." Five days later OpenAI has a 38.3%. Both things are true: the technical finding stands on its own, and the timing is no accident. ## What they actually found [ARC-AGI-3](https://arcprize.org/arc-agi/3) is 135 hand-built 2D games, 25 of them public. Nobody tells the agent the rules. It has to poke at the environment, form a theory, and carry what it learned into the next level. ARC deliberately runs a generic harness with no tools and no per-model tuning, on the reasoning that a plain harness makes model weaknesses visible and keeps comparisons fair. OpenAI looked at what that harness does between actions and found two things. All private reasoning was discarded after every move, so the model saw a list of past actions but none of the thinking that produced them. And context was managed by rolling truncation: past 175,000 characters, oldest messages fall off the front. Amnesia about its own reasoning, and a slowly eroding action log. Rebuilt on the [Responses API](https://developers.openai.com/blog/responses-api), passing `previous_response_id` to retain reasoning and swapping truncation for [compaction](https://developers.openai.com/api/docs/guides/compaction), Sol went from 13.3% to 38.3% on the public set. That 13.3% is [ARC's own published figure](https://arcprize.org/results/openai-gpt-5-6-sol) for Sol at max effort, so the baseline is no straw man. On the game `cd82`, where no frontier model on the leaderboard clears level one, their harness solves all six levels. The mechanism is the interesting bit. With its reasoning retained, the model thought *less* per action, because it no longer had to reconstruct the game every turn. Fewer tokens, better play. ARC's metric, [RHAE](https://docs.arcprize.org/methodology), measures action efficiency against a human baseline, so a harness that stops the model flailing scores higher almost by construction. OpenAI estimates the average human tester at 48% from ARC's [public gameplay logs](https://huggingface.co/datasets/magic-sword/arc_agi_3_public_demo_human_testing). ## The number is real. The comparison is not. 38.3% is the public demo set, run by OpenAI, on a harness OpenAI wrote. 30.2% is the semi-private set, run by ARC, on the harness every model gets. Those quantities do not go in the same column. In fairness to OpenAI, the set difference cuts their way. ARC's [testing policy](https://arcprize.org/policy) states that for ARC-AGI-3 "the public demo is harder than the Semi-Private set." OpenAI measured on the harder half. What they skipped is the thing that would settle it: ARC runs a [Community Leaderboard](https://arcprize.org/leaderboard/community) with a verification path, one-click Kaggle notebook, semi-private evaluation, open-sourced solution, and up to $2,500 reimbursed. Greg Kamradt, ARC Prize's president, applied precisely this caveat three days ago to a much bigger claim. Praising Alexis Fox's harness as [the best he knows of](https://x.com/GregKamradt/status/2081773134651576321) on ARC-AGI-3, at a stated 97% for $1,750 with Fable 5, he added the asterisk himself: "I've only seen their public demo scores, we haven't tested it on semi private yet." As of this morning ARC has said nothing publicly about OpenAI's post. ## Harness scores on the public set already go to 99% Two weeks ago the [Schema harness](https://schema-harness.github.io/) claimed ~99% on the public set with Opus 4.8 and Fable 5, and 95.35% with GPT-5.6 Sol. The [HN thread](https://news.ycombinator.com/item?id=48935905) hit 131 points and spent most of them on why that doesn't count: the harness has the model write its own simulator of the game, then plan against the simulator, which routes around the thing being measured. As `modeless` put it, the public set is known in advance, "so it's possible to hardcode rules that make this easy for the models." That skepticism is an established norm: back on ARC-AGI-3's launch day a commenter dismissed Symbolica's entry because [the harness was benchmark-specific](https://news.ycombinator.com/item?id=47538078), "This isn't a qualifying score for that reason." So a lab-run public-set number sits on a curve that already runs to 99%, and its position on that curve is a fact about the harness. OpenAI's 38.3% is a claim about API defaults, which is what the post says, and it deserves to be read that narrowly. Their own HN submission drew 8 points and zero comments, which suggests it was. ## The claim worth arguing with Buried in this is a real critique of ARC's design that OpenAI has standing to make. ARC's [1 May analysis](https://arcprize.org/blog/arc-agi-3-gpt-5-5-opus-4-7-analysis) of GPT-5.5 and Opus 4.7 concluded that "Opus compressed its observations into a confident-but-wrong theory. GPT-5.5 had difficulty compressing at all." Failure to compress, offered as a fact about the model. Now consider a harness that deletes the model's reasoning after every action. Of course it fails to compress; the place where compression would live is emptied hundreds of times a run. Both teams are describing the same symptom at different layers, and OpenAI's layer is upstream. That's the strongest point in the post, and they underplay it. It also doesn't rescue the model. Fixing the plumbing gets Sol to 38.3% against an estimated 48% human baseline, with everything ARC found about false world models and Tetris-shaped hallucinations still on the table. ## What to do with this The advice transfers and it's cheap: retain reasoning across tool calls, compact rather than truncate. The 6x output-token cut is the part I'd have led with. ARC's leaderboard plots score against cost, and Sol's ARC-AGI-3 run [cost $25.1K](https://arcprize.org/leaderboard) against Opus 5's $20.7K, so six times fewer output tokens moves you left and up at once. Two caveats. Retained reasoning via `previous_response_id` is server-side state on one vendor's API, so the recommendation to standardise on it is also a recommendation to standardise on OpenAI. And if every lab evaluates on its own optimal harness, cross-model comparison stops existing, which is the entire reason ARC runs a boring one. ARC's policy says it plainly: their evaluations are "not a development tool," and providers "should not expect continuous optimization cycles." The clean resolution is available and cheap. Open-source the harness, submit it, let ARC run it on semi-private, and put the same harness under Opus 5. Until then, 38.3% is a useful engineering result wearing a leaderboard's clothes. --- *Sources: [How enabling two settings tripled our scores on the ARC-AGI-3 benchmark (OpenAI, 29 July 2026)](https://openai.com/index/how-two-settings-tripled-our-arc-agi-3-scores/) · [Claude Opus 5 ARC-AGI results](https://arcprize.org/results/anthropic-claude-opus-5) · [GPT-5.6 Sol ARC-AGI results](https://arcprize.org/results/openai-gpt-5-6-sol) · [ARC-AGI-3 overview](https://arcprize.org/arc-agi/3) · [ARC Prize official testing policy](https://arcprize.org/policy) · [Analyzing GPT-5.5 & Opus 4.7 with ARC-AGI-3 (Greg Kamradt, 1 May 2026)](https://arcprize.org/blog/arc-agi-3-gpt-5-5-opus-4-7-analysis) · [ARC-AGI-3 leaderboard](https://arcprize.org/leaderboard) · [RHAE methodology](https://docs.arcprize.org/methodology) · [Compaction guide](https://developers.openai.com/api/docs/guides/compaction) · [Kamradt on the PRO-LONG harness](https://x.com/GregKamradt/status/2081773134651576321)* *Coverage: [Anthropic's Opus 5 blows past Fable 5 and GPT-5.6 Sol (the-decoder)](https://the-decoder.com/anthropics-opus-5-blows-past-fable-5-and-gpt-5-6-sol-on-the-benchmark-designed-to-measure-real-intelligence/) · [HN: Schema Harness achieves ~99% on ARC-AGI-3 Public](https://news.ycombinator.com/item?id=48935905) · [HN: Day 1 of ARC-AGI-3](https://news.ycombinator.com/item?id=47538078) · [HN: OpenAI's post](https://news.ycombinator.com/item?id=49104184) · [r/mlscaling on Sol's 7.78%](https://www.reddit.com/r/mlscaling/comments/1us6kas/gpt56_sol_scores_778_on_arcagi3/)* *Related on this blog: [Agent Harnesses: A Standard for a Word Nobody Agrees On](/posts/agent-harnesses-standard) · [Opus 5 Shipped and Hacker News Argued About a Computer Vision Pipeline](/posts/claude-opus-5-reactions-vision-pipeline) · [The Ralph Wiggum Technique](/posts/ralph-wiggum-technique)*