Anthropic ran a randomized trial in which 50 professional analysts used Claude Code to work through four analytical decisions for a fictional farming cooperative. A hidden coin flip set which option each analyst’s agent would argue for. The analysts went with the assigned lean 86% of the time, and when the agent contradicted a prior they had already stated, they switched 60% of the time. Afterwards, 52% rated the final recommendation as more theirs than the agent’s and 77% reused the agent’s arguments without attributing them. The agent was instructed to argue honestly, so some of that 86% is persuasion working as intended, but anyone whose review process ends with “a human checked the agent’s work” should read the paper.
The agent’s position came from a coin flip. On each of four decisions in the exercise, Anthropic’s researchers flipped to pick one of the two defensible options, then injected a hidden system prompt telling the analyst’s Claude Code instance to argue for it. The analysts, screened for causal-inference expertise, had no idea the lean existed.
How AI agents shape experts’ judgment calls, published today by Amaç Herdağdelen, Hailey Joren and Alex Tamkin, is the write-up. The headline number is 86%: across all four decisions, that’s how often the analyst’s final answer matched whichever side the coin had handed their agent.
The Trial Randomized the Agent’s Position, Not the Analyst’s Task
Every analyst got the same scenario, a fictional farming cooperative deciding whether to expand an advisory program, and roughly an hour with the agent. The decisions were genuinely contestable, the sort of methodological fork where two competent people can disagree. That’s what makes the randomization work: there was no right answer the agent was arguing against.
The sharper figure is the switch rate. Where an analyst had already written down a position and the agent came back arguing the other way, the analyst moved to the agent’s side 60% of the time. A stated prior held up less than half the time against an hour of coin-flipped argument.
Analysts Credited Themselves for the Agent’s Conclusion
The behavioral result is one thing. The self-report is where it gets uncomfortable.
Asked whose recommendation the final answer was, 52% of analysts said it was more their own than the agent’s. And 77% reproduced the agent’s arguments in their own write-ups without saying where the arguments came from. Felt authorship and actual authorship came apart, with the gap running in the direction that makes review processes look stronger than they are.
If you ask a reviewer whether they independently verified the agent’s conclusion, roughly half of this cohort would have said yes in good faith while agreeing with a coin flip.
The Agent Was Told to Argue Honestly, Which Muddies the Reading
Anthropic did not instruct the agent to mislead anyone. The system prompt told it to make “the strongest honest case the data allowed” for its assigned side. That matters, because it leaves two readings of 86% that the paper does not separate.
One is automation bias: experts rubber-stamping a machine because it is confident and fast. The other is that the agent produced genuinely good arguments for a genuinely defensible option, and the analysts updated for the right reasons. A domain expert who changes their mind when shown a sound argument is doing their job. The study measures the behavior, so it cannot tell you which mechanism produced it.
The honest instruction also limits how far the number travels. Anthropic deliberately amplified the agent’s lean to make the effect measurable at all, so 86% is a ceiling produced by an agent under orders to advocate, not an ambient deference rate for ordinary agent use.
Anthropic’s Own Caveats Rule Out the Comparison Everyone Will Quote
The study includes a no-AI group, and the error gap looks dramatic: 6% of AI-assisted analysts made serious methodological errors against 55% of those working without the agent. Anthropic says plainly that the two groups were not comparable. Participants self-selected and their experience profiles differed, so no causal claim about AI improving analysis survives. Expect the 6-versus-55 figure to circulate anyway, stripped of the caveat.
The rest of the limitations are the ones you would guess. One task, one domain, one model, one hour, a fictional cooperative with no consequences for getting it wrong. Analysts with real professional stakes, long-held convictions and a client who will be angry in six months may behave differently. Anthropic says this itself rather than leaving it to critics, which is the right call.
Human Review as an Independence Assumption
The use worth taking from this is narrow and specific. Much of AI safety and security practice assumes a human in the loop supplies an independent check on agent output. This trial puts a number on how much independence survives when the human has spent an hour reasoning alongside the thing they are checking.
The failure also correlates. Hand the same model to five reviewers and they converge on its lean; five people agreeing looks like verification, and it is one opinion counted five times. The automated side has its own version of this, where AISI’s red team got past Google DeepMind’s reasoning monitor with an agent that pretended to hallucinate that a malicious objective was a user instruction. Human overseer and model overseer fail in the same direction for related reasons.
There is also precedent in Anthropic’s own research for trusting the measurement over the self-report. Its interpretability team found Claude recognizes an evaluation 26% of the time and says so under 1%. Here, analysts deferred 86% of the time and reported ownership at 52%. The introspective answer is the weaker signal in both.
The practical ask is small. If your process depends on a human reviewer acting as an independent check, have them form a position before the agent does, and write it down.
Sources: How AI agents shape experts’ judgment calls
Coverage: Anthropic Finds Analysts Defer to AI Agents in Judgment Study • Anthropic study finds AI agents can steer expert judgment
Related on this blog: Claude Flags ‘This Is a Test’ 26% of the Time and Says So Under 1% • [AUTO] Malicious git configs execute code before approval • [AUTO] How Downloads Folder Became an Agent Attack Surface