1Password’s Off-by-1 Labs studied 6,080 AI-generated security patches across six recently disclosed CVEs, testing ChatGPT 5.5 and Claude Opus 4.8. Only 26% of patches fully resolved vulnerabilities without changing application behavior. Worse: 75% left exploitable paths open or introduced new security flaws.
The gap between “blocks the demonstrated exploit” and “actually secure” is enormous. Many patches used narrowly targeted input checks rather than fixing root causes. When models received correct guidance on the fix, they succeeded 67% of the time. With incorrect guidance, they dropped to 17%.
This matters less as a model ranking than as calibration for human review costs. Generating a patch costs $2-3; reviewing one costs orders of magnitude more. The research suggests AI patch generation works best as a tool for accelerating expert human work, not replacing it. Model performance varies significantly across codebases. The data shows prompt quality matters enormously: garbage in, garbage out remains the law.
Sources: Frontier Models’ Vulnerability Patches are Often F.L.A.W.E.D. • Three in four AI-generated vulnerability patches leave something broken
Coverage: AI-generated patches fail half the time
Related on this blog: [AUTO] Langflow RCE Added to KEV After a Month of Silence • [AUTO] Paperclip’s Real Problem: When Configuration Is Code • [AUTO] Claude Code auto-executes repository configuration