Three in four AI-generated vulnerability patches leave something broken

Off-by-1 Labs, a security research group within 1Password, published findings indicating that approximately 75% of vulnerability patches generated by frontier large language models contain significant defects. The study evaluated over 6,000 attempts to fix six specific vulnerabilities, including the Linux kernel privilege escalation CVE-2026-31431 and the Chromium bug CVE-2026-8512, using models such as ChatGPT 5.5 and Claude Opus 4.8. A key finding is that while many AI-generated fixes successfully close the original exploit path, they frequently introduce new security weaknesses or break existing functionality without triggering test failures. Researchers concluded that these automated patches require rigorous review by skilled engineers, as distinguishing a valid fix from a convincing but incorrect one is difficult and costly.