New AI system finds hidden software bugs by writing math proofs

A new collaboration between Anthropic and Theorem has demonstrated that artificial intelligence can now mathematically verify software code with minimal human intervention. The project represents a significant shift in how developers might approach code safety in the future.

For decades, formally proving that software runs without errors under all conditions was a task reserved for highly specialized teams and critical systems like aircraft avionics. Verifying the small operating system kernel seL4 required twenty person-years of expert labor, making the process prohibitively expensive for general use. Anthropic's Frontier Red Team and its partner Theorem changed this dynamic by allowing the Claude model to handle the proof generation. In just a few weeks, the system produced approximately 180 million lines of mathematical proof code based on test programs from 123 open-source libraries in Google's OSS-Fuzz security project.

The process did not involve writing new software. Instead, Claude translated existing crash tests into mathematical problems and analyzed the compiled machine code to generate formal proofs. An independent verification program named Rocq then checked these proofs automatically. Anthropic describes the workflow as fully autonomous, although the underlying tools and formal foundations were built by human engineers at Theorem. This approach requires very few human reviewers compared to traditional methods.

Beyond generating proofs, the system identified hundreds of bugs that conventional testing had missed for years. Many of these flaws only appear under specific conditions, such as unusually large input files or particular sequences of memory access. In one notable instance, the team found as many security bugs in a single week for a widely used PDF reader as are typically reported in an entire year. The system also caught a logical error in another application that had remained undetected for more than a decade. Anthropic notes that even when a complete proof fails, the attempt often surfaces real issues.

Despite these successes, the company acknowledges that formal verification does not guarantee a software product is entirely error-free. The proofs are valid only under strict conditions, including a specific processor architecture, single-threaded execution, and inputs under 64 MB. Since these libraries often run on diverse hardware and configurations, the verification is a powerful but limited tool. The computational cost is also significant, with individual proofs consuming hundreds of millions of tokens.

The financial implications of this technology are still being assessed, as Anthropic has not reported the total project costs. However, the ability to verify code automatically could reduce the time developers spend on manual reviews. A smaller test run indicated that proofs are reusable and that newer models are becoming more efficient. Specifically, the Opus 5.5 model solved half the proofs using about 40 percent fewer tokens than its predecessor, Opus 5.

This development strengthens the argument for broader software automation. If AI can write code and prove its correctness within defined limits, the industry's focus may shift from merely asking if AI can program to asking if it can guarantee safety. Anthropic expects this method to become as routine as standard testing in the near future. Theorem has already targeted critical infrastructure libraries such as glibc, OpenSSL, and cURL for future verification efforts.