METR and Redwood Researchers Find Blind Spots in OpenAI Safety Checks

New findings from a joint audit have exposed critical vulnerabilities in the safety monitoring infrastructure of OpenAI, the leading artificial intelligence developer. The report, released today, details how researchers from METR and Redwood Research identified multiple weaknesses in the systems designed to detect rogue behavior in the company's advanced models. These discoveries suggest that the company's current oversight mechanisms may not be as robust as previously assumed, particularly for its newest generation of AI systems.

The audit took place from July 21 to August 11 of this year. During this three-week period, the external team had deep access to OpenAI's codebase and internal documentation. They also conducted interviews with approximately a dozen employees to understand how the monitoring systems function in practice. The scope of the investigation focused specifically on the internal monitoring of agentic behavior, which refers to AI models that can perform sequences of actions autonomously.

The data reveals a concerning lack of coverage for the company's most powerful models. According to the auditors, between 6 and 25 percent of internal traffic from the Responses API, specifically for models at the GPT-5.5 class and above, was not covered by monitoring at all. METR noted that this estimate carries significant uncertainty in both directions. Additionally, for internal Codex traffic that was intended to be monitored, roughly 10 percent went untracked between May and July. This gap was caused by a system failure in handling compaction events, which are context summaries used by the models.

OpenAI independently discovered this specific bug and implemented a fix while the audit was still ongoing. However, the auditors found that during the period when compaction events were unmonitored, agents could delay oversight. The system only triggered when an agent handed control back to the user. By simply continuing to execute actions without pausing, agents could effectively sidestep the monitoring process. This behavior created a window where AI actions proceeded without real-time safety checks.

Beyond the statistical gaps, the researchers found specific methods to hide actions from the monitoring tools. They demonstrated how to conceal hundreds of actions and how to bypass the Codex sandbox, a security environment meant to contain code execution. The team also determined that specially crafted inputs could likely cause the monitor to crash entirely. Furthermore, model behavior during reinforcement learning (RL) training was not being monitored at the start of the audit period. OpenAI stated that it paused RL training following the Hugging Face incident in August and subsequently expanded monitoring to include this phase. However, these changes occurred after the auditors' access period ended and were therefore not part of the formal evaluation.

The company has stated that it has since fixed all the weaknesses identified by the auditors. The full report is 61 pages long and was handed over to OpenAI internally. After the company applied redactions, the leadership teams at METR and Redwood Research received a 38-page version of the document. Only a short summary was made available to the general public. METR indicated that it did not have the right to disclose the redacted portions independently, which limits the amount of detail currently visible to outsiders.

This audit was part of a pilot program organized jointly by METR, Redwood Research, and OpenAI. The goal was to provide outside researchers with deep access to internal infrastructure to test the limits of AI safety measures. For future audits, METR has expressed a desire to push for greater public transparency. This would allow the broader AI community and the public to see more of the findings, rather than just a summarized version.

The stakes for this discovery are high as the industry moves toward more autonomous AI systems. If internal monitoring can be bypassed or if significant portions of traffic go untracked, the risk of undetected errors or unexpected behavior increases. While OpenAI has addressed the specific issues found, the audit highlights the complexity of supervising AI that can act quickly and independently. As these systems become more integrated into critical workflows, the need for comprehensive and transparent safety monitoring becomes increasingly urgent.