An agent-monitoring system can detect a dangerous action only if the relevant activity reaches it in a form it can inspect. Coverage, logging and the trigger that starts review are therefore part of the safety mechanism, not merely administrative infrastructure around a classifier.
METR’s October 22 disclosure describes a three-week pilot conducted with Redwood Research inside OpenAI from July 21 to August 11. The researchers had deep access to code, configurations and staff, and produced an internal report. Their public account identifies historical coverage gaps and vulnerabilities, while explicitly limiting the assessment to the system examined at the start of the exercise. It does not establish the expanded system’s current coverage. Read the primary source.
A detected gap needs a denominator
The account distinguishes traffic eligible for monitoring from traffic actually monitored and describes uncertainty in its estimates. It also notes changes made during the exercise and limits agreed on public disclosure. Those distinctions prevent a percentage from becoming a general claim that all internal agent activity was either protected or unprotected.
For evaluators, the immediate lesson is to test the observation path as well as the monitor’s judgments. Compaction, failed requests, long-running actions and state transitions can affect which evidence is reviewed. Such tests should verify both what is visible and what is missing, without treating a low monitor-error rate as a substitute for coverage.
High-access outside assessment could provide stronger evidence about internal controls if scope and publication independence are clear. The next useful evidence is follow-up against changed systems and transparent descriptions of access limits. A historical pilot is valuable precisely when its findings remain attached to the period and configuration it actually assessed.
References
Questions, news tips or corrections? Contact [email protected].
