When an agent acts incorrectly, improving its reading seems like a natural remedy. But it may already have read the state correctly.
An October 13 study pairs next-action decisions with independent factual questions about the same states. Across evaluated fast systems, many wrong actions follow correct factual responses. Some controlled failures are repaired by composing the model's own answers in code. Read the research paper.
A fast decision can fail after the facts are read correctly
System One models select an agent's next action by scoring supplied options in a single pass. Their speed makes them attractive for frequent decisions, but their errors provide little explanatory trace. A wrong choice could arise because the model misunderstood the state or because it failed to combine correctly understood facts into a decision. Improving perception addresses only the first possibility.
This study pairs next-action questions with independent factual probes about the same frozen states. The design is behavioural: it tests what the model can answer about the state alongside what it chooses to do. It does not claim to reveal every internal computation behind the action.
Independent probes separate reading from choosing
The benchmark spans simulated environments, customer-service policies, tool use, web navigation and coding. Across seventeen fast systems and a reasoning model, most wrong actions follow correct answers to the factual probes. More accurate systems have a larger fraction of this kind of residual error. The failures also cluster on shared states across systems, suggesting structured difficulty rather than random mistakes unique to one model.
On controlled procedures where the probed facts determine the action, the researchers compose the model's own answers in code. That can repair failures even though no stronger perception model supplied the facts. The missing capability is the combination rule, not necessarily the extraction of another piece of evidence.
Subgoals create a different decision bottleneck
Natural tasks are less fully specified than generated procedures. In ALFWorld-style states, appending factual information helps little in some comparisons because the missing ingredient is the next subgoal. Knowing the state does not specify which intermediate objective moves the agent toward completion. That is a planning problem, requiring a different remedy from a better state summary.
The paper therefore presents the instrument as diagnostic rather than a single leaderboard. Similar overall accuracies can conceal different mixtures of reading and decision failures. A system that already reads well may need explicit composition or selective deliberation; another may benefit more from a clearer state representation or larger context budget.
Composition works best where the rule is known
On generated procedures with sufficient facts and explicit rules, composing narrow factual answers in code brings the tested systems to 0.96–1.00 accuracy in the reported comparison. A confidence cascade retains much of the benefit while requesting a second batch on only about a third of states. This is a concrete intervention, but it applies where the decision rule can be written down.
Open-ended planning lacks that guarantee. The same factual decomposition may not identify which subgoal to pursue, and appending facts can leave performance largely unchanged. That difference explains why the paper recommends diagnosis before escalation: deterministic composition can repair one class of failure cheaply, while another genuinely requires planning, training or selective deliberation.
The probes and labels have limits
Correct factual answers do not prove those facts were used during action selection. Natural-state probes also cannot cover every decision-relevant fact, so some apparent decision failures may reflect unasked information. The authors disclose gold-label issues, including some simulator choices that are optimal despite being marked wrong and location-label problems in an original ALFWorld cohort, and report sensitivity analyses.
Those qualifications make the diagnosis more useful, not less. The diagnosis should determine the remedy. Pair action tests with minimal state probes, then test composition where the rules are explicit and subgoal support where they are not. A more capable reader cannot be assumed to become a more capable decision-maker merely because both functions are implemented by the same model.
Questions, news tips or corrections? Contact [email protected].
