An agent can write a convincing completion message while changing the wrong customer record. Enterprise work therefore needs a test of the resulting application state, including what the agent was required to preserve. A success metric that checks only the explanation can reward the appearance of work.
Prime Intellect’s October 15 account with NVIDIA describes 2,500 verified training tasks and GRPO post-training of Nemotron 3.5 Super VL. The team reports an increase from 34.0% to 57.7% on the public AutomationBench-AA split after 100 training steps. The source specifies about 11 hours on 128 GB200 GPUs. These are the collaborators’ reported results on simulated workflows, not a measured business productivity gain. Read the primary source.
Objectives and guardrails are different checks
The scoring method gives partial credit for completed objectives but removes a task’s credit if a protected guardrail is broken. That captures an important distinction: updating several correct fields does not compensate for an unauthorized change elsewhere. It also makes verifier quality central, because an omitted constraint cannot influence the reward.
A team applying this approach should first ensure that the environment reflects its actual records, permissions and ambiguous requests. Held-out evaluation needs to test generalization beyond the training patterns. Reviewing alternative valid solutions is as important as blocking shortcuts, otherwise the verifier can punish useful behavior or reward a brittle sequence.
The immediate contribution is a concrete example of training against outcomes rather than claims. A longer-term deployment-and-retraining loop could improve specialist agents if real failures become carefully verified tasks. Evidence from independent held-out workflows, with guardrail violations reported separately, would show how much of the public-split gain transfers.
References
Questions, news tips or corrections? Contact [email protected].
