An AI agent can sound competent while changing the wrong record, missing a constraint or claiming that an unfinished workflow succeeded. For an enterprise, these are concrete failures in a particular environment. Training material that merely resembles office work may not fix them. The larger challenge is turning an observed weakness into varied examples that remain realistic, executable and correctly judged.
ServiceNow CoreAI's AutoSynthData announcement on October 2 presents a method for doing that. The company uses failures from a target agent and successful behavior from a stronger teacher to specify what should be learned next. Its significance is not a promise of unlimited synthetic tasks. It is an attempt to make task generation, execution and verification agree before the resulting examples influence a model.
The main contribution
Each generated task combines a system specification, a user request and a verifier. The system specification defines the environment and constraints; the request states the work; the verifier checks whether the result satisfies it. AutoSynthData generates core tasks and then expands accepted examples into new variants, retaining checks on whether solutions work and whether the tasks exercise the intended capability.
The pipeline uses sanitized capability specifications rather than handing the generator original evaluation prompts and trajectories. That is a useful separation between diagnosing weakness and copying a test. It reduces one obvious route for reproducing evaluation examples, although it does not remove the need for broader held-out testing. Familiar environments and capability definitions can still shape what a model learns.
A verifier has to reject plausible failure
Positive verification asks whether an intended solution passes. Negative verification tests incorrect outcomes to see whether the checker rejects them. Both are necessary. A checker that accepts every final state makes training look successful while teaching nothing useful. A checker that demands one exact sequence of actions can penalize valid alternative solutions and narrow the agent's behavior unnecessarily.
This is why synthetic agent data differs from a collection of fluent answers. The meaning of success resides partly outside the text, in state transitions and constraints. A ticket-routing task, for example, is not complete merely because a response says it was routed. The environment must reflect the requested change without prohibited side effects. That relationship has to survive changes to entities and wording.
What the reported experiment establishes
ServiceNow reports improvements after supervised fine-tuning in two EnterpriseOps Gym domains. In the Hybrid experiment, its selected checkpoint improved mean Pass@1 by 7.2 percentage points. An ITSM experiment also improved. These are company-reported controlled results in the environments used for generation and evaluation, not independent proof of lower error rates across live enterprise deployments.
The scope matters because a generated curriculum can adapt closely to one environment's tools and task structure. A real installation may have different permissions, missing data, customized fields and ambiguous requests. Improvements on the Gym are a reason to investigate transfer, rather than assume it. The company's suggestion that the mechanism could support reinforcement learning is a proposed direction beyond the reported supervised experiments.
What changes immediately
For teams building agent datasets, the practical contribution is a clearer unit of quality control. Instead of asking whether a prompt looks convincing, they can ask whether a task is feasible, its reference solution executes and its verifier distinguishes valid from invalid outcomes. Batch review can then identify repetitions and missing capabilities rather than simply counting accepted samples.
The approach also changes how failures should be recorded. A useful diagnostic identifies the relevant constraint, tool combination and incorrect state. That record can support a family of new tasks without reproducing a customer incident's private details. Organizations still need permission and careful handling for the data they use; synthetic variation does not automatically erase sensitive provenance or proprietary workflow content.
The larger impact is a moving target
As a model improves, yesterday's difficult tasks can become easy repetition. A curriculum that keeps measuring remaining weaknesses may spend generation effort more effectively than a static pile of examples. However, repeatedly adapting to the same test can create its own overfitting. An external evaluation set and new deployment conditions are important checks on whether the system gained a capability or learned a familiar pattern.
If independent studies confirm transfer, this style of data construction could make specialized enterprise agents more practical without requiring every company to label every trajectory manually. That benefit depends on reliable environments and verifiers, which themselves cost engineering effort. Generation is scalable only after the definition of success is trustworthy.
The next evidence to watch is performance on unseen workflows, human audits of accepted and rejected tasks, and the total cost of useful training signal. The bigger picture is accountable automation. A training task should teach an agent to change the world correctly, and its checker must be able to tell the difference between that change and a convincing claim that it happened.
References
Questions, news tips or corrections? Contact [email protected].
