An agent extracts a reusable skill from a successful run, then uses it to guide training. The skill may be insightful, incomplete or misleading. Treating every generated skill as equally reliable wastes that distinction.
RSIPO's October 11 paper co-evolves skills and policy behaviour, with empirical-Bayes shrinkage to calibrate trust in teacher scores. The authors report gains on their embodied and search-based agent evaluations. Read the research paper.
Reusable skills can become unreliable teachers
An agent's successful trajectory contains potentially reusable knowledge: a search strategy, a tool sequence or a way to recover from an obstacle. Summarising that knowledge as a skill can make it available to later tasks and training. But success does not guarantee that the summary captures the right cause. It may omit a condition, preserve a lucky shortcut or generalise beyond the situation in which it worked. More self-generated guidance can therefore mean more supervision without more trustworthy information.
RSIPO addresses both sides of the loop: how skills develop as the policy changes, and how much influence a skill-enhanced teacher should have over the student.
Skills and policy evolve together
Recursive Self-Improving Policy Optimization has the policy extract skills from completed trajectories and trains its skill-generation capability through trajectory replay. This avoids treating a skill library as a fixed asset constructed before learning begins. The library can change as the policy encounters new situations and produces different experiences.
The framework combines trajectory-level reward information with denser teacher guidance. Crucially, it uses empirical-Bayes shrinkage to calibrate trust in the teacher's scores. The motivation is that conditioning a teacher on an agentic skill can sometimes make its advice less reliable. Guidance receives influence according to an estimate of its reliability rather than simply because it was generated by the system itself.
Confidence changes the meaning of self-improvement
The paper reports improvements with Qwen-family models on embodied interaction and search tasks. With Qwen2.5-7B, the authors report an average success rate of 98.6 percent on ALFWorld, 2.5 percentage points above the next-best comparison and 17.4 points above standard GRPO. Those results are specific to the tested setups and should not be read as a general reliability estimate for autonomous agents.
The conceptual contribution is the separation of usefulness from trustworthiness. A skill can contain valuable information and still be too uncertain to deserve full control over a training update. Calibrating its influence offers an alternative to either accepting all generated advice or discarding self-generated experience altogether.
Training-time skills are not necessarily served with the agent
The comparison distinguishes methods that retain a skill library at test time from methods that use skills during training. RSIPO acts without skills at test time in the stated evaluation, as do most of its compared methods. Some skill-prompting baselines explicitly keep skills available. This boundary matters when assessing the final policy rather than only the quality of a memory-augmented rollout.
Search evaluation includes both single-hop and multi-hop question-answering datasets, while ALFWorld measures embodied-task success. Those metrics test different forms of agent performance. The broader pattern supports the proposed training loop, but the individual panels are still necessary to see whether a generated skill improves simple retrieval, multi-step evidence gathering or interaction with an environment.
Evaluate the library as well as the learner
An overall success gain does not reveal whether the policy improved its basic actions, produced better skill summaries or became less vulnerable to bad advice. Those components need separate tests. Deliberately incomplete, contradictory and out-of-scope skills can expose whether the trust mechanism responds to the failures it is meant to limit.
For a long-running training process, another question is whether reliability estimates remain useful as both the policy and its library change. A guide that was helpful earlier may no longer fit the current learner. RSIPO makes that moving relationship explicit, which is valuable precisely because a recursive loop cannot assume that yesterday's teacher remains appropriate for tomorrow's student.
Questions, news tips or corrections? Contact [email protected].
