Fine-tuning every state from expert demonstrations can teach a new skill quickly while disturbing old ones. Refusing expert help entirely can leave reinforcement learning struggling at the same difficult state.
TSRL, posted October 13, activates supervised guidance on hard states within an RL training process. The authors evaluate acquisition and retention across embodied and search tasks at two model scales. Read the research paper.
Intervention has a location
The idea is to concentrate the strongest teaching signal where the policy actually fails. Routine decisions remain under the broader learning process, limiting unnecessary parameter movement.
The failure detector becomes part of the method's assumptions. If it misses subtle errors, guidance arrives too late; if it labels every uncertain state hard, the system approaches ordinary supervised training.
Compare expert usage as well as final accuracy. Selectivity is valuable only if it reduces assistance or forgetting at comparable task performance.
The paper's retention findings support the approach in its evaluated sequence of tasks. The general principle is useful beyond one recipe: combining learning methods does not require blending them uniformly. Choosing where an expert intervenes can matter as much as choosing how much expert data the model receives.
References
Questions, news tips or corrections? Contact [email protected].
