A training curve can rise while previously correct answers become wrong. New successes may simply outnumber the losses.
An October 9 self-distillation study associates this degradation with changes in hidden-representation geometry. Its proposed regulariser encourages within-cluster compactness and between-cluster separation, reducing regressions in the authors' evaluated methods. Read the research paper.
Follow the same questions across checkpoints
Aggregate scores cannot identify what survived. Paired evaluation can: mark questions solved earlier, then measure how many become unsolved after the next update.
The geometric association provides a candidate mechanism, not a guarantee that all forgetting has the same cause. A regulariser can also preserve an unhelpful representation too strongly and limit acquisition.
Report retained capability and new capability separately, with a consistent question set and additional unseen tests. That avoids presenting a changing mix of successes as uniform improvement.
The paper supports a targeted intervention in on-policy self-distillation. Its broader contribution is a more demanding definition of progress. A self-learning system should account for the knowledge it destroys while adding something new; a positive average alone is too forgiving to reveal that cost.
Questions, news tips or corrections? Contact [email protected].
