Self-distillation needs a teacher that offers something the student lacks. That advantage can come from extra information—but it can also come from making the same weights work differently.
LoOPSD, posted October 13, searches for layers to repeat while keeping weights fixed, then distils the resulting teacher into the student. Repeated search-and-distillation rounds improve the authors' evaluated mathematics and logic models. Read the research paper.
Additional computation becomes supervision
The teacher is not smarter because it owns another checkpoint. It is stronger under a selected execution pattern. Distillation then attempts to make some of that benefit available without repeating the full search at inference.
Search cost belongs in the training budget. A cheap deployed student can still have an expensive development process, and the best layer pattern on one task may not transfer.
Test retained improvements under equal answer budgets and unseen problem families. Otherwise a longer teacher computation can look like durable student learning when the deployment still needs the extra work.
The method opens a specific route to self-improvement: optimise how the model computes, then teach the ordinary execution path to approximate that behaviour. It is a bounded experimental loop, not evidence that repeating layers indefinitely produces unlimited intelligence.
Questions, news tips or corrections? Contact [email protected].
