Fine-tuning can improve a speciality while weakening general capabilities needed around it. A question requiring both then falls between two otherwise capable checkpoints.
SFT-as-Context, posted October 8, gives the specialised model's response to the parent model as context. Across the authors' evaluated pairs and tasks, the arrangement preserves much of each model's strength and can solve combined questions neither handles alone. Read the research paper.
Composition happens at answer time
This does not repair forgotten weights. It avoids relying exclusively on them. The parent receives domain-specific material but retains responsibility for producing the final answer.
That creates a familiar evidence problem: the specialist may be wrong, and a fluent specialist response can anchor the parent. Test incorrect, irrelevant and contradictory specialist outputs rather than evaluating only helpful ones.
Two-model serving also adds latency and cost. Compare it with a single larger model and with task-specific routing under the same operational budget.
The method's training-free nature is useful when retraining is unavailable. Its deeper implication is that capability retention can be an orchestration choice. A newly fine-tuned checkpoint need not become the sole model responsible for every part of the task it was designed to improve.
Questions, news tips or corrections? Contact [email protected].
