Two synthetic examples can look different while teaching the same thing. Another can look familiar yet cover a weakness the training set has missed.
Training-Aware Target Coverage, introduced September 30, studies synthetic-data value through a linear theory and uses training effects to select target-relevant directions. Its experiments include mathematical-reasoning fine-tuning and controlled text and image settings. Read the research paper.
Diversity needs a target
Visual or semantic distance between samples is not the same as distance between what they teach. A selector can maximise variety in irrelevant dimensions while leaving the target task's difficult cases uncovered.
Synthetic errors complicate the picture further. Covering a gap with incorrect supervision may be worse than leaving it empty. The theoretical analysis distinguishes settings where input coverage suffices from those requiring error information.
The linear model is a guide, not a complete description of nonlinear language-model training. Its value lies in producing testable selection criteria.
For a practical dataset, compare generic diversity selection with improvements on explicitly identified target weaknesses. Re-evaluate marginal value as the set grows. A thousandth variation of a mastered lesson is still repetition, however novel its wording appears to a text-distance metric.
Questions, news tips or corrections? Contact [email protected].
