Optimising a soft distribution over examples can produce an attractive mathematical answer that cannot be turned into an allowed dataset. Rounding is where source quotas or selection budgets can break.
An October 10 paper on multi-source selection constructs probability distributions supported only on feasible subsets. Its ReSGD method combines this representation with a gradient estimator designed to reduce costly lower-level retraining. Read the research paper.
Feasibility belongs in the representation
The advantage is structural: the optimiser searches among choices it could actually return. It does not first solve an easier problem and hope the final conversion preserves its value.
The approximation guarantee depends on stated mathematical assumptions, including weak submodularity. Those conditions should not be silently assigned to an arbitrary language-model training pipeline.
For practitioners mixing sources, this is a reminder to distinguish optimiser performance from constraint satisfaction. A nominally better selection is unusable if it violates licensing limits, source caps or a fixed annotation budget.
The paper's empirical results support the proposed method in its evaluated tasks. Its wider contribution is a design principle: encode operational constraints early enough that optimisation can improve the real decision, rather than a relaxed decision that becomes something different at deployment.
Questions, news tips or corrections? Contact [email protected].
