Choosing a small training subset can itself require enough memory to exclude the hardware that needs the subset most. Influence estimation is a good example: useful in principle, expensive at language-model scale.
ZOIF, posted October 11, replaces backpropagation-dependent influence calculations with zeroth-order estimates and finite differences. Its authors report competitive subset utility at lower peak memory in their evaluated settings. Read the research paper.
Cheap selection must include the search bill
Forward-only does not mean free. Estimating a direction through repeated perturbations can trade memory for computation and introduce variance. The relevant comparison includes all scoring passes before fine-tuning starts.
Approximate influence is also local. An example useful near one checkpoint may become redundant later, especially when adaptation is large. A one-time ranking should not be presented as a permanent property of the data.
For constrained teams, compare total GPU-hours, peak memory and downstream quality under the same selection budget. Include a simple random-subset baseline; an elaborate selector needs to repay its overhead.
ZOIF's practical contribution is access. It makes a principled selection approach feasible in settings where backward-pass memory was the barrier. Whether it saves money depends on the cost of obtaining the ranking as well as the training it avoids.
Questions, news tips or corrections? Contact [email protected].
