A draft model can improve its local predictions while barely changing the number of target-model calls needed to finish an answer. The serving objective and training objective are not identical.
An October 7 paper models speculative decoding as a Markov reward process and derives an Expected Decoding Rounds objective. Its authors use target-model rollouts to optimise the coupled process and report improvements in evaluated parallel drafters. Read the research paper.
Today's rejection changes tomorrow's starting point
How many tokens a round accepts determines where the next round begins. Training each block as an isolated prediction problem ignores that dependence.
The new objective accounts for the states the decoder actually visits. That can identify which mistakes are expensive in repeated execution, rather than treating every local error as equally important.
Round count is still not elapsed time. Drafting, verification and hardware utilisation can change the cost of a round. A deployment comparison should therefore report both the theoretical objective and measured latency.
The paper's useful shift is to train against the process being accelerated. A good speculative model is not simply a small language model with high token accuracy; it is a component whose errors and successes help the complete decoder finish sooner.
Questions, news tips or corrections? Contact [email protected].
