Speculative decoding tries to approve several guessed tokens at once. In an offloaded mixture-of-experts model, those guesses can summon different expert weights from host memory, turning a larger draft into more waiting.
SpecFit, posted October 11, chooses draft subtrees by predicted throughput rather than acceptance length. Its cost model includes the union of experts activated across the candidates. The authors report gains on their evaluated summarisation and mathematics workloads. Read the research paper.
A token's cost depends on its neighbours
Two candidates that use the same experts can share transfer costs. A third may require a new set of weights. Counting all tokens equally misses that marginal expense.
The optimisation is tied to offloading and cache conditions. An all-resident model on different hardware may favour another draft shape.
Measure host-device transfers, verification time and committed tokens together. Acceptance rate alone can reward a configuration that slows the actual service.
SpecFit exposes a useful limit of algorithm-level metrics. A draft is valuable when its expected progress exceeds the hardware work it introduces. The best speculative tree is therefore partly a property of the memory system, not just the drafter's predictive accuracy.
Questions, news tips or corrections? Contact [email protected].
