A retrieval service may keep one model resident to select evidence and another to write an answer. That division can duplicate memory and passage processing, even though the generator will eventually need to read the selected material itself.
Airun Intelligence Labs introduces Glance in an October 29 preprint. A lightweight adapter and pointer head let one resident 9B backbone score candidate passages from a shared prefill. The paper reports roughly 0.1 GiB of additional memory per decision skill. After 39 minutes of MS MARCO fine-tuning, its tested model reaches nDCG@10 of 0.741 on SciFact and 0.345 on NFCorpus. Evaluation covers two retrieval datasets, one question-answering dataset and one backbone family. Read the primary source account.
Shared context changes the serving arrangement
The system encodes candidate passages once and branches short passage-specific probes from the cache. The probes produce relevance scores using the same backbone that can later generate an answer. In the main arrangement, the chosen passages are encoded again with the adapter off; an activated-adapter variant instead allows generation to continue from the original passage cache. Those variants have different cache implications and should not be described as the same execution path.
The authors also evaluate selected context on 300 HotpotQA questions under the same generator and report an answer-F1 improvement of 0.045 over cross-encoder-selected context, with a paired bootstrap confidence interval of 0.014 to 0.076. Holding the generator constant helps connect that result to evidence selection, while the sample and dataset remain part of its scope.
Fewer resident models does not mean every request is faster
The paper explicitly notes that a dedicated cross-encoder remains faster per query. Glance therefore presents a tradeoff rather than a universal speedup: memory residency and shared computation can improve one operating constraint while another architecture wins on request latency. A serving decision needs to name which constraint matters for the actual workload.
The immediate option is a released adapter and code path that teams can benchmark with their candidate counts and context lengths. A meaningful comparison would measure the complete retrieval-to-answer request, including cache occupancy and concurrency, and keep answer quality attached. It should not substitute the adapter’s small size for the memory requirements of the resident backbone and all cached passages.
The broader possibility is a service that shares a model across generation and bounded decision skills without managing several large backbones. Its usefulness depends on predictable switching, suitable quality and acceptable queueing under real demand. The next evidence is representative end-to-end operation across additional backbone families and datasets, with the memory–latency tradeoff preserved.
Source organizations: Airun Intelligence Labs.
References
Questions, news tips or corrections? Contact [email protected].


