How models reason and run: serving speed, memory, inference-time compute and cost per useful result.
Serving & runtime · Compression & efficiency · Inference-time reasoning
4 articles
InferenceSeparating long-prompt processing from answer generation can prevent one request from slowing everyone else. The trade is extra transfer time and a more complex service.
InferenceSGLang, Qwen and NVIDIA compress the memory used by long conversations. The useful saving depends on which answers change and how much repair they need.
InferenceA small model proposes several tokens at once; the main model checks them. Better accepted guesses can reduce expensive model passes without handing the final answer to the smaller model.
InferenceBonsai 2’s reported 5.9 GB footprint raises a deployment question: can the compressed model meet your task standard locally, after context memory and fallback costs are included?
Explore the other topics in this desk.
