InferenceLiquid AI’s fast decisions still have to read the world
The revealing number in the d1 release is the small cost of asking three questions about the same scene.
Inference
How models reason and run: serving speed, memory, inference-time compute and cost per useful result.
Serving & runtime · Compression & efficiency · Inference-time reasoning
8 articles
Inference
InferenceTen minutes of speech in 1.65 seconds is not a robot’s response time
QVAC’s new on-device recogniser makes the distinction between bulk throughput, short-utterance latency and first-load cost unusually visible.
InferenceThe robot’s real latency problem is the age of its last observation
A KAIST workload study shows why faster inference and more aggressive action overlap do not produce a single best operating point.
InferenceA faster robot can keep its score and lose your task
CARE measures the episodes broken by acceleration, revealing regressions that an unchanged average success rate can hide.
InferencevLLM: Keeping AI answers on time can matter more than generating more tokens
Separating long-prompt processing from answer generation can prevent one request from slowing everyone else. The trade is extra transfer time and a more complex service.
InferenceSGLang: Smaller AI memory can serve more users—if the extra errors are affordable
SGLang, Qwen and NVIDIA compress the memory used by long conversations. The useful saving depends on which answers change and how much repair they need.
InferenceInco AI’s DFlash 2 speeds up answers by making better guesses for the same model
A small model proposes several tokens at once; the main model checks them. Better accepted guesses can reduce expensive model passes without handing the final answer to the smaller model.
InferencePrismML’s smaller model could bring useful AI onto hardware you already own
Bonsai 2’s reported 5.9 GB footprint raises a deployment question: can the compressed model meet your task standard locally, after context memory and fallback costs are included?
No articles in this topic yet
Explore the other topics in this desk.