An interactive agent and a busy inference service ask different questions of the same hardware. One user wants the next response quickly, while an operator wants useful work across many simultaneous requests. A single headline token rate can conceal which of those objectives a configuration actually optimizes.
The October 14 SGLang and Miles account describes day-zero support for Step 5 Preview. It reports about 299 tokens per second for one request and about 740 output tokens per second per GPU at 64 concurrent requests, on four GB300 GPUs under different serving choices. These are not interchangeable measurements of one user’s experience. Read the original source.
Keep the denominators attached
Concurrency changes the denominator. Combining work from multiple requests can improve aggregate utilization while each request waits for a share of the system. A service may therefore deliver more total tokens without reducing an individual task’s elapsed time. The right comparison follows the actual workload rather than selecting the larger number.
Long agent histories introduce another distinction between reading a prompt and generating the next output. A request with a large accumulated context can spend significant time before the first token arrives. Later requests may reuse cached history, so measurements with and without cache hits represent different application conditions.
Replay the receiving workload
The report identifies precision and speculative-decoding conditions alongside its results. Those details belong with any reproduced comparison. A serving optimization must still preserve the output behavior required by the application; higher throughput with a different numerical or decoding configuration needs a quality check as well as a timing check.
For developers, the practical experiment is a workload replay that retains prompt lengths, output lengths and arrival patterns. Measure time to first response, completion time, aggregate output and failures together. A system that behaves well at a steady average load may behave differently during a burst or after a cold start.
A service-level comparison
The immediate benefit is an available implementation and a concrete serving configuration to investigate. The longer-term opportunity is less infrastructure work between a new model release and a usable service. That depends on reproducibility across the hardware and request mix the receiving team actually operates.
What to watch is a repeatable service-level comparison with explicit latency targets and accepted outputs. The published numbers are useful starting points precisely when their conditions remain visible, rather than being promoted into a universal speed claim.
References
Questions, news tips or corrections? Contact [email protected].


