Sparse models can spend substantial training time moving tokens between devices rather than processing them. Routing only a few experts per token does not eliminate communication: inputs must reach the devices holding those experts, and results and gradients must return.
Prime Intellect’s October 23 Prime Mega MoE account extends DeepSeek’s forward kernel with an overlapping backward implementation. The company reports up to 3.11 times faster MoE kernels and up to 1.34 times faster end-to-end training against its stated baselines. The latter experiments use H200 clusters and specified Qwen3 models, sequence lengths, precision and parallelism. Read the primary source.
What the evidence establishes
The two speedups answer different questions. A large improvement inside one layer can become a smaller full-step improvement because attention, optimization and other work remain. Keeping both measurements visible helps a team estimate whether the proposed change targets its own bottleneck.
The implementation overlaps communication and computation, allowing some tokens to be processed while others move between devices. That opportunity depends on routing patterns, topology and the amount of computation available to hide the transfer. A different cluster or batch shape can therefore change the result without contradicting the published experiment.
What changes in practice
Adoption should begin with a matched replay of the receiving training job. Record numerical behavior and convergence alongside step time, and account for compilation or warmup separately. Compare complete runs where possible: saving time per step is useful only if the same budget still produces an acceptable model.
The immediate contribution is a concrete extension that makes the fused path usable during training rather than only forward execution. Longer-term savings could matter as more work shifts to sparse models, but they will depend on supported hardware, data types and maintenance across model revisions.
The evidence to watch
The next evidence is independently reproduced full-training performance across configurations, including cases with limited gains. That would show where overlap is a dependable systems improvement and where another stage still determines the training budget.
Source organization: Prime Intellect.
References
Questions, news tips or corrections? Contact [email protected].


