A sparse model can own many experts while consulting only a few for each token. Its useful capacity therefore depends on selection: the most valuable expert is not necessarily the one a single router assigns the highest score.
Baidu’s ERNIE team presents Multi-View MoE in an October 29 preprint, with an October 28 date inside the manuscript. Multiple independently initialized routers contribute selection scores, and their relative preferences also mix learned views into each selected expert’s input. On the reported 24-layer, 10B-total/1B-active model trained on 708 billion tokens, the complete method improves the mean across nine downstream tasks by 1.18 percentage points over its baseline. The authors retain the active-expert count, expert FLOPs and all-to-all communication volume. Read the primary source account.
Scores do two jobs
The routers do more than vote on a shortlist. If two experts receive equal total support but different patterns of support across routers, they can receive different mixtures of input views. Each selected expert still executes once. The mechanism tries to extract more information from a selection process already taking place, instead of obtaining richer evidence by running every expert on every token.
The paper separately evaluates the multiple-router component and reports a smaller improvement than the complete design. That ablation matters: it helps distinguish the value of changing expert selection from the value of changing what selected experts compute. A result for the combined system should not be assigned wholly to one component.
A fixed expert budget is a specific comparison
Keeping expert FLOPs fixed is useful, but it is not identical to showing equal end-to-end serving cost. Routing, view mixing, memory access and implementation details remain parts of the complete model. The reported comparison supports a quality claim under a defined architecture and training run; it does not establish a universal latency improvement across hardware or larger models.
For model developers, the immediate contribution is a concrete alternative to treating one routing score as the entire selection decision. The appropriate reproduction would preserve the baseline, data budget and active-expert configuration, while measuring both task quality and actual execution cost. The comparison is strongest when the extra routing work remains visible rather than disappearing behind the unchanged expert count.
In the longer term, differentiated routing preferences could become information used inside computation as well as a gate before it. That possibility rests on repeated gains beyond the reported backbone and on a practical implementation that maintains its intended efficiency. The next evidence to watch is whether the same separation of selection and input shaping survives changes in scale and workload.
Source organizations: Baidu.
References
Questions, news tips or corrections? Contact [email protected].


