A few prompt tokens explain most MoE prefill speed-up losses
Cutting experts during prefill is cheap, but it damages specific tokens, not the whole prompt
The facts
- A single-author study examines running fewer routed experts during prefill, a common way to cut mixture-of-experts (MoE) serving costs, across four MoE model families
- At half the experts, F1 loss in three families is mostly 'answer form': the model finds the right answer but stops giving it briefly
- In Qwen3-30B-A3B the loss traces to the two chat-template tokens before the answer; in DeepSeek-V2-Lite it traces to the first 8-32 prompt tokens holding the instruction, accounting for about half and a fifth of the damage respectively
- Keeping just ~4% of tokens at full experts, placed at the task interface, keeps F1 within one point of the unmodified model; placing them randomly in documents costs about 3 points, though only two of five planned comparisons were confirmed
- Time to first token falls about 14% on 11K-token prompts but shows no improvement at 2K tokens
A new study on mixture-of-experts (MoE) models finds that cutting the number of routed experts used during prefill does not damage accuracy evenly across a prompt. Prefill is the initial pass over a prompt before generation begins. Instead, the loss concentrates in a handful of tokens. Author Zhiyuan Wang tested four MoE families on short-answer question answering. The study used token-, layer- and attention-path interventions to find where the damage actually lands.
At half the normal expert count, three of the four families lost F1 score mostly to a change in answer form rather than a wrong answer. The model still located the correct answer but failed to state it. In Qwen3-30B-A3B, the paper traces about half of that damage to just two chat-template tokens sitting right before the answer. In DeepSeek-V2-Lite, roughly a fifth of the damage comes from the first 8 to 32 prompt tokens, where the model's instruction is placed. Full-expert processing shifts the first token's attention mass, norm and largest activation by less than 4%. Restoring so-called super experts or massive activations recovers at most a quarter of the lost accuracy — well short of simply keeping the sensitive tokens at full expert count.
On realistic prompts, running only about 4% of tokens through the full expert set keeps F1 within one point of an unmodified model, provided those tokens sit at the task interface, where instructions meet the answer. Scattering that same 4% randomly through supporting documents costs about 3 points instead. The paper notes only two of five pre-specified primary comparisons were confirmed; the rest were undecided or not run. The sensitive tokens always sat at the ends of the prompt in every layout tested, the paper reports. It says this makes it impossible to distinguish a genuine positional rule from one that simply reads surface markers like the answer boundary. On the efficiency side, time to first token fell about 14% on 11,000-token prompts. It showed no improvement at 2,000 tokens.
Why it matters
Anyone cutting prefill experts for cost should stop applying uniform reduction. Instead, protect the instruction and chat-template tokens at full expert count. The efficiency gain only materialises on long prompts. Short-context deployments gain nothing from this trick while still risking answer-form failures.
Primary source
arXiv cs.CL — arxiv.orgWritten from the primary source above by The Neural Net’s automated desk (claude-sonnet-5, theneuralnet/pipeline@1) without human review. How this works. Corrections: check the source first — if we got it wrong, the source link is the record.