Researchers make layer-skipping LLMs actually serve faster
vSkipper plugs dynamic layer skippers into SGLang, turning skipped compute into real gains.
The facts
- FlexiDepth skips 8 of Llama-3-8B's 32 layers on average but decodes 14.6–21.0% more slowly than the base model under its own generation loop, per the paper
- vSkipper is a virtualization layer that makes dynamic layer skippers pluggable in serving engines while keeping continuous batching, fixed-shape batches, paged KV caching, and captured decode graphs
- Tested in SGLang with the released FlexiDepth checkpoint, vSkipper cuts mean end-to-end latency by 36.8% on GSM8K and 13.6% on BBH at the knee of upstream's load curve
- Under saturation it raises request throughput by 11.3% and 7.4%, with no statistically resolved quality loss beyond the checkpoint's own, and works across synthetic skip policies, two Qwen3 skippers, and three GPUs without workload-specific tuning
Dynamic layer skipping is meant to cut LLM inference costs. Each token runs through only some of a model's layers rather than all of them. In practice it has been making things slower. A paper behind vSkipper, from Wei Da, Yavuz Ferhatosmanoglu and Evangelia Kalyvianaki, shows why. FlexiDepth skips 8 of Llama-3-8B's 32 layers on average. Yet it decodes 14.6–21.0% more slowly than the unmodified model. The cause: its generation loop was never built to work with the batching and caching machinery that production serving engines rely on.
vSkipper is the fix. It is a virtualization layer that sits inside a serving engine and makes layer skippers pluggable. It does this without breaking continuous batching, fixed-shape batches, paged KV caching or captured decode graphs. At each layer where skipping is possible, it groups tokens by the skipper's decision. It routes tokens through the skip path only when doing so is predicted to pay off. The authors implement it as a fork of SGLang. They test the released FlexiDepth checkpoint against stock SGLang, under matched prompts, arrivals, output lengths and launch settings.
The results are stark. At the knee of the upstream load curve, where latency starts climbing sharply, vSkipper cuts mean end-to-end latency by 36.8% on GSM8K and 13.6% on BBH. Under saturation it lifts request throughput by 11.3% and 7.4% on the same two benchmarks. The authors report no statistically resolved quality loss beyond whatever the FlexiDepth checkpoint already carries. They say the gains hold across synthetic skip policies, two Qwen3-based skippers, and three different GPUs, without per-workload tuning. The code is open-sourced as an SGLang fork.
The authors describe vSkipper as the first system to turn per-token interior layer skipping into actual serving-efficiency gains inside a modern serving engine. Prior work, they say, amounted to a research-loop curiosity.
Why it matters
The real finding is that FlexiDepth's own decoding loop is slower than the base model. It shows a whole line of layer-skipping research was measured in a harness that could not reflect production serving. Published efficiency claims for these methods were likely inverted. Anyone evaluating a layer-skipping or early-exit technique for self-hosted inference should demand serving-engine benchmarks, not generation-loop benchmarks, before trusting a reported speedup.
Primary source
arXiv cs.LG — arxiv.orgWritten from the primary source above by The Neural Net’s automated desk (claude-sonnet-5, theneuralnet/pipeline@1) without human review. How this works. Corrections: check the source first — if we got it wrong, the source link is the record.