New CUDA kernels speed Llama inference on Hopper GPUs
FP8 prefill and INT4 decode cut latency, at a small cost in accuracy
The facts
- FuTransformer is a CUDA implementation of a Llama-style Transformer block for NVIDIA Hopper GPUs, built by Wei Wang
- On one H100, its FP8 prefill path is up to 1.48x faster than compiled PyTorch FP8 and reaches 1.29 PFLOP/s
- Across 8B and 70B model shapes at batch sizes 1-32, INT4 decode is 1.3-2.4x faster than the better of PyTorch's BF16 and INT4 kernels at each point
- Complete Llama-3.1-8B decode takes 2.8ms per step at batch one, versus 8.1ms for the compiled transformers implementation
- Running every WikiText-2 token through the INT4 weight path raises perplexity from 6.24 to 7.23 with round-to-nearest weights, or 6.99 with GPTQ; the rotated INT4 cache adds a further 0.20
A new paper describes FuTransformer, a hand-tuned CUDA implementation of a Llama-style Transformer block built specifically for NVIDIA's Hopper GPUs. The work, from author Wei Wang, splits inference into its two distinct phases. Prefill reuses weights across many tokens, so the bottleneck is matrix multiplication. Decode generates a single new token per step, so the bottleneck shifts to reading weights and the KV-cache, the stored keys and values from prior tokens.
Prefill runs in FP8 and borrows a GEMM-plus-epilogue approach from a prior system called CODA. It adds direct KV-cache placement, folded RMSNorm normalisation gains, and padding-free batching. On one H100 GPU, the paper reports prefill up to 1.48 times faster than compiled PyTorch FP8, reaching 1.29 PFLOP/s. The FP8 path is 2.02 to 2.18 times faster than the BF16 CODA baseline the authors evaluated.
Decode switches to INT4 for both projection weights and the KV-cache. Weights and keys are laid out to match Tensor Core fragment formats, so they dequantize directly into matrix-multiply operands. A block-diagonal Hadamard transform tames outlier channels that would otherwise blow up quantization error.
The paper reports INT4 decode running 1.3 to 2.4 times faster than the better of PyTorch's BF16 and INT4 kernels. That holds across 8B and 70B model shapes at batch sizes from 1 to 32. In one case PyTorch's own INT4 kernel loses to BF16 at batch 32, because it targets only a single query row. For a complete Llama-3.1-8B decode step, the authors measure 2.8 milliseconds at batch one. The compiled Hugging Face transformers implementation takes 8.1 milliseconds.
That speed carries an accuracy cost. Running all of WikiText-2 through the INT4 weight path raises perplexity from 6.24 to 7.23 with simple round-to-nearest quantization, or to 6.99 using the GPTQ method. The rotated INT4 cache adds a further 0.20. The authors also report their kernels match a quantized reference computation to within 0.001 perplexity, suggesting the losses come from the quantization scheme itself rather than implementation bugs.
Why it matters
For teams self-hosting Llama-family models on Hopper hardware, this closes much of the gap between generic compiled kernels and the peak throughput the silicon allows, nearly tripling decode speed at batch one. The perplexity cost from INT4 weights means the gains are not free. Accuracy-sensitive workloads should weigh that tradeoff before swapping kernels.
Primary source
arXiv cs.AI — arxiv.orgWritten from the primary source above by The Neural Net’s automated desk (claude-sonnet-5, theneuralnet/pipeline@1) without human review. How this works. Corrections: check the source first — if we got it wrong, the source link is the record.