Benchmark finds LLM-generated GPU kernels fail in production

FastKernels shows kernel-level speedups mostly vanish once agents' code runs inside real models.

A new benchmark called FastKernels argues that LLM-based GPU kernel generation has been grading itself on the wrong test. It comes from a team including authors at Carnegie Mellon and researchers linked to SGLang and DeepSpeed. Most existing benchmarks score generated kernels in isolation, against synthetic inputs and weak baselines.

The paper's authors built 384 tasks instead. These are drawn from 47 representative architectures across 8 categories. The kernels are sufficient to reimplement 94.6% of HuggingFace Transformers architectures — 472 of 499 — while matching native outputs exactly.

The gap between isolated and production scoring is the paper's real finding. Candidates are scored both at the kernel level and end to end, inside the models they come from. This runs on the actual execution path production frameworks use.

The authors seeded the leaderboard with five agents run for a combined 6,900 agent-hours. Kernel-level speedups of 1.6-6.6x shrink to at most 1.25x once measured end to end, they report. Only 20% of the kernel sets that won in isolation actually run correctly as-is when dropped into a real model.

The mis-ranking is the sharper result. Claude Code matched or beat a baseline called KDA at every level when kernels were tested in isolation. Yet KDA scored three times higher once both were evaluated end to end inside production models.

The authors' aggregate metric, MacroEval, combines calibrated correctness, coverage and speedup into a single leaderboard score. It is meant to resist this kind of inversion.

Written from the primary source above by The Neural Net’s automated desk (claude-sonnet-5, theneuralnet/pipeline@1) without human review. How this works. Corrections: check the source first — if we got it wrong, the source link is the record.