PyTorch 2.15 brings FP8 and much faster LLM decoding to Apple's MPS

Hugging Face's Metal kernels cut video VAE memory from 270 GiB to 8 GiB

Hugging Face's PyTorch team has shipped a set of upgrades to the MPS backend in version 2.15. MPS is the path that lets PyTorch run on Apple silicon GPUs. The headline addition is FP8 support: torch.float8_e4m3fn tensors now run on MPS. They take half the memory of BF16 and multiply directly on the GPU through F.scaled_mm, with either tensorwise or rowwise scaling. The team says that at the start of 2026, running even small models on MPS often meant working around unsupported operations, performance cliffs and unexpected memory growth. This release, it says, is the product of systematically testing real Hugging Face Hub models and fixing gaps upstream.

The most consequential change for anyone running local language models is decoding speed. Hugging Face reports that mixture-of-experts models now decode 5 to 6.5 times faster than under PyTorch 2.14. On an M3 Ultra with torch.compile, Qwen3.6-35B-A3B rises from 9.4 to 61.3 tokens per second. LFM2.5-8B-A1B goes from 23.6 to 123.6 tokens per second. That comes from a native grouped matmul that handles uneven expert routing entirely on the GPU, instead of looping with CPU syncs. It also comes from matrix-vector kernels now wired into F.linear for the single-row inputs that dominate decoding. Dense models benefit too: softmax over a 128,256-token vocabulary, the kind used for sampling, drops from 57 microseconds to 7, an 8x speedup.

Video and image workloads get a separate but striking fix. Moving constant padding from MPSGraph to a dedicated Metal kernel eliminates runaway driver-memory growth during tiled VAE decoding. Wan2.2's video decoder makes 8,764 padding calls across a 28-tile decode. It now peaks at about 8 GiB of driver memory, down from 270 GiB. The same change also fixes a correctness bug in Qwen-Image-2.1's VAE encoder, a problem previously reported in ComfyUI that had been silently corrupting images on MPS. An encode-decode round trip now matches CPU output at 29.95 dB PSNR, up from 10.06 dB. The release also adds complex-valued linear algebra, faster small-matrix inversion and QR decomposition, and public MPS stream control from Python. It raises the minimum requirements to macOS 15 and Python 3.11.

Hugging Face previews PyTorch 2.16 work already landing in nightlies. That includes committing GPU command buffers every 16 kernels, so the GPU starts work before the CPU finishes encoding the rest. The change pushes LFM2.5-8B-A1B to 126.0 tokens per second and Qwen3.6-35B-A3B to 62.8, further gains on top of 2.15.

Written from the primary source above by The Neural Net’s automated desk (claude-sonnet-5, theneuralnet/pipeline@1) without human review. How this works. Corrections: check the source first — if we got it wrong, the source link is the record.