Inference and serving: how LLMs answer at scale
The gap between a demo and a product is what happens to memory and GPUs under load.
What happens when a model answers?
Inference is the process of running a trained model to produce an answer, as opposed to training it. For a large language model, answering a prompt means generating tokens one at a time. Each one is chosen by looking at every token that came before it (Kwon et al., 2023, the vLLM paper).
That sequential habit is the whole problem. Each new token depends on all previous ones, specifically on cached key and value vectors from the attention mechanism. So generation cannot be parallelised the way training can. The vLLM paper notes this makes the workload memory-bound: it leans on GPU memory bandwidth far more than on raw compute, which leaves expensive accelerators underused.
Prefill versus decode: why are they different?
The vLLM paper splits a request into two phases. The prompt phase, often called prefill, takes the entire input prompt at once and computes the probability of the first output token, generating key and value vectors for every prompt token in the process. Because all prompt tokens are known upfront, this step runs as matrix-matrix multiplication and uses GPU parallelism efficiently.
The autoregressive generation phase, or decode, then produces the remaining tokens one by one. Each step takes a single new token as input and reuses the key and value vectors cached from every prior step. It computes only the next token's. This step relies on matrix-vector multiplication, which the vLLM paper describes as less efficient. It accounts for most of a request's latency, because the steps cannot run in parallel.
Why does batching matter, and what makes it hard?
Batching, or processing several requests together, improves GPU use because the cost of moving model weights is shared across them, the vLLM paper explains. But naive batching is awkward for two reasons. Requests arrive at different times, so waiting for a full batch delays early ones. And requests have wildly different prompt and output lengths, so padding them to match wastes both compute and memory.
Fine-grained approaches such as iteration-level scheduling, cited in the vLLM paper as used by the Orca system (Yu et al., 2022), batch at the level of a single decoding step rather than a whole request. Finished requests leave the batch and new ones join after each iteration, without waiting for the entire batch to complete. Specialised GPU kernels remove the need for padding.
What limits batch size: the KV cache problem
Even with fine-grained batching, the number of requests a server can handle together is capped by GPU memory. Mostly this means the key-value cache, or KV cache, that holds each request's attention history. The vLLM paper gives a concrete figure: a single token's KV cache for a 13-billion-parameter OPT model takes 800 kilobytes, and a full 2,048-token request can need 1.6 gigabytes.
Systems that store each request's KV cache in one contiguous memory block waste much of it, the vLLM paper found. Pre-allocating for a request's maximum possible length left only 20.4% to 38.2% of reserved memory actually holding token data in the systems it tested. PagedAttention addresses this by splitting the KV cache into fixed-size blocks that need not sit next to each other in memory, borrowing the page-table idea from operating systems. Built into the vLLM engine, this nearly eliminates that waste and lets the KV cache be shared across sequences from the same request, such as parallel samples or beam-search candidates.
Latency versus throughput: what's the trade-off?
Latency is how long one request takes to answer; throughput is how many requests a server completes per unit of time. The two pull in opposite directions. Larger batches raise throughput by keeping the GPU busier, but can slow any individual request's turn in the queue. The vLLM paper reports that its system raised throughput by 2 to 4 times over FasterTransformer and Orca while holding the same level of latency. The gain grew larger for longer sequences, bigger models, and more complex decoding methods such as beam search.
A separate inefficiency sits inside the attention computation itself. The FlashAttention paper (Dao et al., 2022) found that standard attention implementations read and write a full sequence-length-squared matrix to the GPU's high-bandwidth memory, the slower of its two memory tiers. This dominates runtime for memory-bound operations like softmax. FlashAttention restructures the computation to tile it through fast on-chip memory instead, cutting those memory accesses without changing the result. It reports up to 3 times faster attention for sequence lengths up to 2,048 on an A100 GPU.
What do serving engines actually do?
A serving engine is the software layer that takes incoming requests, batches them, manages GPU memory, and runs the model's forward passes efficiently. vLLM, described in the PagedAttention paper, pairs a centralised scheduler with a KV-cache manager that allocates and frees fixed-size memory blocks on demand. It uses a first-come-first-served policy and preempts whole sequences together when memory runs short.
FlashAttention operates one level down, as a GPU kernel that any engine's attention layer can use; the vLLM paper's own prefill step uses a conventional self-attention algorithm of this kind. Both papers point to the same lesson. GPU memory management, not raw compute, usually decides how many requests a given set of accelerators can serve at once.
Key terms
- KV cache
- The cached key and value vectors from a sequence's earlier tokens, stored so the attention mechanism does not recompute them for every new token.
- Prefill
- The phase of generation that processes the full input prompt at once, computing the first output token's probability in parallel across all prompt tokens.
- Decode
- The autoregressive phase that generates each remaining output token one at a time, dependent on all previous tokens in the sequence.
- PagedAttention
- An attention algorithm that stores a sequence's key-value cache in non-contiguous, fixed-size memory blocks, inspired by operating-system virtual memory paging.
- Iteration-level scheduling
- A batching technique that adds and removes requests from a batch after every decoding step rather than waiting for the whole batch to finish.
- HBM (high-bandwidth memory)
- The larger, slower tier of GPU memory, as distinct from the smaller, much faster on-chip SRAM.
- Throughput
- The number of requests or tokens a serving system completes per unit of time, as distinct from the latency of any single request.
Latest developments
- llama.cpp b11716 adds Q1_0/Q2_0 ultra-low-bit quantisation support for Qualcomm Hexagon NPUs, easing on-device inference.
- llama.cpp release b11696 fixed a flash-attention sink bug in HMX kernels on Qualcomm Hexagon NPU hardware (PR #30581).
- IBM's Granite Switch 4.1 runs chat, decision-making and speech adapters in one endpoint, sharing the base model's KV cache to avoid extra VRAM.
- llama.cpp v0.7.0 adds a GPU cache (--moe-cache-mib) that pulls MoE experts from host memory to GPU, giving up to 2.2x faster decode.
- llama.cpp release b11645 adds a BF16 flash attention kernel for its Metal backend, used for inference on Apple Silicon GPUs.
- AWS detailed three vLLM kernel/routing fixes for MoE models on Hopper GPUs, upstreamed to vLLM, giving up to 10% higher throughput and 9% faster first-token latency.
- llama.cpp's b11588 pre-release (13 Oct 2026) lets Qwen3.8 Flash Next accept image embeddings for multi-token prediction, a narrow serving-engine compatibility fix.
Written from
Inference and serving in the news
llama.cpp adds Q1_0, Q2_0 quant support for Hexagon NPUs
Snapdragon devices get another notch of compression for on-device inference
llama.cpp adds in-app model discovery and download resume
A built-in Hub browser, resumable downloads, and an HTML sanitization fix
llama.cpp patches Hexagon NPU attention bug in b11696
A single fix for Qualcomm's HMX flash-attention sinks ships in the latest build
llama.cpp b11690 trims memory copies in speculative decoding cache
A single-line fix, but it is the kind that adds up at scale
Meta details how it made AMD GPUs production-ready for training
Porting recommendation models from Nvidia to AMD took more than a driver swap
IBM packs chat, decisions and speech into one Granite endpoint
Granite Switch 4.1 adds adapters, not GPUs, to cover three task types
llama.cpp v0.7.0 adds MoE GPU cache, new default server port
A new expert cache and port change reshape how MoE models get served
llama.cpp adds BF16 flash attention kernel for Metal backend
A small patch note for a release that still runs on every Apple GPU
Study: signed LLM weights don't verify the quantized files you run
Across 198 popular GGUF repos, zero could be independently re-derived
AWS details MoE kernel fixes for SageMaker, upstreams them to vLLM
Three upstreamed vLLM fixes squeeze more throughput from Hopper GPUs
llama.cpp adds image-embedding support for Qwen3.8 Flash Next
A single pull request, the release notes offer little else
llama.cpp adds runtime support for Prism Bonsai 2 27B
A pre-release build lets the open-source engine load the Hadamard-based model
llama.cpp patches server bug that corrupted busy slot's context
A slot-pinning bug could silently swap context mid-generation
Asana cuts browser-agent model costs 76x using GPT-6.1 Sol
GPT-6 Astra redesigned caching in Codex. The per-run cost of Asana's StackAI agent fell from $36 to 47 cents.
Paper: KV-cache tool gives LLMs a 50M-token memory on disk
galahad-kv reloads stored key-value state instead of recomputing it, cutting cost on one H100.
Quantized GLM-5.3-Flash lands on Hugging Face for DGX Spark
A community NVFP4 quantization of Zai's GLM-5.3-Flash is tuned to run on Nvidia's GB10-based DGX Spark desktop box.
AWS shows agents paying per API call via Bedrock AgentCore payments
Incarna cut x402 payment integration from months to days, AWS says, using a managed wallet and spending-limit service with inference router BlockRun.
llama.cpp tunes CUDA kernel for Gated DeltaNet state updates
Release b11494 defaults GDN state updates to four columns per warp, a CUDA scheduling change in PR #30087.
llama.cpp adds support for LiquidAI's d1-3B decision model
The b11483 release renamed internally to lfm2-d1 brings a specialised "decision" model type to the inference engine.
Quantization's effect on agent tool-failure recovery flips with eval design
A single-author study finds 8-bit vs 4-bit comparisons reverse depending on prompt, task set and parsing strictness.
Written from the documents above by The Neural Net’s automated desk (claude-sonnet-5, theneuralnet/guides@1) without human review, and updated as news on this topic publishes. How this works.