llama.cpp v0.6.0 adds new batch API and five decision models
A server endpoint now runs dedicated pipelines for decision-style models.
The facts
- Introduces llama_batch_ext, an extended batch API with llama_process() for mixed token/embedding inputs and MTP/deepstack state embeddings
- Adds support for GLM-5.3-Flash (GLM5-Next), a 320B hybrid text+vision model, and the Clef decision model with text and vision
- Ships a new /v1/systemone server API for five decision models: laya, julia-1, lev, openjev (plus vision), and kev, later extended to nimble
- Adds MTP speculative decoding for Qwen4Exp (the project reports roughly 1.5x decode speedup on DGX Spark), a Metal flash attention kernel for F16 KV, and sparse flash attention for quantized K/V on Vulkan; ggml bumped to v0.26.0
llama.cpp has shipped v0.6.0. The headline change is a new way of feeding data into the engine. The llama_batch_ext API, paired with a llama_process() call, lets a single batch mix raw tokens with embeddings. It can also carry per-token 'state' embeddings. The project needs this for MTP (multi-token prediction) and deepstack-style models. Examples, speculative decoding, multimodal input and the server have all been migrated onto it. Batches can now hold both token and embedding inputs at once.
The release also widens the model roster. GLM-5.3-Flash, also called GLM5-Next, is a 320B-parameter hybrid text-and-vision model. It combines KDA/DSA attention with mixture-of-experts routing. The Clef decision model arrives with full text and vision support. Qwen4Exp gets what the project calls 'high-quality support,' including MTP speculative decoding. llama.cpp says this yields roughly a 1.5x decode speedup on Nvidia's DGX Spark hardware. Ling 3.0 VL has been folded into the existing BailingMoeV3 architecture rather than given its own code path.
On the server side, a new /v1/systemone endpoint runs a dedicated pipeline for five 'decision models': laya, julia-1, lev, openjev (with vision support), and kev. It was later extended to a sixth, nimble. The GET /v1/models endpoint now reports a model's input and output modalities directly. And /v1/embeddings accepts typed vision, audio and video content. The Web UI gets a rebuilt model-download pipeline built on a Hugging Face Hub data layer, with memory-fit estimation before a download starts.
Hardware-side, Metal picks up a tensor-API flash attention kernel for F16 key-value caches. It also gets new few-row matrix-multiply kernels. The project says these run up to 3x faster for speculative and batched decoding on Apple GPUs. Vulkan gains sparse flash attention for quantised key/value tensors. The underlying ggml library moves to v0.26.0. It adds a new buffer allocation API, sparse flash attention on SYCL, Metal and Vulkan, BF16 ops on the CPU backend, and a model-driven W4A4 (NVFP4/MXFP4) matrix-multiply path on CUDA.
Why it matters
The batch API rework matters most for anyone building on llama.cpp directly. Code that constructs batches by hand will need to migrate to llama_batch_ext, since the project has already moved its own examples and server onto it. The decision-model additions and the /v1/systemone endpoint point to llama.cpp broadening beyond chat-style LLMs into structured decision pipelines. That is worth tracking for anyone choosing an inference engine for more than text generation.
Primary source
llama.cpp releases — github.comWritten from the primary source above by The Neural Net’s automated desk (claude-sonnet-5, theneuralnet/pipeline@1) without human review. How this works. Corrections: check the source first — if we got it wrong, the source link is the record.