llama.cpp v0.7.0 adds MoE GPU cache, new default server port

A new expert cache and port change reshape how MoE models get served

Llama.cpp's v0.7.0 release adds a GPU cache for mixture-of-experts (MoE) models. It keeps unused experts in host memory and pulls them onto the GPU only when needed, via a new flag called --moe-cache-mib.

The technique uses a least-recently-used cache shared across multiple GPUs. The project says it delivers up to 2.2x faster decode when experts are offloaded this way, letting larger MoE models run on hardware that cannot hold every expert in VRAM at once.

The release also adds a speculative decoding technique called LiliCorr. It uses a small draft transformer layered on top of an existing method called DFlash to pick candidate tokens. It also adds multi-token prediction (MTP) support for GLM5-Next with an optimised sparse attention path.

A long list of new models lands alongside these. K2 Horizon arrives in dense and MoVA variants, alongside the ternary Prism Bonsai 2 27B and a text-vision-audio embedding model called EmbeddingGemma-2. There are also several new 'decision' models from LiquidAI, vLLM and pplx-decider, plus MiniCPM-V 4.7, Nemotron Super 3.5 VL, Nemotron Bolt 74B A9B and a maion-coder architecture.

Embedding models get a separate change that skips the logits buffer and the LM head entirely when pooling is enabled. The project says this can cut VRAM use by up to 10x.

On the server side, the chat layer is rebuilt around a new common_chat_session API with feed() and finish() calls. Formatted chat output now comes back as an array of parts rather than a single string, a breaking change for anyone parsing responses directly.

The default listening port moves from 8080 to 9931. Docker images keep 8080 inside the container, though, to avoid breaking existing compose files.

Prompt-cache handling is also reworked so only restorable slot states get saved. Rerank and embedding requests no longer pollute the cache as a result, and the web UI gains a full models manager with search, filters, favorites and resumable downloads.

The underlying ggml library moves to v0.27.0. This bumps the RPC protocol to v8 for tensor-parallel RPC servers and adds backend-specific kernel work across CUDA, Vulkan, SYCL, OpenCL, Metal and Hexagon.

Written from the primary source above by The Neural Net’s automated desk (claude-sonnet-5, theneuralnet/pipeline@1) without human review. How this works. Corrections: check the source first — if we got it wrong, the source link is the record.