Transformers v5.19.0 adds EmbeddingGemma 2, ships breaking MoE changes
Hugging Face's release adds Google's multimodal embedding model and changes how MoE routers and continuous batching behave.
The facts
- EmbeddingGemma 2 is a multimodal embedding model from Google built on the Gemma 4 architecture, encoding text, image, audio and video into a shared 768-dimensional vector space.
- It uses Matryoshka Representation Learning, letting embeddings be truncated to 512, 256 or 128 dimensions, and unused vision or audio towers can be disabled at load time to save memory.
- All MoE models that compute router logits now return them when output_router_logits=True, following the Qwen3-MoE pattern, which can change output classes for existing code.
- The "paged|" prefix for SDPA and flash attention is deprecated; regular attention implementations now support continuous batching directly, and expert parallelism gains a token-dispatch implementation that removes the requirement that EP size equal TP size.
Hugging Face has shipped Transformers v5.19.0. The headline addition is EmbeddingGemma 2, a multimodal embedding model from Google built on the Gemma 4 architecture. It encodes text, images, audio and video, alone or combined in a single input, into a shared 768-dimensional vector space for retrieval, semantic similarity, clustering and classification. Matryoshka Representation Learning lets developers truncate those embeddings to 512, 256 or 128 dimensions. Unused vision or audio towers can also be switched off at load time to save memory.
The release carries five changes marked as breaking. Every mixture-of-experts (MoE) model that computes router logits now returns them when output_router_logits=True. This follows the pattern already used by Qwen3-MoE. Code built around the old, often empty, outputs needs updating. Owlv2ForObjectDetection now picks its image query box by objectness score rather than the old OWL-ViT heuristic, which can change detections from earlier releases.
The library is also deprecating the "paged|" prefix on SDPA and flash attention implementations. Regular sdpa and flash_attention_2 now support continuous batching directly. Continuous-batching callers should therefore drop the prefix, though eager attention still needs "paged|eager". A related change fuses the cache-update call for continuous batching's index-based and block-table paths into one. That alters the function's behaviour for any custom code that still calls the separate paths.
On parallelism, expert parallelism gets a token-dispatch implementation, selected via a new ep_dispatch_experts plan rule. It is now the default for Qwen3 MoE and Mellum. Hugging Face says this removes the requirement that expert-parallel size equal tensor-parallel size. The Trainer has been adapted to work with expert parallelism as a result. The release also fixes quantized cache handling: generate no longer mutates a user's cache_config, and QuantizedLayer.reorder_cache is repaired. It adds per-layer cache configuration too, so DynamicCache and StaticCache can initialise each layer from its own settings. That is useful for models that mix sliding-window and full attention.
Why it matters
Three of the breaking changes fail silently rather than throwing errors. Anyone with custom code reading MoE router outputs, continuous-batching code using the "paged|" prefix, or OWLv2 pipelines should check behaviour before upgrading. The EP/TP decoupling matters more for teams scaling MoE training. It removes a sizing constraint that previously forced expert and tensor parallelism to match.
Primary source
Transformers releases — github.comWritten from the primary source above by The Neural Net’s automated desk (claude-sonnet-5, theneuralnet/pipeline@1) without human review. How this works. Corrections: check the source first — if we got it wrong, the source link is the record.