Transformers v5.20.0 adds Meta's 816B-parameter Muse Spark Open

Seven new model families land alongside breaking changes to cache and config APIs

Hugging Face has shipped Transformers v5.20.0. It adds native support for seven model families, plus a run of breaking changes to the library's cache and configuration internals.

The headline addition is Muse Spark Open, a multimodal mixture-of-experts model from Meta. It takes text, images, audio and video as input and generates text. It carries 816 billion total parameters with 42 billion active per token, and a context window of up to 1,048,576 tokens shared across modalities. Meta trained it for agentic coding work such as multi-file editing, debugging and tool use. Checkpoints ship in BF16 and NVFP4 formats.

The rest of the additions span a wide range of uses. Ling 3.0 Flash from InclusionAI is a hybrid mixture-of-experts language model that groups five Kimi Delta Attention layers with one Multi-head Latent Attention layer; it now loads natively without trust_remote_code=True. Aleph Alpha's Kolibri 1 is a German-and-English reasoning model that routes each token to 6 of 384 experts via sigmoid routing across 50 layers. OpenBMB's MiniCPM-V 4.7 adds canvas M-RoPE, laying visual tokens on a 2-D canvas so image and video position is preserved. Baidu's UnlimitedOcr is a 3B-parameter OCR model built on DeepSeek OCR that replaces decoder attention with Reference Sliding Window Attention, keeping the KV cache constant during decoding. UltraBERT brings a family of bidirectional text encoders, at 150M and 380M parameters plus an embedding variant, supporting sequences up to 32,768 tokens. MOSS-Transcribe-Diarize is a 0.9B end-to-end audio model that produces joint transcription and speaker diarization in a single pass.

The release carries five flagged breaking changes. Deprecated methods in cache_utils.py have been removed, affecting models including minimax_m3_vl and pi0. Callers must switch to the current cache API. Accidental cross-model imports were stripped from files such as embedding_gemma2, ijepa and lightglue. Code that relied on those indirect imports will need to import from the original modules instead. The minimum kernels dependency is now 0.17.2, enabling a fallback mode so a failed kernel load no longer raises an error outright. A new SubConfigSpec changes how sub-configs are initialised across many model configs, which may require updates to custom nested configs. And the LUKE and mLUKE tokenizers now treat entity tokens like <ent> as proper special tokens. That can change tokenization and decoding output for anyone depending on the old behaviour. The release also flags an unresolved regression: inference with expert parallelism is currently slower, a fix the team says is still in progress.

Written from the primary source above by The Neural Net’s automated desk (claude-sonnet-5, theneuralnet/pipeline@1) without human review. How this works. Corrections: check the source first — if we got it wrong, the source link is the record.