What are open-weight models?

What does "open weights" mean?

An open-weight model is one whose trained parameters — the numbers learned during training — are published for anyone to download. That is narrower than it sounds. Meta released Llama 2 at 7, 13 and 70 billion parameters, with both pretrained and chat-tuned versions, for research and commercial use (Touvron et al., 2023). Mistral AI released Mixtral 8x7B, a sparse mixture-of-experts model, under the Apache 2.0 licence, with both code and weights public (Jiang et al., 2024).

Open weights are not the same as open source. Releasing weights lets you run and fine-tune a model. It does not by itself tell you what data trained it, in what proportions, or with what filtering. Llama 2's paper describes its training corpus only as "a new mix of publicly available online data" that excludes Meta's own products, without naming the sources.

What's included, and what's left out?

Both papers describe what ships alongside the weights. Meta's Llama 2 release included model cards, a responsible-use guide and code examples for deployment. It also came with a detailed account of its fine-tuning and safety process (Touvron et al., 2023). Mistral released Mixtral's code and contributed changes to the vLLM inference project, so the community could run the model on an open-source stack (Jiang et al., 2024).

What neither paper discloses in full is the raw training data itself. Llama 2 reports total token counts — 2 trillion for the main models — and says data was up-sampled from "the most factual sources", but it gives no dataset list. Mixtral's paper names its architecture and training context length (32,000 tokens) but gives no breakdown of its pretraining corpus either. A reader deciding whether a model is safe to build on should note that weights, not data, are usually what's open.

How do licences differ?

Mixtral's licence is the simplest to describe: Apache 2.0, applied to both the base and instruct models. Mistral says this makes it "free for academic and commercial usage" (Jiang et al., 2024). That is a permissive open-source licence with few restrictions.

Llama 2's terms are narrower. Meta says it is releasing the models "to the general public for research and commercial use", but it withheld a 34-billion-parameter variant entirely. It cited a lack of time to "sufficiently red team" it before release (Touvron et al., 2023). The paper reports the model's weights and benchmark numbers but does not reproduce its full licence text. Anyone planning a commercial deployment should read the actual licence rather than assume it matches Apache 2.0's terms.

How do open models compare with closed ones?

On release, Llama 2 70B came close to GPT-3.5 on MMLU and GSM8K but trailed badly on coding benchmarks. Meta's own paper says there is "still a large gap" versus GPT-4 and PaLM-2-L (Touvron et al., 2023). Llama 2-Chat's human evaluations suggested it was "on par with some of the closed-source models" for helpfulness and safety — Meta's own framing, not an independent one.

Mixtral, five months later, closed much of that gap. Mistral reports Mixtral 8x7B matching or beating Llama 2 70B and GPT-3.5 on most benchmarks, with a wide lead in maths, code and multilingual tasks. It does this while using only 13 billion active parameters per token (Jiang et al., 2024). Mixtral's instruct version scored 8.30 on MT-Bench. Per an independent LMSys leaderboard snapshot from 22 December 2023, it beat GPT-3.5-Turbo, Gemini Pro and Claude-2.1 on the Chatbot Arena Elo rating, making it, at that point, the leaderboard's top open-weight entry.

What does it take to run one yourself?

Running an open-weight model locally still means owning the hardware bill. Meta trained Llama 2 on Nvidia A100 GPUs across 3.3 million GPU-hours in total. Meta says the point of releasing weights publicly is that other companies won't have to repeat that training cost (Touvron et al., 2023). Running inference is far cheaper than training, but larger models still need serious memory. Mixtral's total parameter count is 47 billion, which sets its memory footprint even though only 13 billion parameters are active for any given token (Jiang et al., 2024).

Mixtral's sparse mixture-of-experts design was built partly with self-hosting in mind. A router picks just two of eight expert blocks per token. Mistral says this gives faster inference at small batch sizes and higher throughput at large ones. The team upstreamed support into vLLM, an open-source serving engine, plus Megablocks kernels for efficient expert computation (Jiang et al., 2024). Expert parallelism — splitting experts across GPUs — introduces its own load-balancing problems, since some experts can get more tokens than others.

Meta's 2025 release Muse Spark Open pushes this trade-off much further. It has about 816 billion total parameters, routing each token through 8 of 256 experts per layer for roughly 42 billion active parameters. But all 256 experts must stay resident in memory regardless of how many run per token, so deployment needs an 816B-parameter footprint even though compute cost tracks closer to 42B. The BF16 checkpoint runs to about 1.5 terabytes and needs four nodes of 8 Nvidia H200 GPUs; a 4-bit NVFP4 version fits on a single Nvidia B200 node, per Hugging Face, which hosts the release. A full 1,048,576-token context window demands roughly 266GB of KV cache in BF16, Hugging Face calculates.