Dharma-AI Says Tokenizer Rebuild Cut Inference Cost 77%
Rebuilding a model's tokenizer, not the model, did most of the work
The facts
- Dharma-AI's VocabX method rebuilds a specialized model's vocabulary while keeping the base model, data and training recipe unchanged.
- On a Qwen3-4B Portuguese essay grader, the output layer shrank from ~152,000 to ~65,000 rows and average response length fell from 350 to 101 tokens.
- Measured on 500 requests on an RTX 5090 with vLLM, cost dropped 77.0% while the bench score rose from 0.703 to 0.810.
- Across five public domains and three model families, Dharma-AI reports VocabX beat the strongest of three reference methods (FVT, MWT, in-place BPE) by 1.2 to 9.3 points on compression.
- The company says a generated token cost 7.4 times a read token in its setup because decoding is memory-bandwidth bound, which is why VocabX weights response tokens far more heavily than input tokens.
- Results come from a VocabX paper Dharma-AI says is still under review and may change before publication.
A production system that grades Portuguese essays cut its inference cost by 77%. It runs Qwen3-4B, and the fix was rebuilding its tokenizer, according to Dharma-AI, the company that runs it.
The base model, training data and training recipe stayed identical. Only the vocabulary changed, under a method the company calls VocabX. The approach is detailed in a paper Dharma-AI says is still under review and may change before publication.
The waste, Dharma-AI argues, came from inheriting a general-purpose tokenizer built for many languages and domains. The grader's vocabulary of 151,669 entries forced repeated JSON envelopes and competency names into many tokens each. The model still scored tens of thousands of entries, including Chinese-language tokens, at every generation step it took.
Decoding is memory-bound rather than compute-bound. The company says a generated token in this system cost 7.4 times as much as a token read from the input, so compressing output mattered far more than compressing input. VocabX targets tokens by that per-field cost rather than by raw frequency. It works on both sub-word merges and whole-phrase "macro tokens" under one criterion, and it estimates savings for spans a sampling pass would otherwise miss. Dharma-AI says that last step lifted token reduction on a contracts corpus from 44.7% to 52.8%.
Applied to the grader, the rebuild shrank the LM head's output layer from about 152,000 rows to roughly 65,000. Average response length fell from 350 to 101 tokens. Measured cost on 500 requests dropped 77.0% on an RTX 5090 running vLLM.
The company reports that its bench score, a combined measure of grading accuracy and off-topic screening, actually rose from 0.703 to 0.810, after a correction. An initial design fused each grade into a single macro token with surrounding JSON. That compressed harder but scored only 0.689, because the model could no longer separate grade from structure. Giving the grade its own token restored accuracy and still beat the fused version on cost.
Dharma-AI says it tested VocabX against three published methods, FVT, MWT and Liquid AI's in-place BPE. The test spanned five domains and three model families, Qwen3, Llama-3.1 and Gemma-3. The company reports it compressed more than the best reference method in all fifteen combinations, by 1.2 to 9.3 points.
Why it matters
Anyone fine-tuning a small model for a narrow domain is likely running on a tokenizer sized for a general-purpose model it no longer resembles. If Dharma-AI's numbers hold up under review, vocabulary specialisation looks like a cheap lever to check before reaching for a bigger GPU.
Primary source
Hugging Face — huggingface.coWritten from the primary source above by The Neural Net’s automated desk (claude-sonnet-5, theneuralnet/pipeline@1) without human review. How this works. Corrections: check the source first — if we got it wrong, the source link is the record.