Hugging Face rewrote tokenizers to stop the CPU starving the GPU
Tokenisation was once rounding error. It is not any more.
The facts
- Version 1 of the tokenizers library is a performance rewrite, which the maintainers say is often tens of times faster than v0.23.
- The regex-based pre-token split is replaced by Boolean operations over bitstreams using SIMD instructions.
- The merge working set moves to a caller-owned scratch buffer, so the merge loop never touches the allocator.
- Hugging Face credits rival tokenizer projects, including gigatoken, tiktoken and kitoken, with demonstrating several of the ideas it adopted.
Hugging Face has published the engineering behind version 1 of its tokenizers library. The team says the result is often tens of times faster than version 0.23. The framing is a specific failure: tokenisation putting enough pressure on the CPU that it starves the GPU of data.
The changes are structural. The single crate becomes a workspace, so tk-encode is the only required runtime dependency. Serialisation, conversion and training code sit in separate crates, linked only when an application needs them. The merge working set moves into a caller-owned scratch buffer, keeping the allocator out of the hot loop. Pieces being merged sit in an intrusive doubly-linked list inside one preallocated buffer, so a merge updates two indices rather than moving data.
The splitting stage was rewritten outright. Byte-pair encoding uses a regular expression to break input into pre-tokens, within which merges happen and across whose boundaries they never do. That split now runs as Boolean operations over bitstreams using SIMD instructions. A thread-local cache means a repeated word is merged once and then looked up. Hugging Face credits rival projects, among them gigatoken, tiktoken and kitoken, with showing which of these ideas were worth trying.
Concurrency was handled separately. One shared tokenizer can now encode from many threads at once, each drawing its scratch buffer and word cache from its own sub-pool, so threads no longer queue on a single lock. Of the ten model families measured, eight use byte-pair encoding; the other two use WordPiece and Unigram.
Why it matters
This is a useful marker of where inference optimisation has reached. Tokenisation was ignored for years because it was rounding error next to the forward pass. But it becomes worth a SIMD rewrite only once everything downstream is optimised hard enough that the CPU leaves the accelerator idle. Measure the tokenizer before buying more GPU.
Primary source
Hugging Face — huggingface.coWritten from the primary source above by The Neural Net’s automated desk (claude-opus-5, manual-seed@2026-09-24 (voice: economist)) without human review. How this works. Corrections: check the source first — if we got it wrong, the source link is the record.