What is tokenization in NLP?
Models read pieces of words, not letters or whole words, and that choice still shapes cost and quality.
What is tokenization?
Tokenization is the step that turns raw text into the discrete units, called tokens, that a neural network actually reads. Rather than feeding a model individual characters or whole words, most systems split text into subword units. These are pieces smaller than a word but often larger than a letter.
The Sennrich, Haddow and Birch paper (2015) frames this as a solution to an open-vocabulary problem. A translation model has a fixed vocabulary, typically 30,000 to 50,000 words at the time they wrote. But language keeps generating new words through compounding, inflection and borrowing. Subword units let a fixed vocabulary still represent any input.
Why don't models just read characters or whole words?
Whole-word vocabularies break on anything they have not seen. Sennrich et al. (2015) note that word-level models cannot translate or generate unseen words at all, and can only fall back to copying them. That works for names shared across alphabets, but fails once alphabets differ or a word needs real translation.
Characters avoid the unseen-word problem entirely but create a different one: sequence length. The paper's own statistics show the German training corpus needs about 550 million character tokens versus 100 million word tokens. Encoder-decoder networks scale at least linearly with sequence length, so that is a real cost. Subword units sit between these extremes, giving a compact, open vocabulary without exploding sequence length.
How does byte-pair encoding build a vocabulary?
Byte-pair encoding, or BPE, adapts a data-compression algorithm from Gage (1994) for word segmentation. Sennrich et al. (2015) start with each word split into characters plus an end-of-word symbol. They then repeatedly find the most frequent adjacent pair of symbols in the training text and merge it into one new symbol. Each merge is a single hyperparameter choice; the final vocabulary size equals the starting character inventory plus the number of merges performed.
Frequent character sequences, and eventually whole frequent words, collapse into single symbols. Rare words stay split into smaller pieces. On their German data, 59,500 merges produced a network vocabulary of around 63,000 symbols with zero unknown tokens at test time, against 1.75 million distinct word types in the unsegmented corpus. The paper reports BPE improved German and Russian translation by up to 1.1 and 1.3 BLEU over a back-off dictionary baseline on WMT 2015 test sets.
What does SentencePiece add on top of BPE?
SentencePiece, described by Kudo and Richardson (2018), is a tokenizer and detokenizer that trains subword models directly from raw sentences. It does not need text that has already been split into words by a language-specific pre-tokenizer. It implements both BPE and a separate unigram language model algorithm, and treats whitespace itself as an ordinary symbol, escaping it with a meta character so the original text can be reconstructed exactly.
That last point, which the paper calls lossless tokenization, matters because conventional pre-tokenized output cannot always be detokenized without ambiguity. A tool has to guess, language by language, where spaces belong. Kudo and Richardson also report an engineering gain: on Japanese text, SentencePiece's segmentation ran about 380 times faster than the subword-nmt implementation of BPE, because it uses an algorithm of roughly N log N cost instead of the naive quadratic scan.
Why do token counts differ across languages?
Pre-tokenization by whitespace assumes words are already marked off. That holds for English but not for Japanese, Chinese or Korean, languages with no spaces between words. Kudo and Richardson (2018) tested English-Japanese translation on the Kyoto Free Translation Task and found pre-tokenization unnecessary for BLEU. In the English-to-Japanese direction, BLEU actually dropped when pre-tokenization was applied, from 21.62 to 20.86.
They attribute the larger gains from SentencePiece specifically to Japanese. As a non-segmented language, forcing a fixed external tokenizer on it constrains the vocabulary the model can learn. Training the subword model straight from raw characters finds a better, more domain-specific segmentation instead. This is one documented reason the same sentence can tokenize into very different counts depending on the target language and script.
What are the trade-offs and limits?
Vocabulary size is a knob, not a solved problem. Sennrich et al. (2015) call their own choice of vocabulary size "somewhat arbitrary" and flag learning an optimal size automatically as future work. They also found that a smaller, well-chosen subword vocabulary can outperform a larger word vocabulary. Subword units occur more often in training data, so they are less data-sparse.
Segmentation can also be inconsistent between languages when BPE is learned separately for source and target. The paper traces specific transliteration errors, such as a Russian rendering of "rakfisk" gaining a spurious letter, to training examples where the English and Russian sides were split differently. Learning a joint BPE encoding over both vocabularies reduced, but did not eliminate, this problem.
Key terms
- Token
- A discrete unit of text, often a subword piece, that a neural network reads or generates one at a time.
- Byte-pair encoding (BPE)
- A word-segmentation algorithm, adapted from a data-compression method by Gage (1994), that repeatedly merges the most frequent adjacent pair of symbols in a training corpus to build a fixed-size subword vocabulary.
- Subword unit
- A sequence of characters shorter than a full word, used so that rare or unseen words can still be represented by combining known pieces.
- Out-of-vocabulary (OOV) word
- A word that does not appear anywhere in the training text, which a word-level model cannot translate or generate directly.
- SentencePiece
- A language-independent subword tokenizer and detokenizer, described by Kudo and Richardson (2018), that trains directly from raw sentences and supports both BPE and a unigram language model algorithm.
- Lossless tokenization
- SentencePiece's design in which decoding a tokenized sequence exactly reproduces the normalized input text, including whitespace, with no information lost.
- Unigram language model (subword algorithm)
- A subword segmentation algorithm, proposed by Kudo (2018) and implemented in SentencePiece, that trains and segments with cost linear in the size of the input data.
Written from
Tokenization in the news
llama.cpp adds support for K2 Horizon dense and MoVA models
Release b11454 brings a new converter, tokenizer and chat template fixes for the K2 Horizon model family.
Study: French costs up to 58% more tokens than English across 7 LLMs
A paper measures a persistent "language tax" in 2026 tokenizers, and tests a French-optimised alternative to Mistral's Tekken.
Written from the documents above by The Neural Net’s automated desk (claude-sonnet-5, theneuralnet/guides@1) without human review, and updated as news on this topic publishes. How this works.