Tokenization
Tokenization is the process of breaking text into smaller units, called tokens, that a language model can process — typically words, subwords, or characters.
Definition
Tokenization is the process of breaking text into smaller units, called tokens, that a language model can process — typically words, subwords, or characters.
Overview
Before a language model can process text, the raw string must be converted into a numeric format the model can operate on. Tokenization is the first step in this pipeline: it splits input text into discrete units, and each unique token is mapped to an integer ID from the model's vocabulary. Most modern large language models use subword tokenization algorithms, such as byte-pair encoding (BPE), which split text into pieces smaller than whole words for rare terms while keeping common words as single tokens — this balances vocabulary size against the model's ability to represent any possible input, including unusual words, code, and multiple languages. Once tokenized, each token ID is converted into a numeric embedding vector, which is then processed by the model's neural network layers, including its attention mechanism layers. The number of tokens a model can process at once is bounded by its context window, and providers typically price API usage per token, making tokenization directly relevant to both technical performance and cost. Tokenization choices affect a model's efficiency and behavior in subtle ways — languages with less common vocabulary, code with unusual syntax, and numbers can all be tokenized less efficiently than common English text, which is one reason token counts can vary meaningfully across languages for equivalent content.
Key Concepts
- Converts raw text into discrete units (tokens) a model can process
- Most modern LLMs use subword tokenization like byte-pair encoding
- Each token maps to a numeric ID from the model's fixed vocabulary
- Token counts determine context window usage and API pricing
- Balances vocabulary size against ability to represent any input text
- Efficiency varies across languages, code, and numeric content