A token is the basic unit of text that a language model processes: a word, part of a word, punctuation mark, character or byte sequence that the model’s tokenizer maps to a fixed integer ID from its vocabulary. In language models, every prompt is converted into a sequence of tokens before the model sees it, and every answer is generated one token at a time.
Tokens are also the unit of measurement for almost everything practical about an LLM: the size of the context window, generation speed, rate limits and usage pricing are all counted in tokens rather than in words or characters.
How text becomes tokens
The conversion from text to tokens is called tokenization, and it is performed by a tokenizer trained separately from the model. Most current tokenizers use subword algorithms such as byte pair encoding, WordPiece or Unigram. They keep frequent words whole and split rare words into reusable pieces, so the word “unbelievably” might become “un”, “believ” and “ably”.
Each token ID is then looked up in an embedding table, which turns it into a vector of a few hundred to several thousand numbers. From that point the model works only with vectors; the original characters are gone. On output, the model produces a probability distribution over the whole vocabulary, one token is selected, and the tokenizer decodes the ID back into text.
Different granularities have been used over time. The table compares them.
| Granularity | Vocabulary size | Sequence length | Main drawback |
|---|---|---|---|
| Word-level | Very large, open-ended | Short | Unknown words cannot be represented |
| Character-level | Small, under a few hundred | Very long | Slow and costly to process |
| Subword | About 30,000 to 260,000 | Moderate | Splits can look arbitrary |
| Byte-level | 256 base symbols plus merges | Moderate to long | Non-Latin scripts use more tokens |
Typical token counts and ratios
There is no fixed conversion between words and tokens; it depends on the tokenizer, the language and the kind of text. For ordinary English prose with modern subword tokenizers, a common rule of thumb is that one token averages about four characters, or roughly 0.75 words per token.
| Content | Relative token count | Reason |
|---|---|---|
| Common English prose | Lowest, about 1.3 tokens per word | Frequent words are single tokens |
| Source code | Higher | Symbols, indentation and identifiers split |
| Numbers and dates | Higher than expected | Digits often split into groups |
| Non-Latin scripts | Often 1.5 to 3 times English | Fewer merges learned for them |
| Rare names and jargon | Several tokens each | Absent from vocabulary as whole words |
Because counts vary, exact budgets must be measured with the model’s own tokenizer. Two models from different families can produce noticeably different counts for the same paragraph.
Why tokens govern cost, speed and limits
Tokens determine the practical envelope of every language model deployment. The context window is the maximum number of tokens the model can attend to at once, including the prompt, any retrieved documents, the conversation history and the generated answer.
| Concern | How tokens apply | Practical effect |
|---|---|---|
| Context window | Input plus output must fit | Long documents need chunking or truncation |
| Pricing | Billed per input and output token | Output tokens usually cost more |
| Latency | Output generated token by token | Long answers take proportionally longer |
| Rate limits | Tokens per minute quotas | Large batches must be throttled |
| Attention cost | Grows with sequence length | Very long prompts slow processing |
Generation speed is usually reported in tokens per second, and the delay before the first output is called time to first token. Both depend on model size, hardware and prompt length.
Tokens compared with related concepts
A token is the unit, while tokenization is the process that produces it and the vocabulary is the fixed set of all possible tokens. A token is also distinct from an embedding, which is the numerical vector the model assigns to each token ID.
Settings such as temperature act on tokens directly: they reshape the probability distribution from which the next token is drawn. A hallucination is likewise produced token by token, since the model chooses each plausible next token without checking facts. In multimodal models, images and audio are also cut into patch or frame tokens, so the concept extends beyond text.
Tokenizers also add special tokens that never appear as visible text, such as markers for the beginning of a sequence, the end of a turn or the boundary between system and user messages. They consume context space like any other token.
Common pitfalls with tokens
Many surprising model behaviors trace back to tokenization. Knowing them helps practitioners write better prompts and budget correctly.
- Letter-level tasks are hard: counting letters or reversing a word fails because the model sees subword pieces, not characters.
- Arithmetic is fragile: long numbers may split into inconsistent digit groups.
- Leading spaces matter: ” cat” and “cat” are often different tokens with different IDs.
- Word-count estimates mislead: budgets based on words can overflow the context window.
- Truncation is silent: some systems drop the oldest tokens without warning when the limit is reached.
The safest habit is to count tokens programmatically before sending a request and to reserve enough room for the expected answer.
Explained without jargon
A language model does not read whole sentences the way people do. It first chops text into small pieces called tokens. A short, common word is often one piece, while a long or unusual word is split into several. Tokens are the building blocks the model reads and writes.
Think of tokens as the coins the model spends. Everything has a price in tokens: your question, the documents you attach and the answer it writes. The model can only hold a certain number of tokens in mind at once, and services usually charge by the token, so shorter and clearer input saves both space and money.
A typical use case
A product team builds a support assistant on a model with a 128,000-token context window. Each request includes a 1,500-token system prompt, up to 6,000 tokens of retrieved help articles, the conversation so far and room for the answer.
After launch, long conversations start failing in Japanese, although English chats of similar length work. Measuring with the model’s tokenizer shows that Japanese text uses roughly twice as many tokens per message, so the history grows much faster. The team adds a rule that summarizes older turns once the history passes 20,000 tokens and caps answers at 800 output tokens.
The changes eliminate context overflow errors and reduce monthly token spend, because the model no longer receives the full history on every turn.