Visual tokens, also called image tokens, are the discrete or continuous units into which a model converts an image or video frame so that a transformer can process visual content in the same sequence as text. In multimodal AI, they are the bridge between pixels and the language model: each token usually represents one small image patch or a learned summary of several patches.
The number of visual tokens per image is one of the most consequential design choices in artificial intelligence systems that see. It sets the balance between visual detail and compute cost, because attention cost grows with sequence length, and it determines how much of the context window an image consumes.
How images become visual tokens
An image becomes a sequence of visual tokens through patch splitting, encoding and projection. A vision encoder, typically a Vision Transformer, cuts the image into a grid of square patches, often 14 or 16 pixels per side, flattens each patch and maps it to a vector called a patch embedding.
The encoder then refines these embeddings with self-attention across all patches, so that each vector carries information about its neighborhood and the whole scene. A multimodal projector, usually a small multilayer perceptron or a resampling module, maps the result into the embedding space of the language model. Those projected vectors are the visual tokens the language model actually reads.
The patch grid explains the basic arithmetic. The following table shows how input resolution and patch size determine the raw token count before any compression.
| Input resolution | Patch size | Patch grid | Visual tokens |
|---|---|---|---|
| 224 x 224 px | 16 px | 14 x 14 | 196 |
| 224 x 224 px | 14 px | 16 x 16 | 256 |
| 336 x 336 px | 14 px | 24 x 24 | 576 |
| 448 x 448 px | 14 px | 32 x 32 | 1,024 |
Because the count grows with the square of the resolution, doubling the side length of an image roughly quadruples the token count. This quadratic growth is the central engineering problem behind every design choice discussed below.
Continuous and discrete visual tokens
Visual tokens come in two families: continuous embeddings and discrete codebook indices. Continuous tokens are real-valued vectors produced by an encoder and projector; they preserve fine detail and suit models that only need to understand images, such as most vision-language models.
Discrete tokens come from an image tokenizer such as a VQ-VAE or VQGAN, which compresses each region into the index of the nearest entry in a learned codebook of typically 8,192 to 16,384 codes. Discrete tokens let one model predict images the same way it predicts words, which is the basis of native multimodality and any-to-any models.
| Property | Continuous tokens | Discrete tokens |
|---|---|---|
| Representation | Real-valued vectors | Integer codebook indices |
| Typical source | Vision encoder plus projector | VQ-style image tokenizer |
| Main strength | High fidelity for understanding | Unified generation and understanding |
| Main weakness | Cannot be sampled as output directly | Quantization loses fine detail |
| Typical use | Vision-language models | Native multimodal and any-to-any models |
The choice is a trade-off between fidelity and unification. Quantization error makes discrete tokens weaker at reading small text or counting objects, while continuous tokens require a separate decoder, such as a diffusion model, when the system must also generate images.
Controlling the visual token budget
The visual token budget is the number of tokens a model allows per image, and engineers control it with resampling, merging and dynamic resolution. A fixed budget keeps latency predictable, whereas an adaptive budget spends more tokens on dense documents and charts and fewer on simple photographs.
Each technique compresses or expands the sequence differently. The table below summarizes the main approaches and their typical effect.
| Technique | How it works | Typical effect |
|---|---|---|
| Learned query resampler | A fixed set of queries attends to all patches | Compresses to about 32-64 tokens |
| Spatial token merging | Neighboring patches are concatenated or pooled | Cuts tokens by about 4x |
| Image tiling | The image is split into crops encoded separately | Raises tokens to several thousand |
| Native dynamic resolution | The patch grid follows the original aspect ratio | Token count scales with image size |
| Token pruning | Low-importance tokens are dropped during inference | Saves compute with small accuracy loss |
Video multiplies the problem, because every sampled frame adds its own tokens. Video-language models therefore sample a limited number of frames, merge tokens across time and sometimes use a dedicated video tokenizer that compresses temporal redundancy between consecutive frames.
Visual tokens and positional information
Visual tokens must carry spatial position, because a flattened sequence loses the two-dimensional layout of the image. Encoders add positional embeddings to each patch, and newer models use two-dimensional rotary position encodings so the language model can relate a token to its row and column.
Positional information is what makes visual grounding possible: a model can only point to the location of an object if its tokens preserve where each patch came from. Weak positional encoding is a known cause of errors in spatial reasoning, such as confusing left and right or misreading table layouts.
Choosing the right number of visual tokens
The right token count depends on how much detail the task needs. Practitioners select resolution settings, detail modes or tiling limits according to the content, as the following decision table shows.
| Task | Recommended budget | Reason |
|---|---|---|
| Scene classification | Low, a few hundred tokens | Global content matters more than detail |
| Document understanding | High, with tiling | Small text needs high resolution |
| Chart understanding | Medium to high | Axis labels and values are small |
| GUI grounding | High, native resolution | Buttons and icons must be located precisely |
| Long video analysis | Low per frame | Many frames must fit in context |
A practical rule is to start with the lowest budget that preserves the needed detail and increase it only when evaluation shows errors on small elements. Higher budgets raise latency, memory use and, in hosted APIs, the per-request cost, because image tokens are usually billed like text tokens.
Limitations and common misconceptions
Visual tokens are a lossy representation, and several frequent misunderstandings follow from ignoring that fact.
- More tokens do not guarantee accuracy. Beyond a certain budget, gains flatten while cost keeps rising.
- Visual tokens are not words. A single token rarely corresponds to a nameable object; meaning is spread across many tokens.
- Compression can erase small details, which contributes to multimodal hallucination when a model guesses at text it cannot resolve.
- Token counts differ between models, so the same image can consume very different shares of the context window.
Research also shows that many visual tokens receive little attention in deeper layers, which motivates pruning methods. Pruning, however, carries a risk: the discarded tokens may contain the one detail a question depends on.
Stripped to the essentials
A language model can only read sequences of small pieces. To let it look at a picture, the picture is cut into a grid of tiny squares, and each square is turned into a number pattern the model understands. Those pieces are visual tokens.
Think of a mosaic. A mosaic with a few large tiles shows the overall scene, while one with thousands of small tiles shows fine print. More tiles mean more detail but more work for the model, so every system has to decide how many tiles each picture deserves.
Where it is applied
An insurance company processes scanned claim forms with a vision-language model. At first, the team sends each A4 scan at a fixed low resolution of 336 x 336 pixels, which yields 576 visual tokens per page. The model summarizes the forms well but misreads policy numbers and small handwritten amounts in about one case out of ten.
The engineers switch to tiled high-resolution encoding, splitting each page into several crops plus a thumbnail, which raises the budget to roughly 2,500 tokens per page. Errors on numbers drop sharply, while processing time per page roughly triples. To control cost, they keep the high budget only for form pages and use the low setting for attached photographs of damage, where overall content matters more than fine print. The result is accurate extraction at a manageable cost.