Visual tokens

Also known as: image tokens

On this page 8

Visual tokens, also called image tokens, are the discrete or continuous units into which a model converts an image or video frame so that a transformer can process visual content in the same sequence as text. In multimodal AI, they are the bridge between pixels and the language model: each token usually represents one small image patch or a learned summary of several patches.

The number of visual tokens per image is one of the most consequential design choices in artificial intelligence systems that see. It sets the balance between visual detail and compute cost, because attention cost grows with sequence length, and it determines how much of the context window an image consumes.

How images become visual tokens

An image becomes a sequence of visual tokens through patch splitting, encoding and projection. A vision encoder, typically a Vision Transformer, cuts the image into a grid of square patches, often 14 or 16 pixels per side, flattens each patch and maps it to a vector called a patch embedding.

The encoder then refines these embeddings with self-attention across all patches, so that each vector carries information about its neighborhood and the whole scene. A multimodal projector, usually a small multilayer perceptron or a resampling module, maps the result into the embedding space of the language model. Those projected vectors are the visual tokens the language model actually reads.

The patch grid explains the basic arithmetic. The following table shows how input resolution and patch size determine the raw token count before any compression.

Input resolutionPatch sizePatch gridVisual tokens
224 x 224 px16 px14 x 14196
224 x 224 px14 px16 x 16256
336 x 336 px14 px24 x 24576
448 x 448 px14 px32 x 321,024
Raw visual token counts for common encoder settings

Because the count grows with the square of the resolution, doubling the side length of an image roughly quadruples the token count. This quadratic growth is the central engineering problem behind every design choice discussed below.

Continuous and discrete visual tokens

Visual tokens come in two families: continuous embeddings and discrete codebook indices. Continuous tokens are real-valued vectors produced by an encoder and projector; they preserve fine detail and suit models that only need to understand images, such as most vision-language models.

Discrete tokens come from an image tokenizer such as a VQ-VAE or VQGAN, which compresses each region into the index of the nearest entry in a learned codebook of typically 8,192 to 16,384 codes. Discrete tokens let one model predict images the same way it predicts words, which is the basis of native multimodality and any-to-any models.

PropertyContinuous tokensDiscrete tokens
RepresentationReal-valued vectorsInteger codebook indices
Typical sourceVision encoder plus projectorVQ-style image tokenizer
Main strengthHigh fidelity for understandingUnified generation and understanding
Main weaknessCannot be sampled as output directlyQuantization loses fine detail
Typical useVision-language modelsNative multimodal and any-to-any models
Comparison of the two token families

The choice is a trade-off between fidelity and unification. Quantization error makes discrete tokens weaker at reading small text or counting objects, while continuous tokens require a separate decoder, such as a diffusion model, when the system must also generate images.

Controlling the visual token budget

The visual token budget is the number of tokens a model allows per image, and engineers control it with resampling, merging and dynamic resolution. A fixed budget keeps latency predictable, whereas an adaptive budget spends more tokens on dense documents and charts and fewer on simple photographs.

Each technique compresses or expands the sequence differently. The table below summarizes the main approaches and their typical effect.

TechniqueHow it worksTypical effect
Learned query resamplerA fixed set of queries attends to all patchesCompresses to about 32-64 tokens
Spatial token mergingNeighboring patches are concatenated or pooledCuts tokens by about 4x
Image tilingThe image is split into crops encoded separatelyRaises tokens to several thousand
Native dynamic resolutionThe patch grid follows the original aspect ratioToken count scales with image size
Token pruningLow-importance tokens are dropped during inferenceSaves compute with small accuracy loss
Techniques for managing visual token counts

Video multiplies the problem, because every sampled frame adds its own tokens. Video-language models therefore sample a limited number of frames, merge tokens across time and sometimes use a dedicated video tokenizer that compresses temporal redundancy between consecutive frames.

Visual tokens and positional information

Visual tokens must carry spatial position, because a flattened sequence loses the two-dimensional layout of the image. Encoders add positional embeddings to each patch, and newer models use two-dimensional rotary position encodings so the language model can relate a token to its row and column.

Positional information is what makes visual grounding possible: a model can only point to the location of an object if its tokens preserve where each patch came from. Weak positional encoding is a known cause of errors in spatial reasoning, such as confusing left and right or misreading table layouts.

Choosing the right number of visual tokens

The right token count depends on how much detail the task needs. Practitioners select resolution settings, detail modes or tiling limits according to the content, as the following decision table shows.

TaskRecommended budgetReason
Scene classificationLow, a few hundred tokensGlobal content matters more than detail
Document understandingHigh, with tilingSmall text needs high resolution
Chart understandingMedium to highAxis labels and values are small
GUI groundingHigh, native resolutionButtons and icons must be located precisely
Long video analysisLow per frameMany frames must fit in context
Matching the token budget to the task

A practical rule is to start with the lowest budget that preserves the needed detail and increase it only when evaluation shows errors on small elements. Higher budgets raise latency, memory use and, in hosted APIs, the per-request cost, because image tokens are usually billed like text tokens.

Limitations and common misconceptions

Visual tokens are a lossy representation, and several frequent misunderstandings follow from ignoring that fact.

  • More tokens do not guarantee accuracy. Beyond a certain budget, gains flatten while cost keeps rising.
  • Visual tokens are not words. A single token rarely corresponds to a nameable object; meaning is spread across many tokens.
  • Compression can erase small details, which contributes to multimodal hallucination when a model guesses at text it cannot resolve.
  • Token counts differ between models, so the same image can consume very different shares of the context window.

Research also shows that many visual tokens receive little attention in deeper layers, which motivates pruning methods. Pruning, however, carries a risk: the discarded tokens may contain the one detail a question depends on.

Stripped to the essentials

A language model can only read sequences of small pieces. To let it look at a picture, the picture is cut into a grid of tiny squares, and each square is turned into a number pattern the model understands. Those pieces are visual tokens.

Think of a mosaic. A mosaic with a few large tiles shows the overall scene, while one with thousands of small tiles shows fine print. More tiles mean more detail but more work for the model, so every system has to decide how many tiles each picture deserves.

Where it is applied

An insurance company processes scanned claim forms with a vision-language model. At first, the team sends each A4 scan at a fixed low resolution of 336 x 336 pixels, which yields 576 visual tokens per page. The model summarizes the forms well but misreads policy numbers and small handwritten amounts in about one case out of ten.

The engineers switch to tiled high-resolution encoding, splitting each page into several crops plus a thumbnail, which raises the budget to roughly 2,500 tokens per page. Errors on numbers drop sharply, while processing time per page roughly triples. To control cost, they keep the high budget only for form pages and use the low setting for attached photographs of damage, where overall content matters more than fine print. The result is accurate extraction at a manageable cost.