Multimodal projector

Also known as: vision-language connector · Q-Former

On this page 7

A multimodal projector is the trainable module that maps features from a non-text encoder, usually a vision encoder, into the input embedding space of a language model, so that image, audio or video content can be processed as a sequence of tokens. In multimodal AI, it is the bridge that lets a pretrained encoder and a pretrained large language model work together without retraining either one from scratch.

The component is also called a vision-language connector, an adapter or an alignment module. Its simplest form is a single linear layer; more elaborate designs, such as the Q-Former introduced with BLIP-2, use learned queries and attention to compress hundreds of patch features into a fixed, small set of visual tokens.

How a multimodal projector connects two models

A multimodal projector solves a dimension and distribution mismatch. A vision encoder such as a ViT produces one feature vector per image patch, for example 576 vectors of 1,024 dimensions for a 336-pixel image with 14-pixel patches. The language model expects vectors with its own hidden size, often 4,096 or 5,120 dimensions, and with statistics that resemble its learned word embeddings.

The projector transforms each encoder output, or a pooled summary of several outputs, into vectors the language model accepts. These visual tokens are then concatenated or interleaved with ordinary text embeddings, and the language model attends to them through its normal self-attention layers. Nothing in the language model has to know that a token came from an image.

The following table traces the data flow through a typical vision-language model built around a projector.

StageComponentTypical output
Image inputPreprocessing and resizing336 × 336 or 448 × 448 pixels
Feature extractionFrozen vision encoder (CLIP or SigLIP)256-1,024 patch vectors
ProjectionMultimodal projectorSame or fewer vectors at the LLM hidden size
Sequence assemblyPrompt templateVisual tokens plus text tokens
ReasoningLanguage model decoderGenerated text answer
Data flow through a projector-based vision-language model

Main projector architectures

Projector designs differ mainly in how many visual tokens they emit and how much computation they add. Simple layers keep every patch, while query-based and pooling designs reduce the token count to save context length and inference cost.

Projector typeMechanismToken countTypical use
Linear projectionOne matrix multiplication per patchUnchangedEarly LLaVA versions
Two-layer MLPLinear, GELU activation, linearUnchangedLLaVA-1.5 and many open models
Q-FormerLearned queries with cross-attentionFixed, often 32BLIP-2 and InstructBLIP
Perceiver resamplerLatent queries attend to featuresFixed, often 64Flamingo-style models
Pooling or pixel shuffleMerges neighboring patchesReduced by 4× or moreHigh-resolution document models
Common projector types and their properties

The two-layer MLP has become a common default because it is cheap, stable to train and preserves spatial detail. The Q-Former offers strong compression, but its fixed query budget can discard fine detail needed for reading small text or counting objects.

How the projector is trained

Projector training usually follows a two-stage recipe. In the first stage, called feature alignment or pretraining, the vision encoder and the language model stay frozen and only the projector learns, using large image-caption datasets. The objective is ordinary next-token prediction on the caption, which forces the projector to produce visual tokens the language model can describe.

In the second stage, visual instruction tuning, the projector continues to train together with the language model, either fully or through low-rank adapters, on conversation-style data such as visual question answering and multi-turn dialog. The vision encoder often remains frozen, although some recent models unfreeze it at a low learning rate for better fine-grained perception.

StageTrainable partsDataGoal
Alignment pretrainingProjector onlyImage-text pair datasetMap visual features into word space
Instruction tuningProjector and LLMVisual conversations and tasksFollow multimodal instructions
Optional refinementAll modules, low learning rateHigh-resolution and domain dataImprove detail and specialization
Training stages for a projector-based model

Because the projector holds only a few million to a few hundred million parameters, the alignment stage is fast and inexpensive compared with full multimodal pretraining.

Projector compared with other fusion approaches

A projector implements a form of early fusion at the input level: visual information enters the language model as tokens before any reasoning begins. This differs from gated cross-attention, where new attention layers inside the language model read image features, and from native multimodality, where one model is trained on mixed data from the start.

ApproachWhere vision entersChanges to the LLMMain trade-off
Projector (adapter)Input token sequenceNone or light tuningConsumes context window
Gated cross-attentionExtra layers inside the LLMNew layers insertedMore parameters, less context cost
Native multimodal modelShared tokenizer from pretrainingModel built multimodalVery expensive to train
Late fusionSeparate model outputs combinedNoneWeak cross-modal reasoning
Projector versus related integration methods

Design choices and common pitfalls

The most consequential decision is the visual token budget. Every visual token occupies context and attention compute, so a projector that keeps 2,880 tokens for a tiled high-resolution image can slow inference sharply, while aggressive compression can lose text and small objects. Engineers balance these effects against the target tasks, such as document understanding versus general captioning.

  • Weak alignment data produces a projector that describes images vaguely and contributes to multimodal hallucination.
  • Skipping the alignment stage and tuning everything at once often destabilizes the language model and degrades its text skills.
  • Mismatched image resolution between training and inference changes the patch count and confuses position handling.
  • Assuming the projector adds knowledge is a misconception: it translates features, while recognition ability comes mainly from the encoder.

A non-technical view

Imagine two experts who speak different languages. One is excellent at looking at pictures, and the other is excellent at writing and reasoning. A multimodal projector is the interpreter that sits between them. It takes what the picture expert noticed and rewrites it in the vocabulary the writing expert understands.

The interpreter is small and quick to train, which is why many companies and researchers can build image-aware chat assistants cheaply. The key point is simple: the projector does not see or think by itself; it only makes sure the two experts can understand each other.

A concrete example

A research team at a university wants an assistant that answers questions about laboratory instrument photos. Instead of training a new model, it combines a frozen SigLIP vision encoder with an open 7-billion-parameter language model and adds a two-layer MLP projector with roughly 20 million parameters.

In the first stage, the team trains only the projector on about 500,000 image-caption pairs, which takes a few hours on eight GPUs. In the second stage, it fine-tunes the projector and the language model with low-rank adapters on 40,000 annotated question-answer pairs about gauges, displays and labels. Each image becomes 729 visual tokens placed before the question.

The result is an assistant that reads instrument displays and explains settings reliably, built for a small fraction of the cost of full multimodal pretraining. When the team later needs to read very small labels, it adds image tiling, accepting a longer context in exchange for better detail.