A multimodal projector is the trainable module that maps features from a non-text encoder, usually a vision encoder, into the input embedding space of a language model, so that image, audio or video content can be processed as a sequence of tokens. In multimodal AI, it is the bridge that lets a pretrained encoder and a pretrained large language model work together without retraining either one from scratch.
The component is also called a vision-language connector, an adapter or an alignment module. Its simplest form is a single linear layer; more elaborate designs, such as the Q-Former introduced with BLIP-2, use learned queries and attention to compress hundreds of patch features into a fixed, small set of visual tokens.
How a multimodal projector connects two models
A multimodal projector solves a dimension and distribution mismatch. A vision encoder such as a ViT produces one feature vector per image patch, for example 576 vectors of 1,024 dimensions for a 336-pixel image with 14-pixel patches. The language model expects vectors with its own hidden size, often 4,096 or 5,120 dimensions, and with statistics that resemble its learned word embeddings.
The projector transforms each encoder output, or a pooled summary of several outputs, into vectors the language model accepts. These visual tokens are then concatenated or interleaved with ordinary text embeddings, and the language model attends to them through its normal self-attention layers. Nothing in the language model has to know that a token came from an image.
The following table traces the data flow through a typical vision-language model built around a projector.
| Stage | Component | Typical output |
|---|---|---|
| Image input | Preprocessing and resizing | 336 × 336 or 448 × 448 pixels |
| Feature extraction | Frozen vision encoder (CLIP or SigLIP) | 256-1,024 patch vectors |
| Projection | Multimodal projector | Same or fewer vectors at the LLM hidden size |
| Sequence assembly | Prompt template | Visual tokens plus text tokens |
| Reasoning | Language model decoder | Generated text answer |
Main projector architectures
Projector designs differ mainly in how many visual tokens they emit and how much computation they add. Simple layers keep every patch, while query-based and pooling designs reduce the token count to save context length and inference cost.
| Projector type | Mechanism | Token count | Typical use |
|---|---|---|---|
| Linear projection | One matrix multiplication per patch | Unchanged | Early LLaVA versions |
| Two-layer MLP | Linear, GELU activation, linear | Unchanged | LLaVA-1.5 and many open models |
| Q-Former | Learned queries with cross-attention | Fixed, often 32 | BLIP-2 and InstructBLIP |
| Perceiver resampler | Latent queries attend to features | Fixed, often 64 | Flamingo-style models |
| Pooling or pixel shuffle | Merges neighboring patches | Reduced by 4× or more | High-resolution document models |
The two-layer MLP has become a common default because it is cheap, stable to train and preserves spatial detail. The Q-Former offers strong compression, but its fixed query budget can discard fine detail needed for reading small text or counting objects.
How the projector is trained
Projector training usually follows a two-stage recipe. In the first stage, called feature alignment or pretraining, the vision encoder and the language model stay frozen and only the projector learns, using large image-caption datasets. The objective is ordinary next-token prediction on the caption, which forces the projector to produce visual tokens the language model can describe.
In the second stage, visual instruction tuning, the projector continues to train together with the language model, either fully or through low-rank adapters, on conversation-style data such as visual question answering and multi-turn dialog. The vision encoder often remains frozen, although some recent models unfreeze it at a low learning rate for better fine-grained perception.
| Stage | Trainable parts | Data | Goal |
|---|---|---|---|
| Alignment pretraining | Projector only | Image-text pair dataset | Map visual features into word space |
| Instruction tuning | Projector and LLM | Visual conversations and tasks | Follow multimodal instructions |
| Optional refinement | All modules, low learning rate | High-resolution and domain data | Improve detail and specialization |
Because the projector holds only a few million to a few hundred million parameters, the alignment stage is fast and inexpensive compared with full multimodal pretraining.
Projector compared with other fusion approaches
A projector implements a form of early fusion at the input level: visual information enters the language model as tokens before any reasoning begins. This differs from gated cross-attention, where new attention layers inside the language model read image features, and from native multimodality, where one model is trained on mixed data from the start.
| Approach | Where vision enters | Changes to the LLM | Main trade-off |
|---|---|---|---|
| Projector (adapter) | Input token sequence | None or light tuning | Consumes context window |
| Gated cross-attention | Extra layers inside the LLM | New layers inserted | More parameters, less context cost |
| Native multimodal model | Shared tokenizer from pretraining | Model built multimodal | Very expensive to train |
| Late fusion | Separate model outputs combined | None | Weak cross-modal reasoning |
Design choices and common pitfalls
The most consequential decision is the visual token budget. Every visual token occupies context and attention compute, so a projector that keeps 2,880 tokens for a tiled high-resolution image can slow inference sharply, while aggressive compression can lose text and small objects. Engineers balance these effects against the target tasks, such as document understanding versus general captioning.
- Weak alignment data produces a projector that describes images vaguely and contributes to multimodal hallucination.
- Skipping the alignment stage and tuning everything at once often destabilizes the language model and degrades its text skills.
- Mismatched image resolution between training and inference changes the patch count and confuses position handling.
- Assuming the projector adds knowledge is a misconception: it translates features, while recognition ability comes mainly from the encoder.
A non-technical view
Imagine two experts who speak different languages. One is excellent at looking at pictures, and the other is excellent at writing and reasoning. A multimodal projector is the interpreter that sits between them. It takes what the picture expert noticed and rewrites it in the vocabulary the writing expert understands.
The interpreter is small and quick to train, which is why many companies and researchers can build image-aware chat assistants cheaply. The key point is simple: the projector does not see or think by itself; it only makes sure the two experts can understand each other.
A concrete example
A research team at a university wants an assistant that answers questions about laboratory instrument photos. Instead of training a new model, it combines a frozen SigLIP vision encoder with an open 7-billion-parameter language model and adds a two-layer MLP projector with roughly 20 million parameters.
In the first stage, the team trains only the projector on about 500,000 image-caption pairs, which takes a few hours on eight GPUs. In the second stage, it fine-tunes the projector and the language model with low-rank adapters on 40,000 annotated question-answer pairs about gauges, displays and labels. Each image becomes 729 visual tokens placed before the question.
The result is an assistant that reads instrument displays and explains settings reliably, built for a small fraction of the cost of full multimodal pretraining. When the team later needs to read very small labels, it adds image tiling, accepting a longer context in exchange for better detail.