Vision-language model

Also known as: VLM

On this page 7

A vision-language model (VLM) is a neural network that jointly processes images and text, so it can describe a picture, answer questions about it, read documents and charts, or follow instructions that refer to visual content. In multimodal AI, the VLM is the most widely deployed class of models, because it connects the mature field of computer vision with the reasoning and language abilities of large language models.

The term covers two families. Contrastive dual encoders such as CLIP and SigLIP map images and text into a shared embedding space for matching and retrieval, while generative VLMs feed visual features into a language model that produces free-form text. Most modern assistants that accept image uploads are generative VLMs, often also called multimodal large language models.

How a vision-language model connects pixels to words

A generative VLM works by converting an image into a sequence of vectors that the language model can read alongside ordinary text tokens. The pipeline has three parts: a vision encoder, a multimodal projector and a language model backbone, and each part can be trained, frozen or fine-tuned separately.

The vision encoder, usually a Vision Transformer (ViT), splits the image into fixed-size patches of 14 or 16 pixels and turns each patch into an embedding. The projector, often a small multilayer perceptron or a cross-attention resampler, maps those embeddings into the token space of the language model. The language model then attends to these visual tokens exactly as it attends to words, and generates the answer one token at a time.

The table below summarizes what each component does and the typical design choices.

ComponentRoleTypical implementation
Vision encoderExtracts visual features from patchesViT pretrained with CLIP or SigLIP objectives
Multimodal projectorAligns visual features with text embeddingsTwo-layer MLP, Q-Former or perceiver resampler
Language modelReasons over tokens and generates textDecoder-only transformer, 1B to over 70B parameters
Tokenizer and templateMarks where image tokens sit in the promptSpecial image tokens inside a chat template
Main components of a generative vision-language model

A single 336 by 336 pixel image at a patch size of 14 produces 576 visual tokens, so image resolution directly drives context length, latency and memory cost. High-resolution models tile the image into crops or use dynamic resolution, which can raise the count to several thousand tokens per image.

Training stages of a VLM

Most VLMs are built in stages that gradually align vision and language, rather than trained from scratch in one pass. Reusing a strong pretrained encoder and a pretrained language model saves compute and preserves the linguistic ability of the backbone.

StageTypical dataWhat it teaches
Contrastive pretrainingHundreds of millions of image-text pairsWhich captions match which images
Alignment pretrainingCaptioned images, projector trained onlyMapping visual features into word space
Visual instruction tuningImage-question-answer conversationsFollowing instructions about an image
Preference tuningRanked or corrected answersFewer hallucinations, safer responses
Common training stages and their data

Data quality matters more than volume in the later stages. Web-scale datasets such as LAION-5B provide breadth, but noisy alt text teaches weak grounding; curated, interleaved image-text data and synthetic captions written by stronger models are now standard for improving detail and factual accuracy.

A VLM differs from neighboring concepts mainly in which modalities it handles and whether it generates text. The comparison below separates terms that are often used interchangeably.

Model typeInputOutputMain use
Contrastive VLMImage or textEmbedding vectorRetrieval, zero-shot classification
Generative VLMImages plus textTextCaptioning, visual question answering
Multimodal large language modelSeveral modalitiesText, sometimes other mediaGeneral assistants
Natively multimodal modelTokens of all modalities from the startAny supported modalityUnified reasoning and generation
Classic vision modelImageFixed labels or boxesDetection, segmentation
VLM versus related model types

The boundary between a generative VLM and a multimodal large language model is blurry, and many authors treat them as synonyms. The useful distinction is that early fusion or native models mix modalities from the first layer, while adapter-based VLMs attach vision to an existing text model through late or mid fusion.

Typical tasks and capabilities

VLMs handle a broad set of open-ended visual tasks without a task-specific head. The same model can caption a photo, extract a table from a scanned invoice and explain a chart, provided its training data covered those domains.

  • Visual question answering: answering natural-language questions about an image.
  • Document understanding: reading forms, receipts and contracts, including layout.
  • Chart understanding: extracting values and trends from plots.
  • Visual grounding: locating the region that a phrase refers to, often as coordinates.
  • GUI grounding: identifying buttons and fields on screenshots for computer-use agents.
  • Zero-shot image classification: labeling images using text prompts instead of trained classes.

Capability is uneven across task types. Models that score well on general benchmarks can still fail at counting, precise spatial relations and reading small text, which is why evaluation should match the intended workload.

Limitations, risks and common mistakes

The most serious weakness of VLMs is multimodal hallucination: the model describes objects, text or attributes that are not in the image, because the language prior overrides visual evidence. Hallucination increases when the image is low resolution, when the question presupposes something false and when answers are long.

SymptomLikely causeMitigation
Invented objects or textStrong language prior, weak groundingAsk for evidence, use grounding prompts, verify
Misread small printDownsampled input resolutionCrop regions or use high-resolution tiling
Wrong countsPatch-level features blur instancesRequest coordinates or use a detector
Following text inside imagesPrompt injection through visual contentTreat image text as untrusted data
Slow or costly inferenceThousands of visual tokensLower resolution or token compression
Frequent failure modes and mitigations

Prompt injection via images is a real security concern: instructions written inside a screenshot or document can steer an agent. Production systems should separate visual content from instructions and keep a human or rule-based check on consequential actions. Biases in web training data also carry over, so outputs about people deserve careful review.

In everyday language

A vision-language model is software that can look at a picture and talk about it. It first turns the picture into a long list of numbers that capture shapes, colors and text, and then a language model reads those numbers much like it reads words.

Think of a person describing a photo to a friend over the phone. The model does the looking and the describing in one step. It is helpful but not infallible: it sometimes claims to see things that are not there, so important answers should still be checked.

Seen in the real world

An insurance claims team receives about 2,000 photos and scanned repair invoices per day. Previously, clerks typed the invoice totals, part names and vehicle damage descriptions into the claims system by hand, which took several minutes per claim.

The team deploys a generative VLM that reads each invoice at high resolution, extracts line items into structured fields and writes a short description of the visible damage. Any extracted total that does not match the sum of the line items is flagged for human review, and text found inside images is never treated as an instruction.

After a pilot, manual data entry time fell substantially, and reviewers concentrated on the minority of claims the model flagged. The main remaining errors came from blurred photos and handwritten notes, which the team now routes directly to staff.