Visual instruction tuning

On this page 7

Visual instruction tuning is the supervised fine-tuning stage that teaches a vision-language model to follow natural-language instructions about images, by training it on image, instruction and response triples. In multimodal AI, it is the step that turns a model which merely aligns pictures with captions into a conversational assistant that can describe, compare, count, read and reason about visual content on request.

The method extends text-only instruction tuning to the visual modality. It was popularized in 2023 by the LLaVA project, which showed that a frozen vision encoder, a small multimodal projector and a pretrained large language model could acquire broad visual chat skills from a comparatively small instruction dataset of roughly 150,000 examples.

How visual instruction tuning trains a multimodal model

Visual instruction tuning works by feeding the model an image together with a user instruction and optimizing it to produce the reference response token by token. The image passes through a vision encoder such as a CLIP or SigLIP vision transformer, which outputs a grid of patch features. A projector maps those features into the embedding space of the language model, where they become visual tokens placed before or between the text tokens.

The training objective is the ordinary next-token cross-entropy loss, but it is computed only on the assistant’s response. Tokens of the system prompt, the instruction and the image are masked out of the loss, so the model learns to answer rather than to reproduce questions. Multi-turn conversations are flattened into one sequence, and every assistant turn contributes to the gradient.

Most recipes use two training stages. The table below summarizes what each stage updates and why the order matters.

StageDataTrainable partsPurpose
Feature alignmentHundreds of thousands of image-caption pairsProjector onlyMap visual features into the language embedding space
Instruction tuningImage-grounded conversations and tasksProjector and language modelTeach instruction following, dialogue and reasoning
Optional encoder unfreezingHigh-resolution or text-rich imagesVision encoder at a low learning rateImprove fine detail such as small text
Typical two-stage training recipe

Freezing the language model in the first stage protects its linguistic knowledge while the projector is still producing noisy inputs. Learning rates follow the same logic: the alignment stage commonly uses a rate around 1e-3, while the second stage drops to about 2e-5 so that the pretrained model is adjusted gently.

Sources of visual instruction data

The quality and diversity of the instruction data largely determine what the tuned model can do. Early work had no large pool of human-written visual dialogues, so it used a text-only language model as a data generator: it received symbolic descriptions of an image, namely captions and object bounding boxes, and wrote questions and answers as if it could see the picture.

Later recipes combine synthetic conversations with converted academic datasets, human annotations and outputs from stronger multimodal models. The table compares the main data types and their typical weaknesses.

Data typeHow it is producedStrengthTypical weakness
Generated conversationsLanguage model writes dialogue from captions and boxesNatural, varied phrasingDetails absent from captions get invented
Detailed descriptionsPrompted long captions of whole scenesRich scene coverageEncourages verbose, speculative answers
Complex reasoningQuestions requiring inference over the sceneBuilds multi-step reasoningHard to verify automatically
Converted academic tasksVQA, OCR and grounding sets rewritten as promptsPrecise, checkable answersShort, formulaic responses
Distilled model outputsAnswers from a stronger multimodal modelScales cheaply to millionsInherits the teacher’s errors
Common categories of visual instruction data

Mixing short-answer academic data with long conversational data creates a format conflict: the model may answer every question with one word or ramble when brevity is needed. A widely used fix is to append an explicit response-format prompt, such as an instruction to answer with a single word or phrase, to the short-answer examples.

Visual instruction tuning sits between multimodal pretraining and preference alignment in the life cycle of a multimodal large language model. It differs from pretraining in its objective and data scale, and from contrastive models such as CLIP in that it produces free-form text rather than similarity scores.

MethodTraining signalTypical data scaleResulting capability
Contrastive pretrainingMatching images to captionsHundreds of millions to billions of pairsShared embedding, zero-shot classification
Multimodal pretrainingCaptioning or interleaved next-token predictionMillions to billions of samplesBroad visual knowledge, weak instruction following
Visual instruction tuningSupervised responses to image-grounded promptsRoughly 100,000 to a few million samplesConversational, task-following behavior
Multimodal preference tuningRankings of better and worse answersThousands to hundreds of thousands of pairsFewer hallucinations, preferred style
Position of visual instruction tuning among related methods

Text-only instruction tuning remains the closest relative. The only structural change is that some input positions carry projected image features instead of word embeddings, which is why the same training frameworks and chat templates can usually be reused with small modifications.

Design choices that shape the tuned model

Several engineering decisions influence results as much as the dataset itself. Image resolution sets the ceiling for reading small text in documents and charts; higher resolution or dynamic image tiling raises the number of visual tokens and therefore memory use and latency.

  • Projector design: a linear layer is cheap, a two-layer MLP usually aligns better, and a query-based resampler compresses many patches into a fixed token budget.
  • Which modules to train: full fine-tuning gives the best quality, while LoRA adapters cut memory needs at a modest cost in accuracy.
  • Text-only data in the mix: retaining pure language examples limits catastrophic forgetting of writing and reasoning skills.
  • Chat template consistency: the same role markers and image placeholder must be used in training and inference, or responses degrade.

The decision table below maps frequent project goals to the setting that usually matters most.

GoalRecommended choiceReason
Read invoices and formsHigh resolution with tilingSmall glyphs vanish at low resolution
Run on limited GPUsLoRA on the language modelTrains a small fraction of parameters
Keep strong general chatMix in text-only instructionsReduces loss of language ability
Answer precise factual questionsAdd short-answer task dataTeaches concise, checkable outputs
Choosing settings by project goal

Limitations and common failure modes

The best-known weakness is multimodal hallucination: the tuned model confidently describes objects, text or relations that are not in the image. Synthetic training data contributes directly, because a data generator working from captions can write plausible details it never verified, and the student model learns that fluent guessing is rewarded.

Evaluation is a second difficulty. Open-ended answers resist exact matching, so many benchmarks use a language model as judge, which introduces its own bias toward long, confident responses. Dedicated probes for object hallucination and grounding complement these scores and should be part of any serious evaluation plan.

SymptomLikely causeTypical remedy
Invented objects in descriptionsSpeculative synthetic captionsFilter data, add preference tuning
One-word answers to open questionsToo much short-answer dataUse explicit format prompts
Weaker writing than the base modelNo text-only data in the mixBlend in language instructions
Misread small textLow input resolutionIncrease resolution or tile images
Symptoms after tuning and their usual remedies

Finally, visual instruction tuning teaches behavior more than knowledge. It cannot supply visual concepts that the encoder never learned in pretraining, so a model will remain weak on specialized imagery, such as microscopy slides, unless domain images enter earlier training stages or the encoder is adapted.

A non-technical view

A model that has seen millions of captioned pictures knows what things look like. It does not yet know how to help a person. Visual instruction tuning is the coaching stage that closes this gap. The model sees a picture, a request and a good answer. It practices until it can answer new requests about new pictures.

It resembles training a new employee who has the knowledge but no customer experience. Worked examples teach the employee to listen and reply usefully. The key takeaway is that the examples shape the habits. Careless examples produce an assistant that guesses. Careful ones produce an assistant that looks before it speaks.

A concrete example

An insurance company wants an internal assistant that reads photos of damaged vehicles and answers adjusters’ questions. Its machine learning team starts from an open multimodal model and assembles about 40,000 instruction examples: adjusters’ real questions about archived claim photos, written answers reviewed by senior staff, and short-answer items such as identifying which panel is dented. Each short-answer item carries a format instruction asking for a brief phrase.

The team freezes the vision encoder, trains the projector and LoRA adapters on the language model for three epochs, and keeps 10 percent general text instructions in the mix. Before tuning, the base model described cars in generic terms and often invented scratches. After tuning, it names damaged panels correctly in most held-out test photos and says when an area is not visible. Adjusters use it to draft first damage summaries, which they then verify before approving any estimate.