LLaVA (Large Language and Vision Assistant) is an open-source family of vision-language models that answers questions about images by connecting a pretrained image encoder to a pretrained large language model through a small trainable projector. Among AI models & assistants, it is best known as the project that introduced visual instruction tuning and showed that a capable multimodal chat assistant could be built with modest compute and public components.
The first version was released in 2023 by researchers from the University of Wisconsin-Madison, Microsoft Research and Columbia University. Its simple recipe became a reference architecture for open multimodal models in artificial intelligence, and many later systems describe themselves as LLaVA-style or build directly on its code and data.
How the LLaVA architecture works
LLaVA uses three components: a frozen vision encoder, a projector and a decoder-only language model. The original model paired the CLIP ViT-L/14 encoder with the Vicuna chat model, which is itself a fine-tuned Llama model.
The vision encoder turns an image into a grid of patch embeddings. The projector maps each patch embedding into the word embedding space of the language model, producing visual tokens that are placed in the prompt next to ordinary text tokens. The language model then generates the answer autoregressively, exactly as it would for a text-only prompt.
| Component | Role | Typical choice | Trained when |
|---|---|---|---|
| Vision encoder | Converts pixels to patch features | CLIP ViT-L/14, later SigLIP | Usually frozen |
| Projector | Maps image features to text space | Linear layer, later two-layer MLP | Both stages |
| Language model | Reasons and writes the answer | Vicuna, Llama, Mistral, Qwen | Instruction tuning stage |
Because the image enters the model as a fixed set of tokens, image resolution directly drives cost. A 336-pixel input to ViT-L/14 yields 576 visual tokens, which already exceeds the length of most text prompts.
Visual instruction tuning and training data
LLaVA is trained in two stages. In the feature alignment stage, only the projector is trained on roughly 600,000 filtered image-caption pairs so that visual tokens become intelligible to the language model. In the instruction tuning stage, the projector and the language model are fine-tuned together on conversational data about images.
The key idea was to create that conversational data with a text-only GPT-4. Researchers gave GPT-4 the captions and object bounding boxes of COCO images and asked it to write questions and answers, yielding about 158,000 instruction-following samples in three formats.
| Data type | Approximate samples | What it teaches |
|---|---|---|
| Conversations | 58,000 | Multi-turn questions about image content |
| Detailed descriptions | 23,000 | Rich, complete image descriptions |
| Complex reasoning | 77,000 | Inference beyond what is directly visible |
This approach showed that synthetic instruction data generated from symbolic image descriptions can transfer conversational skill to a multimodal model. It also introduced a known weakness: answers sometimes reflect what GPT-4 imagined from the captions, which contributes to multimodal hallucination.
Versions of the LLaVA family
The family evolved quickly, with each release improving resolution, data mixture or modality coverage while keeping the same basic design. The table lists the main public versions.
| Version | Released | Key change | Practical effect |
|---|---|---|---|
| LLaVA | April 2023 | Linear projector, GPT-4 generated data | First open visual chat assistant of its kind |
| LLaVA-1.5 | October 2023 | MLP projector, 336-pixel input, academic VQA data | Strong benchmarks with about a day of training on 8 GPUs |
| LLaVA-NeXT (1.6) | January 2024 | Dynamic high resolution through image tiling | Better OCR, charts and documents |
| LLaVA-OneVision | August 2024 | Single image, multi-image and video in one model | Transfer across visual scenarios |
Domain variants also appeared, most notably LLaVA-Med for biomedical images. Independent groups have released many derivatives that swap in different language backbones, so the name now refers to a lineage as much as to a single checkpoint.
How LLaVA compares with other vision-language approaches
LLaVA represents the projector-based, late-connection design, in which a separately pretrained vision encoder feeds tokens into a language model. Other designs connect the modalities differently, with consequences for cost and capability.
| Design | Representative approach | Strength | Limitation |
|---|---|---|---|
| Projector into LLM | LLaVA | Simple, cheap to train, easy to modify | Many visual tokens per image |
| Gated cross-attention | Flamingo-style models | Handles interleaved images well | More architectural changes |
| Query transformer | BLIP-2 style Q-Former | Compresses image to few tokens | May discard fine detail |
| Native multimodality | Models trained jointly from the start | Deep cross-modal integration | Very expensive pretraining |
Compared with proprietary assistants such as Gemini or GPT-class models, LLaVA checkpoints are smaller and less capable on hard reasoning, but they can be run locally, inspected and fine-tuned, which makes them valuable for research and private deployments.
The projector design also makes LLaVA a convenient testbed for experiments. Researchers can replace the vision encoder, change the language backbone or alter the data mixture while keeping the rest of the pipeline intact, which is why so many papers report results relative to a LLaVA baseline.
Limitations and practical considerations
LLaVA models share the general weaknesses of vision-language models and add some of their own. Teams evaluating them should consider the following points.
- Hallucination: the model may describe objects that are not present, especially when asked leading questions.
- Resolution limits: early versions downscale images, so small text and fine detail are often misread.
- Licensing: code is permissively licensed, but weights inherit the terms of the base language model and data sources, which must be checked before commercial use.
- Benchmark drift: published scores often use specific prompts; results on internal data can differ substantially.
For production use, a domain-specific fine-tune on a few thousand curated image-question pairs usually improves reliability more than switching to a larger general checkpoint.
A non-technical view
LLaVA is a free, openly shared AI model that can look at a picture and talk about it. It joins two existing parts. One part understands images. The other part understands and writes text.
A small connector translates what the image part sees into a form the text part can read. The result is a chat assistant with eyes. Its creators taught it by showing it many example conversations about pictures.
A concrete example
A regional insurance company wants to speed up the review of vehicle damage photos without sending customer images to an external cloud service. Its data team deploys a 7-billion-parameter LLaVA-1.5 checkpoint on a single on-premises GPU server.
The team fine-tunes the model on about 4,000 labeled claim photos paired with questions such as “Which panels are damaged?” and “Is the airbag deployed?”. After tuning, the model produces a structured first assessment for each claim in a few seconds.
Adjusters still approve every decision, but triage time drops by roughly a third, and photos never leave the company network. The team also logs cases where the model invents damage, using them as new training examples.