Multimodal large language model

Also known as: MLLM · multimodal LLM · multimodal transformer

On this page 8

A multimodal large language model (MLLM) is a large language model that accepts several kinds of input, such as text, images, audio and video, and reasons over them in one shared sequence of tokens. In Multimodal AI, the MLLM is the dominant architecture for assistants that describe photos, read documents, answer questions about charts or follow spoken instructions.

Most MLLMs keep a pretrained language model as the reasoning core and attach encoders that translate other modalities into representations the language model can process. The output is usually text, although any-to-any models can also generate images or speech.

How a multimodal large language model processes inputs

An MLLM converts every non-text input into embedding vectors in the language model’s input space, then lets the transformer attend across all of them at once. A typical pipeline has three parts: a modality encoder, a connector or projector, and the language model backbone.

For images, a vision encoder such as a Vision Transformer trained with CLIP or SigLIP objectives splits the picture into patches and produces one feature vector per patch. The multimodal projector, often a small two-layer MLP or a query-based resampler, maps those features to visual tokens that sit next to text tokens in the prompt. The language model then predicts the answer token by token, exactly as it would for a text-only prompt.

The table below summarizes what each component contributes and what typically limits it.

ComponentRoleTypical limitation
Modality encoderExtracts features from pixels, waveforms or framesFixed input resolution loses fine detail
Projector or connectorAligns encoder features with the language embedding spaceToo few tokens discard spatial information
Language model backbonePerforms reasoning and generates the responseContext length caps how many images or frames fit
Output decoder (optional)Turns generated tokens into images or speechAdds latency and training complexity
Core components of a typical MLLM

Training stages and data

Most MLLMs are trained in stages that reuse pretrained parts instead of learning everything from scratch. This approach is far cheaper than full pretraining and preserves the language ability of the base model.

  1. Alignment pretraining: the encoder and language model stay frozen while the projector learns from large image-text pair datasets, such as captioned web images.
  2. Multimodal pretraining on interleaved data: documents that mix images and text teach the model to handle several images in context and to follow longer visual narratives.
  3. Visual instruction tuning: curated question-answer and conversation data, often partly synthetic, teach the model to follow instructions about images.
  4. Preference optimization: human or AI feedback reduces unsupported claims and improves refusals and formatting.

Data quality matters more than raw volume in the later stages. Noisy alt text in web-scale corpora teaches weak grounding, which is one reason why multimodal hallucination remains a persistent failure mode.

Architectural variants of MLLMs

MLLM designs differ mainly in where and how modalities are fused. The choice affects cost, flexibility and how deeply the model can relate visual detail to language.

DesignHow modalities meetStrengthsWeaknesses
Projector-based late fusionVisual tokens are prepended to the text promptSimple, cheap, reuses strong encodersEncoder is not tuned for language reasoning
Gated cross-attentionText layers attend to visual features through added layersKeeps the language model intactMore parameters and harder scaling
Native early fusionAll modalities are tokenized and trained jointly from the startDeep cross-modal reasoning, any-to-any outputVery expensive pretraining
Mixture of experts by modalitySeparate expert weights per modality inside one modelEfficient specializationRouting complexity
Common fusion strategies in multimodal large language models

Native multimodality has become more common in frontier systems, while open-weight models frequently use the projector approach because it lets a team combine an existing vision encoder with an existing language model.

An MLLM is broader than a vision-language model, which handles only images and text, and different from a contrastive model such as CLIP, which embeds images and text but does not generate language. The comparison below clarifies the boundaries.

Model typeInputsOutputsTypical use
Text-only LLMTextTextWriting, coding, question answering
Contrastive model (CLIP, SigLIP)Image and text separatelySimilarity scores and embeddingsSearch, zero-shot classification
Vision-language modelImages and textTextCaptioning, visual question answering
Multimodal large language modelText, images, audio, videoMostly textGeneral assistants, document understanding
Any-to-any modelSeveral modalitiesSeveral modalitiesVoice assistants, image editing by instruction
MLLMs and neighboring concepts

Practical use and evaluation

MLLMs are used where visual or audio evidence must be interpreted together with language. Common tasks include document understanding, chart understanding, GUI grounding for software agents, video question answering and accessibility descriptions for people with low vision.

Choosing a model depends on input resolution, context length and grounding accuracy more than on headline benchmark scores. The following decision table reflects typical engineering trade-offs.

SituationRecommended choiceReason
Dense scanned formsHigh-resolution tiling or dynamic resolutionSmall text needs many visual tokens
Long videosFrame sampling with temporal groundingContext window limits frame count
On-device inferenceSmall backbone with compressed visual tokensMemory and latency budgets
Regulated dataSelf-hosted open-weight modelInputs stay inside the organization
Choosing an MLLM configuration

Evaluation combines task benchmarks for visual question answering, optical character recognition and reasoning with targeted hallucination tests. Practitioners also build domain-specific test sets, because public benchmarks often leak into training data.

Limitations and common failure modes

MLLMs inherit every weakness of language models and add new ones from perception. The most serious is confident description of objects that are not present, known as multimodal hallucination.

  • Weak counting and spatial reasoning: models often miscount objects or confuse left and right.
  • Resolution loss: downscaled images hide small text, so answers about fine print can be invented.
  • Language prior dominance: the model answers from common expectations instead of the actual image.
  • Prompt injection through images: text embedded in a picture can override user instructions.
  • High token cost: one image can consume hundreds to several thousand tokens, raising latency and price.

Human review remains necessary for medical, legal or safety-critical interpretation, and outputs should be verified against the source image whenever an error would be costly.

In simple terms

A multimodal large language model is a chatbot that can look and listen, not only read. You can show it a photo, a scanned letter or a short video and ask a question in ordinary words, and it answers in text.

Inside, a separate part turns the picture or sound into a kind of code the language part understands, a bit like a translator who describes a scene to a writer. The writer then answers. The answer is only as good as that description, so the model can still miss small details or imagine things that are not there.

A practical example

An insurance claims team receives about 2,000 damage reports a week, each with several phone photos and a handwritten form. The team deploys a self-hosted MLLM that reads the form, describes visible damage in each photo and fills a structured summary with fields such as vehicle part, damage type and estimated severity.

Images are tiled at high resolution so that license plates and handwriting stay legible, and every summary links each statement to the photo it came from. Adjusters review the draft instead of starting from a blank screen. In a pilot of this kind, review time per claim typically falls substantially, while adjusters still approve every final decision because the model occasionally misreads reflections as dents.