Modality

Also known as: data modality

On this page 7

A modality is a distinct type of data signal, such as text, images, audio, video or sensor readings, that has its own structure, statistics and preferred way of being encoded by a model. In multimodal AI, the term describes the channels through which a system perceives inputs and produces outputs, and the number and combination of modalities define what a model can actually do.

The concept matters because each data modality differs in dimensionality, sampling, noise and information density. A model that handles several modalities must translate them into a shared representation space before it can reason across them, and that translation is where most of the engineering effort and most of the failure modes of multimodal systems are found.

What makes a modality distinct

A modality is distinct when its data has a different underlying structure that requires a different processing approach. Text is a discrete sequence of symbols, images are dense two-dimensional pixel grids, audio is a one-dimensional waveform sampled thousands of times per second, and video adds a temporal axis to images. These differences determine which architecture, preprocessing and tokenization work well.

The boundary between modalities is partly a matter of convention. Practitioners usually treat RGB photographs and depth maps as separate modalities, because they carry different physical information, while photographs from two different cameras count as the same modality. Structured inputs such as tables, code, graphs and time series are sometimes called modalities as well, especially when a model needs a dedicated encoder for them.

The following table summarizes the common modalities and the properties that shape how models handle them.

ModalityRaw structureTypical encodingMain difficulty
TextDiscrete token sequenceSubword tokenizer and embedding tableAmbiguity and long context
Image2D grid of pixelsPatch embeddings from a vision encoderHigh dimensionality, spatial detail
Audio1D waveform, 16-48 kHzSpectrogram or neural audio codecLong sequences, speaker variation
VideoImage frames over timeSpatiotemporal patches or sampled framesVery large token counts
Depth and 3DPoint clouds or depth mapsPoint or voxel encodersSparse, irregular geometry
Sensor time seriesMultichannel numeric signalsConvolutional or transformer encodersDomain-specific noise and drift
Core modalities and their structural properties

How models bring modalities into one space

A multimodal model handles several modalities by converting each one into vectors of the same dimensionality that a shared network can process. The usual pipeline uses a modality-specific encoder, such as a vision transformer for images, followed by a projector that maps encoder outputs into the embedding space of a language model. The resulting visual tokens or audio tokens then sit alongside text tokens in a single sequence.

Two broader strategies exist. Contrastive alignment, used by models such as CLIP and SigLIP, trains separate encoders so that matching image-text pairs land close together in a joint embedding space. Generative integration, used by vision-language models and multimodal large language models, feeds encoded inputs directly into a decoder that produces text or other outputs. Natively multimodal models go further and train on interleaved modalities from the start rather than attaching encoders to a pretrained text model.

The point at which modalities are combined is a key design choice, compared below.

Integration pointHow it worksStrengthWeakness
Early fusionRaw or lightly encoded inputs share one networkRich cross-modal interactionHigh compute, harder training
Intermediate fusionEncoder features merged through cross-attention or projectionBalances flexibility and costAlignment depends on projector quality
Late fusionSeparate models combine their predictionsSimple and modularWeak cross-modal reasoning
Joint embeddingEncoders trained to share one vector spaceFast retrieval and zero-shot matchingLimited generation ability
Where modalities are combined in a model

Input and output modalities

Model capabilities are usually described by their input modalities and output modalities, which need not be the same. Many vision-language models accept images and text but produce only text, while text-to-image systems accept text and produce images. Any-to-any models aim to accept and generate every supported modality, often by converting all of them into discrete tokens with image, audio and video tokenizers.

The table below lists common modality pairings and the tasks they support.

Input to outputRepresentative taskTypical model family
Image and text to textVisual question answering, document understandingVision-language model
Text to imageImage generation from a promptDiffusion or autoregressive generator
Audio to textSpeech recognition, audio captioningAudio language model
Audio to audioSpoken dialogue, speech translationSpeech-to-speech model
Video and text to textVideo question answering, summarizationVideo-language model
Any to anyMixed generation across modalitiesNatively multimodal model
Modality pairings and representative tasks

A claim that a model is multimodal should therefore always be read with the exact modality matrix in mind. A system that reads images but cannot generate them, or that transcribes speech through a separate pipeline rather than processing audio natively, has narrower capabilities than its marketing label may suggest.

Modality gaps and imbalance

Combining modalities introduces problems that do not exist in single-modality models. The best known is the modality gap: even after contrastive training, embeddings of images and texts often occupy separate regions of the shared space, which weakens cross-modal retrieval and transfer. A related issue is modality imbalance, where the model learns to rely on the easier or more informative modality and ignores the other.

In vision-language systems, imbalance often appears as language prior dominance: the model answers from text patterns rather than from the image, which is one cause of multimodal hallucination. Data scarcity adds a further constraint, because high-quality paired data for audio, video or 3D is far smaller than the web-scale image-text pair datasets used for vision.

ProblemTypical symptomCommon mitigation
Modality gapPoor cross-modal retrieval accuracyBetter contrastive objectives, temperature tuning
Modality imbalanceModel ignores image or audio inputBalanced sampling, modality dropout
Missing modalityFailure when one input is absent at inferenceTraining with randomly dropped inputs
Token budget overloadVideo or high-resolution images exceed contextFrame sampling, token pooling, cropping
Misaligned timingAudio and video events do not matchTemporal grounding and synchronization
Common modality problems and mitigations

Choosing modalities for a system

Adding a modality is worthwhile only when it carries information unavailable elsewhere. A support assistant that receives screenshots gains real value from vision, while adding audio to a document pipeline usually adds cost without benefit. Each extra modality raises inference cost, latency and evaluation effort, because every combination must be tested.

  • Map the task to the minimum set of modalities that contain the needed evidence.
  • Check whether the model processes each modality natively or through a chained conversion such as speech-to-text.
  • Estimate the token cost per input, since one high-resolution image or a minute of video can equal thousands of text tokens.
  • Evaluate performance per modality and per combination, not only on aggregate benchmarks.

In simple terms

A modality is simply a kind of information, much like our senses. Reading words, looking at a picture and listening to a sound are three different modalities. Each one arrives in a different shape, so a computer has to learn to understand each of them separately.

A multimodal AI system is one that can work with several of these kinds of information together. The key idea is translation: the system turns words, pictures and sounds into one common internal language so that it can connect them, for example by explaining what is happening in a photo.

A practical example

An insurance company builds a multimodal claims assistant for car damage. Customers upload two or three photos of the vehicle, a short written description and sometimes a voice note recorded at the scene. The team first maps the modalities: images show the damage, text gives the policy context, and audio adds the customer account.

Testing shows that the vision-language model often answers from the written description alone, even when the photos contradict it. The engineers add training examples where text and images disagree and evaluate image-only accuracy separately. They also transcribe voice notes with a speech recognition model rather than sending raw audio, which cuts cost per claim by roughly a third. The final system weighs the photos properly and flags mismatches for a human adjuster instead of trusting the text.