A modality is a distinct type of data signal, such as text, images, audio, video or sensor readings, that has its own structure, statistics and preferred way of being encoded by a model. In multimodal AI, the term describes the channels through which a system perceives inputs and produces outputs, and the number and combination of modalities define what a model can actually do.
The concept matters because each data modality differs in dimensionality, sampling, noise and information density. A model that handles several modalities must translate them into a shared representation space before it can reason across them, and that translation is where most of the engineering effort and most of the failure modes of multimodal systems are found.
What makes a modality distinct
A modality is distinct when its data has a different underlying structure that requires a different processing approach. Text is a discrete sequence of symbols, images are dense two-dimensional pixel grids, audio is a one-dimensional waveform sampled thousands of times per second, and video adds a temporal axis to images. These differences determine which architecture, preprocessing and tokenization work well.
The boundary between modalities is partly a matter of convention. Practitioners usually treat RGB photographs and depth maps as separate modalities, because they carry different physical information, while photographs from two different cameras count as the same modality. Structured inputs such as tables, code, graphs and time series are sometimes called modalities as well, especially when a model needs a dedicated encoder for them.
The following table summarizes the common modalities and the properties that shape how models handle them.
| Modality | Raw structure | Typical encoding | Main difficulty |
|---|---|---|---|
| Text | Discrete token sequence | Subword tokenizer and embedding table | Ambiguity and long context |
| Image | 2D grid of pixels | Patch embeddings from a vision encoder | High dimensionality, spatial detail |
| Audio | 1D waveform, 16-48 kHz | Spectrogram or neural audio codec | Long sequences, speaker variation |
| Video | Image frames over time | Spatiotemporal patches or sampled frames | Very large token counts |
| Depth and 3D | Point clouds or depth maps | Point or voxel encoders | Sparse, irregular geometry |
| Sensor time series | Multichannel numeric signals | Convolutional or transformer encoders | Domain-specific noise and drift |
How models bring modalities into one space
A multimodal model handles several modalities by converting each one into vectors of the same dimensionality that a shared network can process. The usual pipeline uses a modality-specific encoder, such as a vision transformer for images, followed by a projector that maps encoder outputs into the embedding space of a language model. The resulting visual tokens or audio tokens then sit alongside text tokens in a single sequence.
Two broader strategies exist. Contrastive alignment, used by models such as CLIP and SigLIP, trains separate encoders so that matching image-text pairs land close together in a joint embedding space. Generative integration, used by vision-language models and multimodal large language models, feeds encoded inputs directly into a decoder that produces text or other outputs. Natively multimodal models go further and train on interleaved modalities from the start rather than attaching encoders to a pretrained text model.
The point at which modalities are combined is a key design choice, compared below.
| Integration point | How it works | Strength | Weakness |
|---|---|---|---|
| Early fusion | Raw or lightly encoded inputs share one network | Rich cross-modal interaction | High compute, harder training |
| Intermediate fusion | Encoder features merged through cross-attention or projection | Balances flexibility and cost | Alignment depends on projector quality |
| Late fusion | Separate models combine their predictions | Simple and modular | Weak cross-modal reasoning |
| Joint embedding | Encoders trained to share one vector space | Fast retrieval and zero-shot matching | Limited generation ability |
Input and output modalities
Model capabilities are usually described by their input modalities and output modalities, which need not be the same. Many vision-language models accept images and text but produce only text, while text-to-image systems accept text and produce images. Any-to-any models aim to accept and generate every supported modality, often by converting all of them into discrete tokens with image, audio and video tokenizers.
The table below lists common modality pairings and the tasks they support.
| Input to output | Representative task | Typical model family |
|---|---|---|
| Image and text to text | Visual question answering, document understanding | Vision-language model |
| Text to image | Image generation from a prompt | Diffusion or autoregressive generator |
| Audio to text | Speech recognition, audio captioning | Audio language model |
| Audio to audio | Spoken dialogue, speech translation | Speech-to-speech model |
| Video and text to text | Video question answering, summarization | Video-language model |
| Any to any | Mixed generation across modalities | Natively multimodal model |
A claim that a model is multimodal should therefore always be read with the exact modality matrix in mind. A system that reads images but cannot generate them, or that transcribes speech through a separate pipeline rather than processing audio natively, has narrower capabilities than its marketing label may suggest.
Modality gaps and imbalance
Combining modalities introduces problems that do not exist in single-modality models. The best known is the modality gap: even after contrastive training, embeddings of images and texts often occupy separate regions of the shared space, which weakens cross-modal retrieval and transfer. A related issue is modality imbalance, where the model learns to rely on the easier or more informative modality and ignores the other.
In vision-language systems, imbalance often appears as language prior dominance: the model answers from text patterns rather than from the image, which is one cause of multimodal hallucination. Data scarcity adds a further constraint, because high-quality paired data for audio, video or 3D is far smaller than the web-scale image-text pair datasets used for vision.
| Problem | Typical symptom | Common mitigation |
|---|---|---|
| Modality gap | Poor cross-modal retrieval accuracy | Better contrastive objectives, temperature tuning |
| Modality imbalance | Model ignores image or audio input | Balanced sampling, modality dropout |
| Missing modality | Failure when one input is absent at inference | Training with randomly dropped inputs |
| Token budget overload | Video or high-resolution images exceed context | Frame sampling, token pooling, cropping |
| Misaligned timing | Audio and video events do not match | Temporal grounding and synchronization |
Choosing modalities for a system
Adding a modality is worthwhile only when it carries information unavailable elsewhere. A support assistant that receives screenshots gains real value from vision, while adding audio to a document pipeline usually adds cost without benefit. Each extra modality raises inference cost, latency and evaluation effort, because every combination must be tested.
- Map the task to the minimum set of modalities that contain the needed evidence.
- Check whether the model processes each modality natively or through a chained conversion such as speech-to-text.
- Estimate the token cost per input, since one high-resolution image or a minute of video can equal thousands of text tokens.
- Evaluate performance per modality and per combination, not only on aggregate benchmarks.
In simple terms
A modality is simply a kind of information, much like our senses. Reading words, looking at a picture and listening to a sound are three different modalities. Each one arrives in a different shape, so a computer has to learn to understand each of them separately.
A multimodal AI system is one that can work with several of these kinds of information together. The key idea is translation: the system turns words, pictures and sounds into one common internal language so that it can connect them, for example by explaining what is happening in a photo.
A practical example
An insurance company builds a multimodal claims assistant for car damage. Customers upload two or three photos of the vehicle, a short written description and sometimes a voice note recorded at the scene. The team first maps the modalities: images show the damage, text gives the policy context, and audio adds the customer account.
Testing shows that the vision-language model often answers from the written description alone, even when the photos contradict it. The engineers add training examples where text and images disagree and evaluate image-only accuracy separately. They also transcribe voice notes with a speech recognition model rather than sending raw audio, which cuts cost per claim by roughly a third. The final system weighs the photos properly and flags mismatches for a human adjuster instead of trusting the text.