Models that understand or generate several kinds of data at once, such as text, images, audio and video.
55 terms
An any-to-any model is a multimodal AI system that accepts and generates any mix of text, images, audio and video within one unified model.
Audio captioning is the multimodal AI task of generating a natural-language sentence that describes the sounds, events and acoustic scene in an audio clip.
An audio language model is a multimodal AI system that processes sound as token sequences to understand, describe or generate speech, music and audio.
An audio tokenizer converts continuous sound into a sequence of discrete tokens that language models can read, predict and turn back into audio.
Audio-visual learning trains models to understand sound and moving images together, using their natural alignment as a rich supervisory signal.
Audio-visual speech recognition (AVSR) transcribes speech by combining the audio signal with video of the speaker's lips, staying accurate in heavy noise.
Chart understanding is the multimodal AI capability to read charts and plots, extract their data and answer questions about trends and comparisons.
CLIP is a model that learns a shared embedding space for images and text by contrastive training on image-caption pairs, enabling zero-shot recognition.
Cross-modal generation is the creation of content in one modality, such as images or audio, from input in another, such as a text prompt.
Cross-modal retrieval finds relevant items in one data type, such as images, using a query from another, such as text, through shared embeddings.
Document understanding is the multimodal AI task of reading scanned or digital documents and extracting their text, layout, structure and meaning.
Early fusion in multimodal AI combines raw or low-level features from several modalities into one input before a single model learns from them jointly.
Gated cross-attention is a layer that lets a frozen language model attend to visual features through a learnable gate that starts closed at zero.
GUI grounding is the multimodal AI task of mapping a natural-language instruction to the exact on-screen element or coordinates it refers to.
An image tokenizer converts an image into a sequence of discrete or continuous tokens that a transformer can process alongside text in multimodal AI.
Image-text matching (ITM) is the multimodal AI task of deciding whether an image and a text describe the same content and scoring how well they correspond.
An image-text pair dataset is a training corpus in multimodal AI that links each image to a caption or alt text so models learn shared visual…
Interleaved image-text data is training data in which images and text alternate in natural document order, used to pretrain multimodal AI models.
LAION-5B is an open dataset of about 5.85 billion image-text pairs from the web, filtered with CLIP and used to train multimodal AI models.
Late fusion in multimodal AI trains a separate model for each modality and combines their predictions or scores only at the final decision stage.
A modality in multimodal AI is a distinct kind of data, such as text, images, audio or video, with its own structure, encoder and tokenization.
A multimodal agent is an AI system that perceives text, images, audio or screens, plans steps and uses tools to complete tasks on its own.
Multimodal chain-of-thought is a reasoning method in which an AI model writes intermediate steps grounded in images, text and other inputs before answering.
A multimodal embedding maps text, images, audio or video into one shared vector space so that items with the same meaning lie close together.
Multimodal emotion recognition infers a person's affective state by combining signals such as facial expression, voice, spoken words and physiology.
Multimodal fusion is the set of methods that combine text, image, audio or video representations so a model can reason over them jointly.
Multimodal hallucination is output from a vision or audio language model that contradicts or invents content absent from the image, audio or video input.
Multimodal in-context learning lets a model adapt to a new task from a few interleaved image-text examples in its prompt, without updating its weights.
A multimodal knowledge graph links entities and relations to images, audio, video and text, giving multimodal AI structured, grounded facts to reason over.
A multimodal large language model (MLLM) is a language model that reads and reasons over text together with images, audio or video in one system.
Multimodal pretraining trains one model on large paired or interleaved image, text, audio or video data so it learns shared representations across modalities.
A multimodal projector is the trainable connector that maps vision encoder features into the token embedding space of a large language model.
Multimodal RAG grounds an AI model's answers in retrieved images, tables, audio, video and text instead of relying on text passages alone.
Multimodal reasoning is the ability of an AI model to draw conclusions by combining evidence from text, images, audio, video and other data types.
Multimodal sentiment analysis infers opinion and attitude by combining text, voice and facial cues instead of reading words alone.
Native multimodality is the design of an AI model trained from the start on text, images, audio and video together rather than joined from separate parts.
SigLIP is a contrastive image-text pre-training method that scores each image-caption pair with a sigmoid loss instead of a batch-wide softmax.
Spatial reasoning in multimodal AI is the ability of a model to infer positions, distances, orientation and layout of objects from images, video or 3D data.
A speech-to-speech model is an AI system that takes spoken audio as input and produces spoken audio as output, often without an intermediate text step.
Talking head generation synthesizes video of a speaking face from a portrait and an audio or text input, matching lip motion, expression and head pose.
Temporal grounding is the multimodal AI task of finding the start and end times of a video moment that matches a natural-language query.
Video captioning is the multimodal AI task of generating natural-language descriptions of the events, objects and actions shown in a video clip.
Video question answering (VideoQA) is the multimodal AI task of answering natural-language questions about the content and events of a video clip.
A video tokenizer compresses video clips into compact spatiotemporal tokens that multimodal AI models can understand, predict and generate.
Video understanding is the multimodal AI capability to interpret moving images over time, recognizing actions, events, objects and their temporal relations.
A video-language model is a multimodal AI system that encodes video frames over time and reasons about them in natural language to describe or locate events.
A vision encoder is the neural network in multimodal AI that turns images into feature vectors a language model or retrieval system can use.
A vision-language model (VLM) is a multimodal AI system that jointly processes images and text to describe, answer questions about and reason over visual input.
Visual dialog is a multimodal AI task in which a model answers a sequence of related questions about an image, using both the picture and the…
Visual entailment is a multimodal AI task in which a model decides whether an image supports, contradicts or leaves undecided a text hypothesis.