Artificial intelligence

Multimodal AI

Models that understand or generate several kinds of data at once, such as text, images, audio and video.

Browse A–Z

Terms

55 terms

  • Any-to-any model

    An any-to-any model is a multimodal AI system that accepts and generates any mix of text, images, audio and video within one unified model.

  • Audio captioning

    Audio captioning is the multimodal AI task of generating a natural-language sentence that describes the sounds, events and acoustic scene in an audio clip.

  • Audio language model

    An audio language model is a multimodal AI system that processes sound as token sequences to understand, describe or generate speech, music and audio.

  • Audio tokenizer

    An audio tokenizer converts continuous sound into a sequence of discrete tokens that language models can read, predict and turn back into audio.

  • Audio-visual learning

    Audio-visual learning trains models to understand sound and moving images together, using their natural alignment as a rich supervisory signal.

  • Audio-visual speech recognition AVSR

    Audio-visual speech recognition (AVSR) transcribes speech by combining the audio signal with video of the speaker's lips, staying accurate in heavy noise.

  • Chart understanding

    Chart understanding is the multimodal AI capability to read charts and plots, extract their data and answer questions about trends and comparisons.

  • CLIP

    CLIP is a model that learns a shared embedding space for images and text by contrastive training on image-caption pairs, enabling zero-shot recognition.

  • Cross-modal generation

    Cross-modal generation is the creation of content in one modality, such as images or audio, from input in another, such as a text prompt.

  • Cross-modal retrieval

    Cross-modal retrieval finds relevant items in one data type, such as images, using a query from another, such as text, through shared embeddings.

  • Document understanding

    Document understanding is the multimodal AI task of reading scanned or digital documents and extracting their text, layout, structure and meaning.

  • Early fusion

    Early fusion in multimodal AI combines raw or low-level features from several modalities into one input before a single model learns from them jointly.

  • Gated cross-attention

    Gated cross-attention is a layer that lets a frozen language model attend to visual features through a learnable gate that starts closed at zero.

  • GUI grounding

    GUI grounding is the multimodal AI task of mapping a natural-language instruction to the exact on-screen element or coordinates it refers to.

  • Image tokenizer

    An image tokenizer converts an image into a sequence of discrete or continuous tokens that a transformer can process alongside text in multimodal AI.

  • Image-text matching ITM

    Image-text matching (ITM) is the multimodal AI task of deciding whether an image and a text describe the same content and scoring how well they correspond.

  • Image-text pair dataset

    An image-text pair dataset is a training corpus in multimodal AI that links each image to a caption or alt text so models learn shared visual…

  • Interleaved image-text data

    Interleaved image-text data is training data in which images and text alternate in natural document order, used to pretrain multimodal AI models.

  • LAION-5B

    LAION-5B is an open dataset of about 5.85 billion image-text pairs from the web, filtered with CLIP and used to train multimodal AI models.

  • Late fusion

    Late fusion in multimodal AI trains a separate model for each modality and combines their predictions or scores only at the final decision stage.

  • Modality

    A modality in multimodal AI is a distinct kind of data, such as text, images, audio or video, with its own structure, encoder and tokenization.

  • Multimodal agent

    A multimodal agent is an AI system that perceives text, images, audio or screens, plans steps and uses tools to complete tasks on its own.

  • Multimodal chain-of-thought

    Multimodal chain-of-thought is a reasoning method in which an AI model writes intermediate steps grounded in images, text and other inputs before answering.

  • Multimodal embedding

    A multimodal embedding maps text, images, audio or video into one shared vector space so that items with the same meaning lie close together.

  • Multimodal emotion recognition

    Multimodal emotion recognition infers a person's affective state by combining signals such as facial expression, voice, spoken words and physiology.

  • Multimodal fusion

    Multimodal fusion is the set of methods that combine text, image, audio or video representations so a model can reason over them jointly.

  • Multimodal hallucination

    Multimodal hallucination is output from a vision or audio language model that contradicts or invents content absent from the image, audio or video input.

  • Multimodal in-context learning

    Multimodal in-context learning lets a model adapt to a new task from a few interleaved image-text examples in its prompt, without updating its weights.

  • Multimodal knowledge graph

    A multimodal knowledge graph links entities and relations to images, audio, video and text, giving multimodal AI structured, grounded facts to reason over.

  • Multimodal large language model MLLM

    A multimodal large language model (MLLM) is a language model that reads and reasons over text together with images, audio or video in one system.

  • Multimodal pretraining

    Multimodal pretraining trains one model on large paired or interleaved image, text, audio or video data so it learns shared representations across modalities.

  • Multimodal projector

    A multimodal projector is the trainable connector that maps vision encoder features into the token embedding space of a large language model.

  • Multimodal RAG

    Multimodal RAG grounds an AI model's answers in retrieved images, tables, audio, video and text instead of relying on text passages alone.

  • Multimodal reasoning

    Multimodal reasoning is the ability of an AI model to draw conclusions by combining evidence from text, images, audio, video and other data types.

  • Multimodal sentiment analysis

    Multimodal sentiment analysis infers opinion and attitude by combining text, voice and facial cues instead of reading words alone.

  • Native multimodality

    Native multimodality is the design of an AI model trained from the start on text, images, audio and video together rather than joined from separate parts.

  • SigLIP

    SigLIP is a contrastive image-text pre-training method that scores each image-caption pair with a sigmoid loss instead of a batch-wide softmax.

  • Spatial reasoning

    Spatial reasoning in multimodal AI is the ability of a model to infer positions, distances, orientation and layout of objects from images, video or 3D data.

  • Speech-to-speech model

    A speech-to-speech model is an AI system that takes spoken audio as input and produces spoken audio as output, often without an intermediate text step.

  • Talking head generation

    Talking head generation synthesizes video of a speaking face from a portrait and an audio or text input, matching lip motion, expression and head pose.

  • Temporal grounding

    Temporal grounding is the multimodal AI task of finding the start and end times of a video moment that matches a natural-language query.

  • Video captioning

    Video captioning is the multimodal AI task of generating natural-language descriptions of the events, objects and actions shown in a video clip.

  • Video question answering VideoQA

    Video question answering (VideoQA) is the multimodal AI task of answering natural-language questions about the content and events of a video clip.

  • Video tokenizer

    A video tokenizer compresses video clips into compact spatiotemporal tokens that multimodal AI models can understand, predict and generate.

  • Video understanding

    Video understanding is the multimodal AI capability to interpret moving images over time, recognizing actions, events, objects and their temporal relations.

  • Video-language model

    A video-language model is a multimodal AI system that encodes video frames over time and reasons about them in natural language to describe or locate events.

  • Vision encoder

    A vision encoder is the neural network in multimodal AI that turns images into feature vectors a language model or retrieval system can use.

  • Vision-language model VLM

    A vision-language model (VLM) is a multimodal AI system that jointly processes images and text to describe, answer questions about and reason over visual input.

  • Visual dialog

    Visual dialog is a multimodal AI task in which a model answers a sequence of related questions about an image, using both the picture and the…

  • Visual entailment

    Visual entailment is a multimodal AI task in which a model decides whether an image supports, contradicts or leaves undecided a text hypothesis.