Multimodal embedding

Also known as: joint embedding space

On this page 7

A multimodal embedding is a dense numerical vector produced by a model that maps inputs from different modalities, such as text, images, audio and video, into a single shared vector space, so that a photo of a dog and the sentence “a dog on a beach” end up as nearby points. In multimodal AI, this joint embedding space is the foundation for cross-modal retrieval, zero-shot image classification, multimodal RAG and many vision-language systems.

The value of a multimodal embedding lies in comparability across modalities: once every item is a vector of the same length, a single distance function, usually cosine similarity, can rank images against a text query, sounds against a picture or video clips against a caption without any modality-specific matching logic.

How a multimodal embedding model aligns modalities

A multimodal embedding model aligns modalities by training separate encoders, one per modality, and pulling the outputs of matching pairs together while pushing mismatched pairs apart. The dominant recipe is the dual-encoder architecture popularized by CLIP: an image encoder (a vision transformer or a convolutional network) and a text encoder (a transformer) each end in a linear projection head that outputs vectors of the same dimension.

Training uses contrastive learning on large image-text pair datasets. In each batch of N pairs, the model computes an N by N similarity matrix and applies the InfoNCE loss, which rewards high similarity on the diagonal (true pairs) and low similarity everywhere else. A learned temperature parameter scales the logits and controls how sharply the model separates positives from negatives. SigLIP replaces the softmax over the batch with a pairwise sigmoid loss, which removes the need for very large batches and global normalization.

The table below summarizes the main components and what each one contributes to the final vector.

ComponentRoleTypical choice
Image encoderTurns pixels into a feature vectorViT-B/16, ViT-L/14, ResNet
Text encoderTurns tokens into a feature vector12 to 24 layer transformer
Projection headMaps both outputs to one dimensionLinear layer to 512 to 1,024 dimensions
NormalizationPlaces vectors on the unit sphereL2 normalization
Training objectiveAligns matching pairsInfoNCE or pairwise sigmoid loss
Components of a typical dual-encoder embedding model

Key properties of the joint embedding space

The quality of a joint embedding space depends on a small set of properties that practitioners tune and measure. Dimensionality sets storage cost and expressive capacity, normalization decides which distance metric is meaningful, and alignment describes how well paired items coincide after training.

PropertyTypical rangeWhat it affects
Vector dimension256 to 1,536Index size, recall and latency
Training pairsHundreds of millions to billionsCoverage of concepts and languages
Batch size4,096 to 32,768 for softmax lossNumber of negatives per step
Precisionfloat32, float16, int8 or binaryMemory per vector and accuracy loss
Context length64 to 512 text tokensHow much descriptive text is encoded
Parameters that shape a multimodal embedding

A well-documented weakness is the modality gap: even after training, image vectors and text vectors tend to occupy two separate cones of the sphere. Cross-modal similarity scores are therefore systematically lower than within-modality scores, so a raw threshold that works for text-to-text search is usually wrong for text-to-image search. Many models also compress fine detail, which limits their grasp of counting, spatial relations and negation.

A multimodal embedding differs from multimodal fusion and from generative multimodal models because it produces independent vectors that are compared afterwards rather than a joint representation computed from both inputs together. This independence is what makes precomputed indexes possible: every image in a catalog can be embedded once and searched millions of times.

ApproachHow inputs interactBest suited for
Dual-encoder embeddingOnly through vector similarityFast retrieval at large scale
Cross-encoder matchingFull attention across both inputsAccurate re-ranking of a short list
Early fusion modelInputs combined before encodingTasks needing fine joint reasoning
Multimodal large language modelVisual tokens fed into a language modelQuestion answering and generation
Unimodal embeddingOne modality onlyText-only or image-only search
How embeddings relate to neighboring techniques

In practice these approaches are stacked. A typical pipeline uses a multimodal embedding to retrieve the top 100 candidates, an image-text matching cross-encoder to re-rank them, and a vision-language model to explain or answer questions about the final results.

Practical uses and model selection

Multimodal embeddings power semantic image search, deduplication of media libraries, content moderation, recommendation, dataset filtering and the retrieval stage of multimodal RAG. They also enable zero-shot classification: each class name is embedded as a text prompt such as “a photo of a {label}”, and an image is assigned to the class whose prompt vector is most similar.

Choosing a model is a trade-off between accuracy, latency and domain fit. The decision table below lists common situations.

SituationRecommended choiceReason
General web-style photosPretrained CLIP or SigLIP modelBroad concept coverage out of the box
Medical, satellite or industrial imagesDomain fine-tuning on paired dataGeneral models miss specialist vocabulary
Multilingual queriesMultilingual text encoder variantEnglish-only encoders degrade sharply
Billions of vectorsLower dimension plus int8 or binary quantizationControls memory and index cost
Scanned documents and chartsDocument-specific visual retrieverText-heavy images need finer detail
Choosing an embedding strategy

Vectors are stored in an approximate nearest neighbor index such as HNSW or IVF with product quantization. An important operational rule is that query and corpus must use the same model version: vectors from two different embedding models are not comparable, so upgrading a model means re-embedding the entire corpus.

Limitations and common mistakes

Multimodal embeddings inherit the biases of web-scale training data, including social stereotypes and an overrepresentation of English and of popular consumer imagery. They also behave like a bag of concepts: a query for “a red cube on a blue sphere” often retrieves images with the colors swapped, because compositional structure is weakly encoded.

  • Mixing model versions in one index silently destroys ranking quality.
  • Reusing text-search thresholds ignores the modality gap and filters out good matches.
  • Skipping normalization makes dot-product scores depend on vector length.
  • Evaluating only on public benchmarks hides poor performance on in-house data.
  • Embedding long documents as one vector loses detail; chunking is usually required.

In simple terms

A multimodal embedding turns pictures, sentences, sounds and video into lists of numbers that work like coordinates on one shared map. Things with the same meaning get coordinates close together. A photo of a cat and the words “a sleeping cat” end up as neighbors. One is an image and the other is text, yet they sit side by side.

Everything lives on the same map. So a computer can find the closest matches to any query, whatever form it takes. You type a few words and get back the right photos. Nobody had to tag those photos by hand.

A practical example

An online furniture retailer has about 2 million product photos but inconsistent text descriptions written by many suppliers. Its search team embeds every photo once with a SigLIP-style model at 768 dimensions, stores the vectors in an HNSW index, and embeds each customer query with the matching text encoder at search time.

A shopper typing “mid-century walnut sideboard with tapered legs” now receives sideboards that look right even when the supplier text says only “cabinet, brown”. The team fine-tunes the model on 300,000 of its own photo and title pairs, re-ranks the top 50 results with a cross-encoder, and measures a clear rise in click-through on search results compared with the old keyword engine, with query latency kept under 100 milliseconds.