Cross-modal retrieval

Also known as: image-text retrieval

On this page 7

Cross-modal retrieval is the task of finding relevant items in one modality, such as images, audio or video, using a query expressed in a different modality, most often natural-language text. In multimodal AI, it is usually solved by mapping every input into a shared embedding space where semantically related content lies close together, so that a sentence and a matching photograph receive similar vectors.

The best-known case is image-text retrieval, which covers both text-to-image search and image-to-text search. The same principle powers video search by description, audio retrieval from spoken or written queries, product search from a photo and the retrieval stage of multimodal RAG systems.

How cross-modal retrieval works

Cross-modal retrieval works by encoding queries and candidates with modality-specific encoders and comparing the resulting vectors with a similarity function, typically cosine similarity. A text encoder and a vision encoder are trained jointly so that paired examples score higher than unpaired ones, which aligns the two modalities without requiring explicit labels.

The dominant training method is contrastive learning on large image-text pair datasets. In a batch of N pairs, the model learns to identify the N correct matches among N squared combinations, using an InfoNCE-style loss as in CLIP or a pairwise sigmoid loss as in SigLIP. Once trained, the encoders run independently, so the entire collection can be embedded once and stored in an index.

The following table summarizes the main stages of a production retrieval pipeline and what each stage controls.

StageWhat happensMain design choice
EncodingEach item is converted to a fixed-length vectorEncoder model and embedding dimension
IndexingVectors are stored for fast searchExact search or approximate nearest neighbor index
Query embeddingThe query is encoded in the same spacePrompt wording and text normalization
Candidate searchTop-k nearest vectors are returnedValue of k and similarity metric
RerankingA slower model rescores the shortlistCross-encoder or multimodal LLM judge
Stages of a cross-modal retrieval pipeline

Dual encoders versus cross-encoders

Cross-modal retrieval architectures fall into two families that trade speed against accuracy. A dual encoder embeds each side separately and compares vectors, while a cross-encoder feeds the query and candidate together through a fusion model that attends across both inputs before producing a score.

Dual encoders scale to billions of items because candidate embeddings are precomputed, but they compress each item into one vector and miss fine-grained relations such as counting, negation or spatial layout. Cross-encoders capture those details through token-level interaction, yet they must run once per query-candidate pair, which makes them suitable only for reranking a short list.

ArchitectureInteraction between modalitiesTypical roleMain limitation
Dual encoderNone until the final dot productFirst-stage search over large collectionsWeak on compositional queries
Cross-encoderFull cross-attention at every layerReranking the top 20 to 200 resultsCost grows with every candidate
Late-interaction modelToken-level matching after separate encodingMiddle ground for documents and pagesLarger index storage
Generative retrieverModel generates identifiers or captionsResearch and niche catalogsHard to update with new items
Comparison of retrieval architectures

Most deployed systems therefore use a two-stage design: a dual encoder retrieves a few hundred candidates in milliseconds, and a cross-encoder or a multimodal large language model reorders them.

Cross-modal retrieval is defined by the direction of the search, meaning which modality forms the query and which forms the collection. Each direction has its own difficulties, because a short caption carries far less information than a detailed image or a long video.

DirectionQuery exampleTypical application
Text to imageA red bicycle leaning on a brick wallStock photo and asset search
Image to textA photo of a damaged partFinding manuals or catalog entries
Text to videoGoal scored from a corner kickMedia archive and moment search
Text to audioRain on a metal roofSound effect libraries
Image to image across domainsA sketch of a chairProduct search from drawings
Common retrieval directions

The task is closely related to image-text matching, which scores whether a given pair belongs together, and to multimodal embedding, which is the representation that retrieval relies on. Retrieval differs from visual question answering because it returns existing items rather than generating an answer.

Measuring retrieval quality

Retrieval quality is measured with ranking metrics that check whether correct items appear near the top of the results. The standard academic benchmarks are MS-COCO and Flickr30K for image-text retrieval and MSR-VTT for text-to-video retrieval, each reported in both directions.

MetricWhat it measuresWhen it is most useful
Recall@KShare of queries with a correct item in the top KBenchmarks with one or few correct answers
Mean reciprocal rankAverage inverse position of the first correct hitSearch where the first result matters most
nDCGGraded relevance weighted by positionCollections with many partly relevant items
Median rankTypical position of the correct itemComparing weak and strong models quickly
Core evaluation metrics

Benchmark scores often overstate real-world performance. Public datasets contain short, generic captions and only one labeled match per query, so a model may be penalized for returning a valid but unlabeled image, and it may still fail on domain-specific vocabulary such as medical or industrial terms.

Limitations and common failure modes

The central weakness of embedding-based cross-modal retrieval is that a single vector cannot hold every detail of a rich image or video. Practitioners encounter a recurring set of problems when moving from demos to production.

  • Compositional errors: a query for a dog chasing a cat may return a cat chasing a dog, because word order is weakly encoded.
  • Negation blindness: queries such as a kitchen without people often retrieve kitchens with people.
  • Modality gap: text and image embeddings occupy separate regions of the space, which distorts thresholds and mixed-modality ranking.
  • Dataset bias: web-scale training pairs inherit stereotypes and favor frequent concepts over rare ones.
  • Domain shift: general models trained on web photos perform poorly on scans, charts, X-rays or satellite imagery.

Common remedies include fine-tuning on in-domain pairs, adding a reranking stage, using hard negative examples during training and combining embedding search with metadata filters or keyword search in a hybrid system.

The short version

Cross-modal retrieval lets a computer find a picture from a sentence, or a sentence from a picture. It works because the system has learned to translate both words and images into the same kind of numeric fingerprint. Things that mean the same get similar fingerprints, so searching becomes a matter of finding the closest match.

Think of a librarian who understands both books and photographs equally well and files them on the same shelves by meaning. You describe what you want, and the right image appears, even if nobody ever tagged it with those words.

Where it makes a difference

A regional television broadcaster holds about 400,000 hours of archived news footage, much of it with no transcripts and only basic dates as metadata. Editors used to spend hours scrubbing through tapes to find shots such as a flooded street at night or a crowd outside a courthouse.

The archive team samples one frame every two seconds, embeds the frames with an image encoder and stores the vectors in an approximate nearest neighbor index. An editor types a description, the system returns the 200 closest frames in under a second, and a vision-language reranker reorders the top 50 by how well each frame fits the full query.

In internal testing on 300 real editor requests, a usable clip appeared on the first results page for most queries, and average search time dropped from hours to a few minutes. Failures clustered around queries with negation and precise counts, so the team added date and location filters to narrow results before the visual search runs.