Image captioning

On this page 7

Image captioning is the computer vision task of automatically generating a natural-language description of an image, typically one or more sentences that name the main objects, their attributes, the actions taking place and the scene. It sits at the intersection of visual recognition and language generation, which makes it a core benchmark for multimodal models in computer vision and natural language processing.

Unlike image classification, which returns a single label, or object detection, which returns boxes and classes, captioning must decide what is worth mentioning and express it in grammatical, fluent text. That combination of perception, selection and wording explains both its usefulness and its persistent difficulty.

How image captioning models generate text

Most image captioning systems follow an encoder-decoder architecture. A visual encoder converts the image into a set of feature vectors, and a language decoder generates the caption one token at a time, conditioning each word on the image features and on the words already produced. Training minimizes cross-entropy loss against human-written reference captions.

Early neural systems paired a convolutional network encoder with a recurrent decoder such as an LSTM. The introduction of visual attention let the decoder focus on different image regions for each word. Current systems typically use a vision transformer encoder and a transformer or large language model decoder, connected by a projection layer or a lightweight bridging module.

The table below summarizes the main architectural generations and what each contributed.

GenerationEncoder and decoderKey contribution
Template-basedDetectors filling sentence templatesGrammatical but rigid output
CNN with RNNConvolutional encoder, LSTM decoderEnd-to-end learned captions
Attention modelsRegion features with attention decoderWord-level grounding in regions
Transformer modelsTransformer encoder and decoderBetter long-range context
Vision-language modelsVision transformer linked to a large language modelOpen-ended, detailed descriptions
Generations of image captioning architectures

At inference time, the decoder uses a decoding strategy to turn probabilities into text. Beam search with a width of about 3 to 5 favors safe, generic captions, whereas nucleus sampling produces more varied but less predictable wording.

Training data and evaluation metrics

Captioning quality depends heavily on paired image-text data. Curated datasets such as MS COCO Captions and Flickr30k provide about five human-written captions per image, while large web-scraped collections of alt text supply hundreds of millions of noisier pairs used for pretraining. Curated sets teach style and accuracy; web data teaches broad visual vocabulary.

Evaluation is notoriously hard because many different sentences can describe the same image correctly. The following table compares the most widely reported automatic metrics and what each measures.

MetricWhat it measuresMain weakness
BLEUN-gram precision against referencesPenalizes valid paraphrases
METEORMatches with stems and synonymsStill tied to reference wording
ROUGE-LLongest common subsequence overlapInsensitive to factual errors
CIDErConsensus with weighted n-gramsRewards typical dataset phrasing
SPICEOverlap of parsed scene graphsDepends on parser accuracy
CLIPScoreImage-text embedding similarityInherits embedding model biases
Common automatic metrics for caption evaluation

Because no metric fully captures correctness, serious evaluations add human judgment and targeted tests for object hallucination, the generation of objects that are not present in the image.

Image captioning produces free-form language, which distinguishes it from structured outputs in other computer vision tasks. The comparison below shows how captioning relates to its closest neighbors.

TaskOutputTypical use
Image captioningDescriptive sentence or paragraphAlt text, search indexing
Image classificationOne or more category labelsContent sorting
Object detectionBounding boxes with class labelsCounting and localization
OCRTranscribed text found in the imageDocument digitization
Visual question answeringAnswer to a specific questionInteractive assistance
Dense captioningShort captions for many regionsDetailed scene analysis
Output type of captioning and neighboring tasks

In practice these tasks complement one another. A captioning model may internally rely on detection-like features, and OCR output is often fused into the caption when an image contains signs, labels or documents.

Practical applications and configuration choices

Image captioning is deployed wherever images must become searchable, accessible or summarizable text. The most socially important use is generating alternative text for blind and low-vision users, but captioning also supports media asset management, e-commerce catalogs and the creation of training data for other models.

Choosing a configuration means balancing detail, accuracy, latency and cost. The decision table below maps common situations to a suitable approach.

SituationRecommended approachReason
Accessibility alt textConcise caption with human reviewAccuracy matters more than detail
Large photo archive indexingCompact model in batch modeLow cost per image
Product listing descriptionsFine-tuned model on catalog dataConsistent domain vocabulary
Synthetic training data creationLarge vision-language model, detailed promptsRich, varied descriptions
On-device camera featuresSmall distilled modelLow latency and privacy
Choosing a captioning approach by situation

Limitations, biases and failure modes

The most serious limitation of image captioning is hallucination: models confidently mention objects, actions or text that do not appear in the image, usually because those details are statistically common in training captions. A kitchen photo may be captioned with a sink that is not visible simply because kitchens usually contain one.

  • Dataset bias: captions can reproduce stereotypes about gender, age or culture present in the training data.
  • Generic output: beam search and consensus metrics push models toward bland, average descriptions.
  • Weak counting and spatial reasoning: numbers and relations such as left of or behind are often wrong.
  • Domain shift: models trained on everyday photos struggle with medical, satellite or technical imagery.
  • Missing intent: a caption cannot know which detail matters to a particular reader.

These issues mean that captions used for accessibility, legal or medical purposes should be reviewed by people, and that model outputs should never be treated as verified facts about an image.

A plain-language summary

Image captioning is a way for a computer to look at a picture and describe it in a sentence, much as a person would when telling a friend what a photo shows. The system first recognizes what is in the image and then writes words about it.

It is useful for people who cannot see images, for searching large photo collections and for organizing content. The main caution is that descriptions can include things not really there, so important captions should be checked.

An example from practice

A regional news publisher holds an archive of about 400,000 photographs that lack descriptions, which makes them nearly impossible to search and inaccessible to screen reader users. The digital team runs a vision-language captioning model over the archive in overnight batches, generating a one-sentence caption and a longer description for each image.

Editors review a random sample of 1,000 captions and find that most are accurate, but a noticeable share contain invented details such as wrong counts of people. The team therefore marks archive captions as machine-generated, requires human review for any caption published with a new article, and adds OCR text for photos containing signs. Within months, staff search time for archive photos drops substantially, and new articles ship with reviewed alt text.