Zero-shot image classification

On this page 7

Zero-shot image classification is the task of assigning an image to one of a set of categories without any labeled training examples for those categories, usually by comparing the image with natural-language descriptions of each class. In multimodal AI, it is the signature capability of contrastive vision-language models such as CLIP and SigLIP, which learn a shared embedding space for images and text from hundreds of millions of image-text pairs.

The approach replaces the fixed output layer of a conventional classifier with a list of text prompts written at inference time. Changing the label set therefore requires no retraining, which makes the technique valuable for rapid prototyping, content moderation, data curation and any domain where collecting labeled images is slow or expensive.

How zero-shot image classification works

Zero-shot image classification works by encoding the image and every candidate label into the same vector space and choosing the label whose embedding is most similar to the image embedding. A vision encoder, typically a Vision Transformer, produces one image vector, and a text encoder produces one vector per class prompt.

Both vectors are normalized, and the model computes cosine similarity between the image and each prompt. The similarities are multiplied by a learned temperature parameter and passed through a softmax, which yields a probability distribution over the candidate classes. The class with the highest score is the prediction.

The capability comes from contrastive pretraining. During training, the model sees batches of matching image-caption pairs and learns to pull each correct pair together while pushing mismatched pairs apart. Because captions describe objects, scenes, styles and actions in open vocabulary, the resulting space generalizes to concepts that were never an explicit training label.

The following table summarizes the pipeline stages and what each one contributes.

StageWhat happensTypical choice
Prompt constructionEach class name is inserted into a template“a photo of a {class}”
Text encodingPrompts become normalized text embeddingsTransformer text encoder
Image encodingThe image becomes one normalized embeddingViT-B/16 or ViT-L/14
Similarity scoringCosine similarity per class, scaled by temperatureDot product of unit vectors
DecisionSoftmax or sigmoid scores select the labelTop-1 or thresholded multi-label
Stages of a zero-shot classification pipeline

Prompt engineering and ensembling

The wording of class prompts has a measurable effect on accuracy, because the text encoder learned from web captions rather than from bare category names. A bare label such as “crane” is ambiguous between a bird and a machine, while “a photo of a crane, a type of bird” resolves the ambiguity.

Prompt ensembling averages the text embeddings of several templates per class, for example “a photo of a {class}”, “a blurry photo of a {class}” and “a close-up photo of a {class}”. The averaged vector is a more robust class prototype, and ensembles of dozens of templates typically add a few percentage points of top-1 accuracy on standard benchmarks.

More recent practice uses a large language model to generate descriptive prompts, such as visual attributes of each class. This helps fine-grained categories but can also introduce attributes that are wrong or irrelevant, so generated descriptions need human review in production systems.

StrategyExampleBenefitDrawback
Bare class name“golden retriever”Simplest to set upLowest and least stable accuracy
Single template“a photo of a golden retriever”Matches caption styleSensitive to wording
Template ensemble10 to 80 templates averagedMore robust prototypesMore text encoding at setup
Domain context“a satellite photo of a harbor”Fits unusual imageryRequires domain knowledge
LLM-generated descriptionsLists of visual attributesHelps fine-grained classesRisk of incorrect attributes
Common prompt strategies and their trade-offs

Comparison with other classification approaches

Zero-shot image classification differs from supervised and few-shot methods mainly in where class knowledge comes from: text descriptions instead of labeled images. This makes it the fastest route to a working classifier, but rarely the most accurate one for a narrow, well-defined task.

A related option is the linear probe, which trains only a small classifier on top of frozen image embeddings. With even a handful of labeled images per class, linear probes and adapter methods often overtake zero-shot performance on specialized data. A multimodal large language model can also classify images by answering a question, but it is slower and its output is harder to calibrate.

ApproachLabeled images neededLabel set changeTypical strength
Supervised fine-tuningHundreds to thousands per classRequires retrainingHighest in-domain accuracy
Few-shot linear probeAbout 1 to 16 per classRetrain small headCheap gains on specialized data
Zero-shot classificationNoneEdit the prompt listOpen vocabulary and speed
Generative VLM promptingNoneEdit the questionFlexible reasoning, slower inference
Zero-shot classification compared with related approaches

Evaluation and typical performance

Zero-shot classifiers are evaluated by top-1 and top-5 accuracy on benchmarks whose training splits the model never used, most often ImageNet and its distribution-shifted variants. Large contrastive models reach roughly 75 to 85 percent zero-shot top-1 accuracy on ImageNet, depending on model size, training data and prompt strategy.

A key finding is robustness under distribution shift. Supervised ImageNet models lose much of their accuracy on sketches, renditions or adversarially filtered photos, while zero-shot models trained on diverse web data degrade considerably less. Accuracy is still much lower on specialized domains such as medical scans, satellite imagery or industrial defects, where web captions rarely describe the relevant visual differences.

Evaluation must also check for data contamination. Web-scale pretraining sets can contain near-duplicates of benchmark images, which inflates reported zero-shot scores, so careful studies deduplicate against test sets before reporting results.

Limitations and common mistakes

The main limitation is that zero-shot image classification can only recognize what the pretraining captions described often enough. Rare concepts, fine-grained distinctions, counting and spatial relations remain weak, and the model inherits social biases present in web data.

Practitioners also fall into recurring traps, summarized below with their causes and fixes.

SymptomLikely causeRecommended fix
Confident wrong label on unrelated imagesSoftmax forces a choice among given classesAdd a background class or use thresholds
Accuracy changes with minor wordingPrompt sensitivityUse template ensembles
Text in the image decides the labelTypographic attack vulnerabilityFilter or mask overlaid text
Poor results on medical or aerial imagesDomain gap from web captionsUse domain models or a linear probe
Uneven accuracy across demographic groupsBias inherited from pretraining dataAudit per group before deployment
Frequent problems in deployment

The most important warning concerns closed-set softmax scoring. Probabilities are only relative to the supplied label list, so a high score does not mean the image truly belongs to a class. Production systems should treat scores as rankings rather than calibrated confidence unless they are calibrated on held-out data.

A non-technical view

Imagine a person who has read millions of picture captions on the internet. Show that person a new photo and a short list of possible names, and they can pick the best match even if nobody ever trained them on that exact list. That is what zero-shot image classification does.

The computer turns the picture and each written label into a kind of numeric fingerprint, then checks which label fingerprint sits closest to the picture. Because the labels are just words, anyone can change the categories by typing new ones. The catch is that the system only knows what it saw described often, so it struggles with rare or highly technical images.

A concrete example

A secondhand marketplace receives about 40,000 new product photos per day and wants to route each listing into one of 120 categories, from “leather handbag” to “road bicycle”. The team has no labeled dataset for its new category tree, so an engineer deploys an open CLIP-style model with ViT-L/14 and writes prompts such as “a product photo of a road bicycle”, averaging six templates per category.

On a hand-checked sample of 2,000 listings, the zero-shot system assigns the correct category about 80 percent of the time and puts it in the top three for roughly 94 percent. The team uses the top three predictions as suggestions in the seller interface and sends low-score images to human review. After a month, the corrected labels give enough data for a linear probe, which raises top-1 accuracy further without changing the underlying model.