Visual prompting

On this page 7

Visual prompting is the practice of instructing an AI model with visual inputs such as points, boxes, masks, drawn marks or example images, either instead of text or together with it. In the field of prompting within artificial intelligence, it extends the idea of a prompt beyond words: the user shows the model where to look and what to produce by annotating or supplying images.

The technique serves three broad families of systems: promptable segmentation models that cut out objects from a click, multimodal language models that reason over annotated screenshots and photos, and image generators that take a reference picture as part of the instruction. In each case, the visual prompt carries spatial or stylistic information that text expresses poorly.

How visual prompting conveys intent

Visual prompting works because a prompt encoder turns geometric or pictorial input into embeddings that the model can combine with image features. A click becomes a positional embedding, a box becomes two corner embeddings, and a mask becomes a downsampled feature map that is added to the image representation.

In promptable segmentation, the best-known design pairs a heavy image encoder that runs once per image with a lightweight mask decoder that runs for every new prompt. This split lets an annotator add or remove points interactively, because each refinement costs only tens of milliseconds on a modern GPU rather than a full forward pass.

Multimodal language models take a different route. They receive the annotated image itself as pixels, so a red circle, an arrow or a numbered label becomes part of what the vision encoder sees. The model then links the visible mark to the words of the question, which is why clear, high-contrast marks matter more than precise coordinates.

The table below summarizes the main prompt types and the kind of information each one carries.

Prompt typeInformation carriedTypical model family
Point clickLocation of one object, positive or negativePromptable segmentation
Bounding boxExtent and position of a regionSegmentation and detection
Mask or scribbleExact or rough shape of a regionSegmentation and image editing
Drawn mark or arrowWhich element a question refers toMultimodal language models
Numbered overlay labelsAddressable IDs for many regionsMultimodal language models and agents
Reference imageStyle, identity or composition to followImage generators
Input-output image pairThe transformation to apply to a new imageVisual in-context learners
Common visual prompt types

Main variants of visual prompting

Visual prompting covers several distinct methods that share one principle: the image carries part of the instruction. Practitioners usually distinguish them by what the model must output and how the visual cue is delivered.

  • Interactive segmentation prompts select an object with clicks or boxes and return a mask; negative clicks remove wrongly included areas.
  • Set-of-mark prompting overlays numbered labels on regions produced by a segmenter, so a multimodal model can answer with a label number instead of vague descriptions.
  • Visual in-context learning shows one or more example image pairs and a query image, and the model infers the task, such as colorization or keypoint marking, from the examples alone.
  • Image prompts for generation feed a reference picture through an adapter so that the output follows its style or subject, often weighted against the text prompt.
  • Learned visual prompts are trainable pixel patterns or padding added to inputs, a parameter-efficient adaptation method rather than a user-facing instruction.

The last item deserves care: in research literature, visual prompt tuning means learning prompt tokens or pixels by gradient descent. It shares the name but is a training technique, not something a user draws or uploads.

Visual prompting compared with text prompting

Visual prompting complements rather than replaces text, because each modality is precise about different things. Text excels at goals, constraints and output format, while images excel at location, shape, appearance and layout.

The comparison below shows where each approach is strongest and how the two are usually combined in production pipelines.

AspectText promptVisual prompt
Specifying locationAmbiguous, relies on spatial wordsExact, a click or box
Describing appearanceLossy, hard to name texturesDirect, the example shows it
Stating the taskClear and explicitImplicit, inferred from examples
Output format controlStrong, via format instructionsWeak without accompanying text
ReproducibilityHigh, text is easy to versionModerate, depends on image and coordinates
Effort for the userTyping onlyAnnotation tools or reference assets
Strengths of text and visual prompts

Neighboring techniques in the prompting toolbox interact with visual cues. A negative prompt in image generation removes unwanted content, prompt weighting balances an image prompt against text, and an output format instruction tells a multimodal model to answer with mark numbers or coordinates.

Designing effective visual prompts

Effective visual prompts are unambiguous, sparse and consistent with the accompanying text. A small number of well-placed cues usually outperforms a cluttered image, because each extra mark can occlude detail or compete for the model’s attention.

The following decision table maps common goals to a suitable prompt choice.

GoalRecommended promptReason
Isolate one object quicklyOne positive click, then refineMinimal effort, fast feedback
Select an object among similar onesTight bounding boxRemoves ambiguity between neighbors
Ask about a UI elementNumbered overlay labelsModel answers with a stable identifier
Match a brand lookReference image with moderate weightKeeps style while text sets content
Teach a custom image transformationTwo or three example pairsModel infers the mapping from examples
Choosing a visual prompt for the goal

Practical rules apply across model families. Use high-contrast colors that do not appear in the scene, keep labels large enough to survive downscaling, place marks beside rather than on top of small details, and always restate in text what the marks mean, for example that a red box marks the defect to classify.

Limitations and common failure modes

Visual prompting inherits the weaknesses of the underlying vision encoder, so small objects, low resolution and dense scenes remain the main sources of error. Many multimodal models resize inputs to a fixed resolution, which can make thin marks or small numbers unreadable.

SymptomLikely causeFix
Mask includes the backgroundSingle ambiguous clickAdd a negative click or a box
Model ignores the drawn arrowMark too thin after resizingThicken lines, crop the region
Answer cites the wrong labelOverlapping or crowded labelsFewer regions, offset labels
Output copies the reference too closelyImage prompt weight too highLower the weight, strengthen text
Inconsistent results across runsSampling randomness and prompt sensitivityFix seeds, test several variants
Symptoms, causes and fixes

There are also governance concerns. Reference images can carry copyrighted styles or identifiable faces, and annotated screenshots sent to hosted models may expose confidential on-screen data. Teams that use visual prompts at scale should treat them as inputs subject to the same review as text prompts, including prompt evaluation on a fixed test set.

Explained without jargon

Visual prompting means showing an AI system what you mean instead of only telling it. Imagine asking a colleague to fix something on a printed page. You could describe it in a long sentence, or you could simply circle it with a red pen. Circling is faster and leaves no doubt.

AI models can now read those circles, arrows, boxes and example pictures. A click on a photo can tell a tool which object to cut out, and a sample image can tell an image generator what style to copy. The picture becomes part of the instruction, and words fill in what the picture cannot say.

A typical use case

A quality team at an electronics assembly plant inspects circuit boards with a multimodal model. Inspectors photograph each board at about 12 megapixels, and an automatic segmenter outlines every visible component. A script then overlays numbered labels, from 1 to roughly 40, next to each outlined part.

The model receives the labeled image with a short text instruction: list the label numbers of any components that look misaligned, burned or missing, and answer as a comma-separated list. Because the model can refer to stable numbers rather than vague positions, its answers map directly to the bill of materials.

During a pilot, the team found that labels shrank too much when the model resized the photo, so they split each board into four cropped quadrants. With that change, inspectors reported that review time per board fell noticeably and that most flagged items matched real defects, with a human still confirming every rejection.