Visual prompting is the practice of instructing an AI model with visual inputs such as points, boxes, masks, drawn marks or example images, either instead of text or together with it. In the field of prompting within artificial intelligence, it extends the idea of a prompt beyond words: the user shows the model where to look and what to produce by annotating or supplying images.
The technique serves three broad families of systems: promptable segmentation models that cut out objects from a click, multimodal language models that reason over annotated screenshots and photos, and image generators that take a reference picture as part of the instruction. In each case, the visual prompt carries spatial or stylistic information that text expresses poorly.
How visual prompting conveys intent
Visual prompting works because a prompt encoder turns geometric or pictorial input into embeddings that the model can combine with image features. A click becomes a positional embedding, a box becomes two corner embeddings, and a mask becomes a downsampled feature map that is added to the image representation.
In promptable segmentation, the best-known design pairs a heavy image encoder that runs once per image with a lightweight mask decoder that runs for every new prompt. This split lets an annotator add or remove points interactively, because each refinement costs only tens of milliseconds on a modern GPU rather than a full forward pass.
Multimodal language models take a different route. They receive the annotated image itself as pixels, so a red circle, an arrow or a numbered label becomes part of what the vision encoder sees. The model then links the visible mark to the words of the question, which is why clear, high-contrast marks matter more than precise coordinates.
The table below summarizes the main prompt types and the kind of information each one carries.
| Prompt type | Information carried | Typical model family |
|---|---|---|
| Point click | Location of one object, positive or negative | Promptable segmentation |
| Bounding box | Extent and position of a region | Segmentation and detection |
| Mask or scribble | Exact or rough shape of a region | Segmentation and image editing |
| Drawn mark or arrow | Which element a question refers to | Multimodal language models |
| Numbered overlay labels | Addressable IDs for many regions | Multimodal language models and agents |
| Reference image | Style, identity or composition to follow | Image generators |
| Input-output image pair | The transformation to apply to a new image | Visual in-context learners |
Main variants of visual prompting
Visual prompting covers several distinct methods that share one principle: the image carries part of the instruction. Practitioners usually distinguish them by what the model must output and how the visual cue is delivered.
- Interactive segmentation prompts select an object with clicks or boxes and return a mask; negative clicks remove wrongly included areas.
- Set-of-mark prompting overlays numbered labels on regions produced by a segmenter, so a multimodal model can answer with a label number instead of vague descriptions.
- Visual in-context learning shows one or more example image pairs and a query image, and the model infers the task, such as colorization or keypoint marking, from the examples alone.
- Image prompts for generation feed a reference picture through an adapter so that the output follows its style or subject, often weighted against the text prompt.
- Learned visual prompts are trainable pixel patterns or padding added to inputs, a parameter-efficient adaptation method rather than a user-facing instruction.
The last item deserves care: in research literature, visual prompt tuning means learning prompt tokens or pixels by gradient descent. It shares the name but is a training technique, not something a user draws or uploads.
Visual prompting compared with text prompting
Visual prompting complements rather than replaces text, because each modality is precise about different things. Text excels at goals, constraints and output format, while images excel at location, shape, appearance and layout.
The comparison below shows where each approach is strongest and how the two are usually combined in production pipelines.
| Aspect | Text prompt | Visual prompt |
|---|---|---|
| Specifying location | Ambiguous, relies on spatial words | Exact, a click or box |
| Describing appearance | Lossy, hard to name textures | Direct, the example shows it |
| Stating the task | Clear and explicit | Implicit, inferred from examples |
| Output format control | Strong, via format instructions | Weak without accompanying text |
| Reproducibility | High, text is easy to version | Moderate, depends on image and coordinates |
| Effort for the user | Typing only | Annotation tools or reference assets |
Neighboring techniques in the prompting toolbox interact with visual cues. A negative prompt in image generation removes unwanted content, prompt weighting balances an image prompt against text, and an output format instruction tells a multimodal model to answer with mark numbers or coordinates.
Designing effective visual prompts
Effective visual prompts are unambiguous, sparse and consistent with the accompanying text. A small number of well-placed cues usually outperforms a cluttered image, because each extra mark can occlude detail or compete for the model’s attention.
The following decision table maps common goals to a suitable prompt choice.
| Goal | Recommended prompt | Reason |
|---|---|---|
| Isolate one object quickly | One positive click, then refine | Minimal effort, fast feedback |
| Select an object among similar ones | Tight bounding box | Removes ambiguity between neighbors |
| Ask about a UI element | Numbered overlay labels | Model answers with a stable identifier |
| Match a brand look | Reference image with moderate weight | Keeps style while text sets content |
| Teach a custom image transformation | Two or three example pairs | Model infers the mapping from examples |
Practical rules apply across model families. Use high-contrast colors that do not appear in the scene, keep labels large enough to survive downscaling, place marks beside rather than on top of small details, and always restate in text what the marks mean, for example that a red box marks the defect to classify.
Limitations and common failure modes
Visual prompting inherits the weaknesses of the underlying vision encoder, so small objects, low resolution and dense scenes remain the main sources of error. Many multimodal models resize inputs to a fixed resolution, which can make thin marks or small numbers unreadable.
| Symptom | Likely cause | Fix |
|---|---|---|
| Mask includes the background | Single ambiguous click | Add a negative click or a box |
| Model ignores the drawn arrow | Mark too thin after resizing | Thicken lines, crop the region |
| Answer cites the wrong label | Overlapping or crowded labels | Fewer regions, offset labels |
| Output copies the reference too closely | Image prompt weight too high | Lower the weight, strengthen text |
| Inconsistent results across runs | Sampling randomness and prompt sensitivity | Fix seeds, test several variants |
There are also governance concerns. Reference images can carry copyrighted styles or identifiable faces, and annotated screenshots sent to hosted models may expose confidential on-screen data. Teams that use visual prompts at scale should treat them as inputs subject to the same review as text prompts, including prompt evaluation on a fixed test set.
Explained without jargon
Visual prompting means showing an AI system what you mean instead of only telling it. Imagine asking a colleague to fix something on a printed page. You could describe it in a long sentence, or you could simply circle it with a red pen. Circling is faster and leaves no doubt.
AI models can now read those circles, arrows, boxes and example pictures. A click on a photo can tell a tool which object to cut out, and a sample image can tell an image generator what style to copy. The picture becomes part of the instruction, and words fill in what the picture cannot say.
A typical use case
A quality team at an electronics assembly plant inspects circuit boards with a multimodal model. Inspectors photograph each board at about 12 megapixels, and an automatic segmenter outlines every visible component. A script then overlays numbered labels, from 1 to roughly 40, next to each outlined part.
The model receives the labeled image with a short text instruction: list the label numbers of any components that look misaligned, burned or missing, and answer as a comma-separated list. Because the model can refer to stable numbers rather than vague positions, its answers map directly to the bill of materials.
During a pilot, the team found that labels shrank too much when the model resized the photo, so they split each board into four cropped quadrants. With that change, inspectors reported that review time per board fell noticeably and that most flagged items matched real defects, with a human still confirming every rejection.