The Segment Anything Model (SAM) is a promptable image segmentation model released by Meta in 2023 that produces a pixel-accurate mask for an object when a user or program supplies a simple prompt, such as a click, a bounding box or a rough mask. Among AI models & assistants, it stands out as a foundation model for segmentation: it was trained once on a very large dataset and then applied to new images and domains without task-specific retraining.
SAM separates the question of where an object is from what the object is called. It returns object boundaries, not class labels, which makes it a flexible building block for annotation tools, image editors, robotics and pipelines that pair it with a detector or a vision-language model.
How SAM produces a mask from a prompt
SAM produces a mask by combining a heavy image encoder, a lightweight prompt encoder and a fast mask decoder. The image encoder, a Vision Transformer pretrained with masked autoencoding, runs once per image and produces a dense embedding. Prompts are then encoded and passed with that embedding to the decoder, which outputs masks in tens of milliseconds.
This split is the key design decision: the expensive image embedding is computed once, while each new click or box only reruns the small decoder. Interactive tools can therefore respond almost instantly, even in a web browser, after an initial embedding step on a GPU or server.
| Component | Input | Role |
|---|---|---|
| Image encoder | Image resized to 1024 pixels on the long side | Builds a dense image embedding once |
| Prompt encoder | Points, boxes or a coarse mask | Turns prompts into embedding tokens |
| Mask decoder | Image embedding plus prompt tokens | Predicts masks and confidence scores |
Because a single click is often ambiguous, SAM by default returns three candidate masks, for example a shirt, the person wearing it and the whole group, together with a predicted IoU score for each. The application or the user then selects the most suitable candidate.
Training data and the SA-1B dataset
SAM owes its generalization to SA-1B, a dataset of roughly 11 million licensed images and about 1.1 billion masks. The masks were produced through a data engine with three stages: assisted manual annotation, semi-automatic annotation and finally fully automatic mask generation by the model itself.
Each stage used the current model to speed up labeling, and each round of new labels trained a better model. This model-in-the-loop annotation is the reason the dataset reached a scale that no purely manual effort could match, although masks were never given semantic class names.
SAM versions and lightweight variants
The SAM family has grown from a single image model into several generations and many community derivatives. The original release offered three encoder sizes, ViT-B, ViT-L and ViT-H, trading speed for mask quality. SAM 2, released in 2024, extended promptable segmentation to video by adding a streaming memory that tracks an object across frames. SAM 3, announced in late 2025, added concept prompts, so that a short noun phrase or an example image can request every matching instance.
| Model | Prompt types | Main addition |
|---|---|---|
| SAM | Points, boxes, masks | Promptable segmentation of still images |
| SAM 2 | Points, boxes, masks on frames | Video tracking with memory attention |
| SAM 3 | Visual prompts plus noun phrases and exemplars | Segmentation of all instances of a concept |
| Efficient derivatives | Same as the base model | Smaller encoders for mobile and edge use |
| Domain adaptations | Same as the base model | Fine-tuning for medical or remote sensing images |
Community projects such as MobileSAM, EfficientSAM and FastSAM replace or distill the image encoder to cut latency, while adaptations such as MedSAM fine-tune the model on domain-specific imagery where the original data offers little coverage.
How SAM compares with other vision models
SAM differs from classic segmentation networks because it is class-agnostic and prompt-driven, while semantic or instance segmentation models predict a fixed set of categories. It also differs from vision-language models such as LLaVA, which describe or answer questions about an image but usually do not output precise pixel masks.
| Approach | Output | Needs class list | Typical strength |
|---|---|---|---|
| SAM | Masks for prompted objects | No | Generalizes to unseen objects |
| Semantic segmentation model | Class label per pixel | Yes | Consistent labels in a fixed domain |
| Instance segmentation model | Labeled masks per object | Yes | Counting known categories |
| Open-vocabulary detector | Boxes for text queries | No | Finding objects by name |
| Vision-language model | Text answers or captions | No | Describing and reasoning about scenes |
In practice, these models are often chained together: an open-vocabulary detector finds boxes for a phrase such as “solar panel”, and SAM converts each box into a precise mask. This combination is a common pattern in annotation and analytics pipelines.
Practical use and known limitations
SAM is most valuable where precise outlines are needed quickly and labeled training data is scarce, such as dataset annotation, photo editing, background removal and scientific image analysis. Its weights for the original model were released under a permissive open license, which encouraged broad adoption in commercial and research tools.
| Situation | Recommendation | Reason |
|---|---|---|
| Speeding up mask annotation | Use SAM with human review | Clicks replace manual polygon drawing |
| Fixed categories at high volume | Train a dedicated segmentation model | Faster inference and consistent labels |
| Thin or transparent structures | Verify results carefully | Boundaries are often incomplete |
| Medical or satellite imagery | Use a fine-tuned variant | Domain shift lowers accuracy |
| Edge devices | Choose a distilled encoder | The large encoder is too heavy |
Known weaknesses include imprecise fine details such as hair, wires and lattice structures, sensitivity to ambiguous prompts, and a heavy encoder that dominates compute cost. The original model also provides no semantic understanding, so a reliable label must come from another component or from a person.
The short version
The Segment Anything Model is a tool that cuts objects out of pictures with a single click. Point at a cat, and it draws an exact outline around the cat. Draw a box around a car, and it traces the car’s shape.
It does not need to know what the object is called. It simply finds where one thing ends and another begins. That makes it a fast helper for anyone who must outline many objects by hand.
Where it makes a difference
An agricultural research team needs pixel masks of individual leaves in 5,000 greenhouse photos to measure disease spots. Drawing polygons by hand takes an annotator about four minutes per leaf, which would consume months of work.
The team loads the photos into an annotation tool that runs SAM. An annotator clicks once on each leaf, picks the best of three proposed masks and corrects the occasional error with an extra click. Thin stems and overlapping leaf edges still need manual touch-ups.
Average time per leaf falls to well under a minute, and the finished masks are then used to train a small, fast segmentation model for the greenhouse cameras. The decisive benefit was turning outline drawing into review, which made the project feasible within one season.