Segment Anything Model

Also known as: SAM · Segment Anything

On this page 7

The Segment Anything Model (SAM) is a promptable image segmentation model released by Meta in 2023 that produces a pixel-accurate mask for an object when a user or program supplies a simple prompt, such as a click, a bounding box or a rough mask. Among AI models & assistants, it stands out as a foundation model for segmentation: it was trained once on a very large dataset and then applied to new images and domains without task-specific retraining.

SAM separates the question of where an object is from what the object is called. It returns object boundaries, not class labels, which makes it a flexible building block for annotation tools, image editors, robotics and pipelines that pair it with a detector or a vision-language model.

How SAM produces a mask from a prompt

SAM produces a mask by combining a heavy image encoder, a lightweight prompt encoder and a fast mask decoder. The image encoder, a Vision Transformer pretrained with masked autoencoding, runs once per image and produces a dense embedding. Prompts are then encoded and passed with that embedding to the decoder, which outputs masks in tens of milliseconds.

This split is the key design decision: the expensive image embedding is computed once, while each new click or box only reruns the small decoder. Interactive tools can therefore respond almost instantly, even in a web browser, after an initial embedding step on a GPU or server.

ComponentInputRole
Image encoderImage resized to 1024 pixels on the long sideBuilds a dense image embedding once
Prompt encoderPoints, boxes or a coarse maskTurns prompts into embedding tokens
Mask decoderImage embedding plus prompt tokensPredicts masks and confidence scores
Components of the original SAM architecture

Because a single click is often ambiguous, SAM by default returns three candidate masks, for example a shirt, the person wearing it and the whole group, together with a predicted IoU score for each. The application or the user then selects the most suitable candidate.

Training data and the SA-1B dataset

SAM owes its generalization to SA-1B, a dataset of roughly 11 million licensed images and about 1.1 billion masks. The masks were produced through a data engine with three stages: assisted manual annotation, semi-automatic annotation and finally fully automatic mask generation by the model itself.

Each stage used the current model to speed up labeling, and each round of new labels trained a better model. This model-in-the-loop annotation is the reason the dataset reached a scale that no purely manual effort could match, although masks were never given semantic class names.

SAM versions and lightweight variants

The SAM family has grown from a single image model into several generations and many community derivatives. The original release offered three encoder sizes, ViT-B, ViT-L and ViT-H, trading speed for mask quality. SAM 2, released in 2024, extended promptable segmentation to video by adding a streaming memory that tracks an object across frames. SAM 3, announced in late 2025, added concept prompts, so that a short noun phrase or an example image can request every matching instance.

ModelPrompt typesMain addition
SAMPoints, boxes, masksPromptable segmentation of still images
SAM 2Points, boxes, masks on framesVideo tracking with memory attention
SAM 3Visual prompts plus noun phrases and exemplarsSegmentation of all instances of a concept
Efficient derivativesSame as the base modelSmaller encoders for mobile and edge use
Domain adaptationsSame as the base modelFine-tuning for medical or remote sensing images
Main members of the SAM family

Community projects such as MobileSAM, EfficientSAM and FastSAM replace or distill the image encoder to cut latency, while adaptations such as MedSAM fine-tune the model on domain-specific imagery where the original data offers little coverage.

How SAM compares with other vision models

SAM differs from classic segmentation networks because it is class-agnostic and prompt-driven, while semantic or instance segmentation models predict a fixed set of categories. It also differs from vision-language models such as LLaVA, which describe or answer questions about an image but usually do not output precise pixel masks.

ApproachOutputNeeds class listTypical strength
SAMMasks for prompted objectsNoGeneralizes to unseen objects
Semantic segmentation modelClass label per pixelYesConsistent labels in a fixed domain
Instance segmentation modelLabeled masks per objectYesCounting known categories
Open-vocabulary detectorBoxes for text queriesNoFinding objects by name
Vision-language modelText answers or captionsNoDescribing and reasoning about scenes
SAM compared with related approaches

In practice, these models are often chained together: an open-vocabulary detector finds boxes for a phrase such as “solar panel”, and SAM converts each box into a precise mask. This combination is a common pattern in annotation and analytics pipelines.

Practical use and known limitations

SAM is most valuable where precise outlines are needed quickly and labeled training data is scarce, such as dataset annotation, photo editing, background removal and scientific image analysis. Its weights for the original model were released under a permissive open license, which encouraged broad adoption in commercial and research tools.

SituationRecommendationReason
Speeding up mask annotationUse SAM with human reviewClicks replace manual polygon drawing
Fixed categories at high volumeTrain a dedicated segmentation modelFaster inference and consistent labels
Thin or transparent structuresVerify results carefullyBoundaries are often incomplete
Medical or satellite imageryUse a fine-tuned variantDomain shift lowers accuracy
Edge devicesChoose a distilled encoderThe large encoder is too heavy
When SAM fits and when it does not

Known weaknesses include imprecise fine details such as hair, wires and lattice structures, sensitivity to ambiguous prompts, and a heavy encoder that dominates compute cost. The original model also provides no semantic understanding, so a reliable label must come from another component or from a person.

The short version

The Segment Anything Model is a tool that cuts objects out of pictures with a single click. Point at a cat, and it draws an exact outline around the cat. Draw a box around a car, and it traces the car’s shape.

It does not need to know what the object is called. It simply finds where one thing ends and another begins. That makes it a fast helper for anyone who must outline many objects by hand.

Where it makes a difference

An agricultural research team needs pixel masks of individual leaves in 5,000 greenhouse photos to measure disease spots. Drawing polygons by hand takes an annotator about four minutes per leaf, which would consume months of work.

The team loads the photos into an annotation tool that runs SAM. An annotator clicks once on each leaf, picks the best of three proposed masks and corrects the occasional error with an extra click. Thin stems and overlapping leaf edges still need manual touch-ups.

Average time per leaf falls to well under a minute, and the finished masks are then used to train a small, fast segmentation model for the greenhouse cameras. The decisive benefit was turning outline drawing into review, which made the project feasible within one season.