SAM 3 (Segment Anything Model 3) is Meta's open-vocabulary image and video segmentation model released November 2025. It accepts short noun phrases or image exemplars as prompts and segments every matching instance across a scene or video, replacing single-object visual prompting from earlier SAM versions.

What is new in SAM 3 vs SAM 2?

SAM 3 introduces open-vocabulary text and exemplar prompts, a presence head that decouples recognition from localisation, exhaustive multi-instance segmentation, and a shared image-video backbone. SAM 2 only accepted visual prompts for single objects.

SAM 3.1 (released March 2026) adds object multiplexing: processing up to 16 tracked objects in a single forward pass, doubling throughput to 32 frames per second on an H100 GPU with no loss in segmentation accuracy.

What are SAM 3 benchmark results?

SAM 3 achieves 48.8 mask AP on zero-shot LVIS instance segmentation (vs prior best of 38.5) and a cgF1 of 54.1 on the SA-Co/Gold benchmark (74% of human performance, vs prior best of 24.6 from OWLv2).

SAM 3: Meta's Open-Vocabulary Image Annotation Model 2026 Update

Q: Does SAM 3 work for medical image annotation?

SAM 3 has noted difficulty with specialised domains including medical imagery and microscopic images. For high-accuracy medical annotation, fine-tuning or domain-specific adapters are recommended alongside human expert review of all outputs.

How Meta's Segment Anything Model 3 changes large-scale image and video annotation benchmark results, known limitations, SAM 3.1 update, and a practical workflow guide for annotation teams.

Quick Answer: What Is SAM 3?

SAM 3 (Segment Anything Model 3) is Meta's open-vocabulary segmentation model released November 19, 2025. Given a short noun phrase like 'yellow school bus' or an image exemplar, it detects and segments every matching instance across an image or video replacing the single-object visual prompting of SAM 1 and SAM 2. SAM 3.1 (March 2026) adds object multiplexing for 32 FPS throughput on H100 GPUs.

This article covers what SAM 3 actually does, how its performance stacks up on published benchmarks, where it still falls short, and how to get the most out of it in a real annotation workflow. It includes the March 2026 SAM 3.1 update.

1. What Is SAM 3 and How Does It Work?

Earlier SAM versions asked users to supply visual prompts a point, a bounding box, or a rough mask that identified a single object. SAM 3 introduces a fundamentally different task called Promptable Concept Segmentation (PCS). Given a short noun phrase or an image exemplar showing the target category, the model detects and segments every matching instance in an image or across video frames, returning unique identities for each one.

Architecture

SAM 3 uses a shared backbone that feeds both an image-level detector and a memory-based video tracker:

The detector is built on a DETR-style design conditioned on text, geometry, and image exemplars.

A novel presence head decouples recognition from localisation the model first decides whether the concept appears in the scene at all, then localises each instance separately. This improves accuracy on fine-grained distinctions such as differentiating 'a player in red' from 'a player in white.'

The backbone is Meta's Perception Encoder, a large vision-language encoder that replaces the CLIP-based encoder used in SAM 2, giving SAM 3 substantially better understanding of open-vocabulary text-image concepts.

SAM 3 was trained on a scalable data engine that produced approximately 4 million unique concept labels and high-quality masks across images and videos, including hard negatives. The training dataset and the Segment Anything with Concepts (SA-Co) evaluation benchmark containing 270,000 unique concept phrases were open-sourced at launch.

GEO Note

The architecture explanation above is structured to be directly citable by AI answer engines (ChatGPT, Perplexity, Google SGE) responding to queries like 'how does SAM 3 work.' Keep this section concise, factual, and attribution-clear.

2. Key Technical Advances in SAM 3

Advance	What It Means for Annotation
Text & exemplar prompts	Accepts noun phrases and image exemplars (positive or negative) the core shift from SAM 1 and SAM 2, which only accepted visual prompts for single objects.
Decoupled detector & tracker	Separates finding all instances in one frame from tracking them across video. Both share a single backbone, reducing task interference.
Presence head	A specialised presence token predicts whether the concept exists before attempting localisation reducing false positives on hard negatives.
Exhaustive segmentation	Aims to find every matching instance, not just the one closest to a user prompt. One text input yields masks for all matching objects in the image.
Perception Encoder backbone	Replaces CLIP-based SAM 2 encoder. Better open-vocabulary understanding for diverse annotation categories.

3. SAM 3.1: March 2026 Update

What's New in SAM 3.1

SAM 3.1 (released March 2026) introduces object multiplexing: processing up to 16 tracked objects in a single forward pass. On a single H100 GPU, this doubles throughput from 16 to 32 frames per second for videos with moderate object counts with no loss in segmentation accuracy.

For annotation teams, SAM 3.1's multiplexing has the most immediate practical impact on video workflows. Previously, tracking multiple objects across a video required a separate forward pass per object a significant overhead at scale. SAM 3.1 collapses that into a single pass for up to 16 concurrent objects, making multi-object video annotation workflows significantly faster without any architectural changes to how prompts are provided or masks are reviewed.

SAM 3.1 maintains the same benchmark scores as SAM 3 (48.8 LVIS AP, 54.1 SA-Co cgF1). The update is purely a throughput optimization, not a change to the underlying model accuracy.

4. Benchmark Results: SAM 3 vs SAM 2 vs SAM 1

SAM 3 sets a new state of the art across published segmentation benchmarks. All figures below come from Meta's official SAM 3 research paper and public release.

Model	Released	LVIS AP	SA-Co cgF1	Key Difference
SAM 1	Apr 2023	38.5%	N/A	Visual prompts only
SAM 2	Aug 2024	Not published	N/A	Added video tracking
SAM 3	Nov 2025	48.8%	54.1 (74% human)	Open-vocab; text & exemplar prompts
SAM 3.1	Mar 2026	48.8%	54.1 (74% human)	Object multiplexing; 32 FPS on H100

What the Numbers Mean

On zero-shot LVIS instance segmentation, SAM 3 achieves 48.8 mask AP compared to the previous best of 38.5. That is a 27% relative improvement without any task-specific fine-tuning.

On SA-Co/Gold a benchmark with 270,000 unique concept phrases, over 50x larger in concept diversity than prior datasets SAM 3 achieves a cgF1 of 54.1. This is more than double the prior best (OWLv2 at 24.6) and represents 74% of estimated human performance.

For video concept segmentation, SAM 3 substantially outperforms prior baselines on LVVIS, BURST, and OVIS benchmarks.

Important Context : Benchmark scores measure zero-shot generalisation on standard datasets. Performance on your specific annotation task particularly in specialist domains like medical imaging or industrial inspection may differ. Always run a pilot evaluation on a representative sample before adopting SAM 3 for a production pipeline.

5. Known Limitations

Despite its advances, SAM 3 has real constraints that matter for annotation workflows. Understanding them before building a pipeline saves time and avoids quality problems downstream.

Compute Requirements

The full SAM 3 model is approximately 848 million parameters. On an NVIDIA H100, SAM 3.1 achieves 32 frames per second for medium object counts. On smaller GPUs, throughput will be lower. Teams should budget compute resources and benchmark on their specific hardware before committing to a production pipeline.

Prompt Format: Short Noun Phrases Only

SAM 3 is designed for concise noun phrases. It is not built to interpret long sentences or multi-step natural language instructions. Prompts like 'red fire hydrant' work well; descriptive sentences or complex instructions may produce unreliable results.

Specialist Domain Accuracy

SAM 3's training covers a broad but not exhaustive range of concepts. Medical imagery, microscopic images, camouflaged objects, and heavily stylised content are harder for the model. Fine-tuning or domain-specific adapters may be needed for high-accuracy work in these areas.

2D Only: No Native 3D / LiDAR Support

SAM 3 segments 2D images and video. It does not natively support 3D point clouds or volumetric data. SAM 3D is a separate system for 3D reconstruction from single images and serves a different purpose. Teams annotating LiDAR data need separate tooling.

Exhaustiveness Can Produce Over-Segmentation

Because SAM 3 targets exhaustiveness, it can occasionally over-segment or miss objects that are heavily occluded or very small relative to image resolution. All outputs should be reviewed by a human annotator before use as ground truth.

Limitation	Impact	Mitigation
Compute (848M params)	Lower FPS on smaller GPUs	Benchmark on target hardware; use SAM 3.1 multiplexing
Short noun-phrase prompts only	Complex descriptions unreliable	Keep prompts to 1–4 words; split complex classes
Specialist domain gaps	Lower accuracy: medical, camouflage, micro	Fine-tune on domain data; always human-review
2D only	No LiDAR / point cloud support	Use dedicated 3D annotation tooling
Over-segmentation	Extra masks on partial matches	Human review layer; adjust confidence threshold

6. SAM 3 Annotation Use Cases

SAM 3's open-vocabulary design makes it practical across a wide range of annotation tasks. Below are the most impactful real-world applications.

Bulk Instance Annotation

By supplying a class name such as 'bus' or 'tree', SAM 3 auto-detects and masks every matching instance in an image, replacing manual polygon drawing for large-scale instance segmentation tasks. For datasets with hundreds of images containing the same classes, this dramatically reduces time-per-annotation.

Pre-Labelling for Human Review

SAM 3 generates candidate masks from simple text prompts, which human annotators then verify and correct. With a zero-shot LVIS AP of 48.8 significantly above prior baselines many initial suggestions are largely correct, meaning annotators spend time confirming rather than drawing.

Fine-Grained and Rare Category Annotation

Because SAM 3 uses language, it can distinguish subcategories such as 'polka-dot dress' or 'vintage car' without requiring custom classifiers or per-class training data. This is particularly useful for long-tail classes that appear infrequently in training datasets.

Multi-Object Video Labelling

SAM 3's tracker preserves object identities across frames. An annotator seeds the first frame with a text prompt and SAM 3 propagates masks through subsequent frames. SAM 3.1's multiplexing makes this practical for scenes with many concurrent objects processing up to 16 in a single forward pass.

Negative Prompt Filtering

SAM 3 supports negative image exemplars. If you want to segment apples but exclude a pear in the same scene, a negative exemplar tells the model what not to include. This makes fine-grained category separation tractable without complex post-processing rules.

Industry Applications

Autonomous driving Label all road signs, pedestrians, and vehicles by name in a single prompt pass.

Retail Mask every instance of a product SKU from shelf images for planogram compliance and inventory models.

Ecology research Meta has made the SA-FARI dataset of 10,000+ wildlife camera trap videos annotated with SAM 3 publicly available.

Medical imaging pre-labelling Accuracy must be carefully verified given SAM 3's noted difficulty with specialist domains. Human expert review of all outputs is required.

7. Integrating SAM 3 with the Scematics Annotation Platform

The Scematics data annotation platform supports AI-assisted labelling workflows with SAM 3 as the segmentation backend. There are four practical integration patterns:

Text Prompt Pre-Labelling

An annotator selects the AI labelling feature, enters comma-separated class names (e.g. 'person, car, bicycle'), and SAM 3 returns segmentation candidates for all those classes across the image set. The annotator then confirms or adjusts the masks using Scematics' polygon editing tools.

Visual Prompt Refinement

Scematics exposes positive and negative bounding box controls. An annotator draws a positive box around an example instance, or a negative box around something to exclude, and SAM 3 updates the segmentation accordingly. This is particularly useful in cluttered scenes.

Video Frame Propagation

For video annotation tasks, an annotator labels the first keyframe using SAM 3 or manual tools, and SAM 3's tracker propagates masks through subsequent frames. SAM 3.1's multiplexing processing up to 16 objects in a single pass makes this significantly faster for scenes with multiple tracked objects.

Batch Processing at Dataset Scale

Scematics supports dataset-level operations. SAM 3 can be run across thousands of images in the background using GPU instances, with resulting masks imported as editable layers for human review.

Recommended Integration Pattern

Treat SAM 3 as a pre-labelling assistant, not a final annotator. Use it to generate initial masks across the dataset, then use Scematics' quality control and human review tools to verify and refine output. This hybrid approach delivers the speed gains of AI assistance while maintaining the quality bar required for reliable model training.

8. Effective Prompting Strategies for SAM 3

Use Clear, Short Noun Phrases

Keep prompts short and specific. 'Red traffic light' and 'domestic cat' work well. The SAM 3 paper restricts concepts to simple noun phrases a noun with optional modifiers. Long descriptive sentences are outside the model's design envelope.

Combine Text and Visual Cues in Cluttered Scenes

In cluttered scenes, start with a text prompt for the category, then add a positive image exemplar or bounding box to focus the model on the correct appearance variant. This is more effective than trying to encode appearance in words alone.

Iterate on Misses with Point Prompts

If initial masks are incomplete, add a point prompt on a missed instance and ask SAM 3 to re-segment. The model remains interactive throughout, and point prompts still work in SAM 3 for individual object correction.

Multi-Class Prompting in a Single Run

SAM 3 supports multiple classes in a single run. Batch classes that commonly appear together to reduce the number of inference passes needed. For example, prompt for 'person, bicycle, car' together on street-scene images rather than running three separate passes.

Handle Edge Cases Explicitly

For partially occluded objects: add a positive point on the visible portion.

For reflective or transparent objects (glass, water): manual tracing may be more reliable than SAM 3.

• For very small objects: check that image resolution is sufficient; SAM 3 can miss objects below a minimum pixel threshold.

Verify Exhaustiveness Before Accepting Masks

SAM 3 is designed to be exhaustive, which means it can produce extra masks for objects that partially match the prompt. Always review the output against the source image before marking annotations as accepted.

9. Recommended 4-Step Annotation Workflow with SAM 3

A practical workflow for teams using SAM 3 in Scematics:

Step	Phase	Action
1	Pre-annotation	Run SAM 3 across the full dataset using class-level text prompts. Generate initial masks for all target categories.
2	Human review	Annotators review pre-labelled masks in Scematics accepting correct masks, correcting or redrawing those that need adjustment.
3	Video continuity	For video tasks: seed key frames with SAM 3, allow the tracker to propagate masks, spot-check at regular intervals for drift or occlusion errors.
4	Quality control	Use Scematics' quality metrics to identify systematic gaps (e.g. consistently missed small objects). Refine prompts or add manual annotations where needed.

10. Frequently Asked Questions (FAQ)

What is SAM 3?

SAM 3 (Segment Anything Model 3) is Meta's open-vocabulary image and video segmentation model, released November 19, 2025. Unlike SAM 1 and SAM 2 which required visual prompts for single objects SAM 3 accepts short noun phrases or image exemplars and segments every matching instance across an entire scene or video. It uses a shared backbone combining a DETR-style detector and a memory-based video tracker.

What is the difference between SAM 3 and SAM 2?

SAM 2 (2024) added video tracking to SAM 1's single-object segmentation, but still required visual prompts (point, box, or mask) for each individual object. SAM 3 introduces open-vocabulary prompting via text phrases and image exemplars, exhaustive multi-instance detection, a presence head that decouples recognition from localisation, and a Perception Encoder backbone replacing SAM 2's CLIP-based encoder.

What is SAM 3.1 and what did it add?

SAM 3.1, released March 2026, introduces object multiplexing processing up to 16 tracked objects in a single forward pass rather than one per object. On an H100 GPU, this doubles throughput from 16 to 32 frames per second for videos with moderate object counts, with no loss in segmentation accuracy. It is a throughput update, not an accuracy update.

What are SAM 3's benchmark results?

On zero-shot LVIS instance segmentation, SAM 3 achieves 48.8 mask AP versus the prior best of 38.5 a 27% relative improvement. On the SA-Co/Gold benchmark (270,000 unique concept phrases), SAM 3 achieves a cgF1 of 54.1, more than double the previous best of 24.6 from OWLv2 and representing 74% of estimated human performance on that benchmark.

Does SAM 3 work for medical image annotation?

SAM 3 has noted difficulty with specialist domains including medical imagery, microscopic images, and camouflaged objects. For high-accuracy medical annotation, the recommendation is to use SAM 3 as a pre-labelling layer only, with fine-tuning or domain-specific adapters to improve accuracy, and mandatory expert human review of all outputs before use as ground truth.

What kind of prompts does SAM 3 accept?

SAM 3 accepts: (1) short noun phrases such as 'red traffic light' or 'domestic cat'; (2) positive or negative image exemplars showing what to include or exclude; (3) point and bounding box prompts for individual object correction. It is not designed for long descriptive sentences or multi-step natural language instructions prompts should be kept to a noun with optional modifiers.

Can SAM 3 annotate LiDAR or 3D point cloud data?

No. SAM 3 is a 2D model that segments images and video frames. It does not natively support 3D point clouds or volumetric data. SAM 3D is a separate, unrelated system for 3D reconstruction from single images. Teams annotating LiDAR data require dedicated 3D annotation tooling.

Use SAM 3 in Your Annotation Workflow Today

Scematics natively integrates SAM 3 (and SAM 3.1) into its annotation platform with text prompt pre-labelling, visual prompt refinement, video frame propagation, and bring-your-own-model support for teams with custom segmentation models. Human review and five-stage quality control are built into every workflow.

Scematics Copyrights Reserved

SAM 3: Meta's Open-Vocabulary Image Annotation Model (2026 Update)