Skip to content
Diese Seite ist in Ihrer Sprache noch nicht verfügbar. Englische Version wird angezeigt.

Interactive Segmentation

Click an object and get a mask, in the browser, with no GPU and no network call per click. How Potato runs MobileSAM through ONNX Runtime Web, and how to configure it.

Click an object and get a mask back in about 130 ms, in the annotator's browser, with no GPU and no network round trip. Two commands install it, and once the weights are on the machine it works air-gapped.

Setup

bash
potato download-models onnxruntime     # 13.5 MB, once per install
potato download-models mobile_sam      # 45 MB

Then add sam to a schema's tools:

yaml
annotation_schemes:
  - annotation_type: image_annotation
    name: objects
    description: "Outline every object you find."
    source_field: image
    tools: [sam, polygon, brush, eraser]
    labels:
      - name: object
        color: "#6e56cf"

The brush, eraser and colour-aware fill tools stay available for manual work. Interactive segmentation is a way to start a mask, not a replacement for correcting one.

Run examples/image/interactive-segmentation/ to see it working.

How it works

The model is MobileSAM, a distilled variant of Segment Anything, running through ONNX Runtime Web.

Encoder and decoder are separated on purpose. The encoder is expensive and runs once per image; the decoder is cheap and runs once per click. Keeping them apart, and caching the embedding per image, is the whole reason clicking feels interactive rather than like a network request. The cache is bounded and evicts the least recently used embedding.

You can click positively to add to a mask, click negatively to cut away from it, box an object to constrain the search, or refine a mask you already have.

Why the browser is the default

pip install potato-annotation should give you working segmentation with no GPU, no new Python dependency and no outbound network at annotation time. That last point is not a nicety: several research groups run Potato air-gapped, where a model fetched at click time is a missing feature rather than a slow one.

Browser (default)Server endpoint
Setuppotato download-modelspip install 'potato-annotation[vision]' plus weights
GPUNot requiredRecommended
Air-gappedYes, once models are downloadedYes
Model sizeDistilled, about 45 MBWhatever you supply

A server endpoint exists for labs that have a GPU and want a larger model, but it is the exception rather than the default.

The input contract was verified, not assumed

SAM's encoder takes a preprocessed image, and the specification admits more than one plausible reading. Three of those readings produce confident, plausible, wrong masks: 70 to 148 pixels of centroid error, which looks like a slightly sloppy annotator rather than a broken pipeline. The correct reading lands at 0.1 px.

The difference is only visible if you check against real weights and real ground truth, which is why the contract is pinned by a test rather than by a comment. A wrong reading here would have silently degraded every mask in every dataset built with it.

Agreement over masks

Two annotators segmenting the same object will not produce identical pixels, so mask agreement needs more than an overlap number. Potato uses STAPLE (Warfield, Zou and Wells, 2004), which estimates a latent consensus boundary together with a sensitivity and specificity for each rater.

The difference from a per-pixel majority vote is not cosmetic. With two careful annotators outnumbered three-to-two by noisy ones, majority vote scores Dice 0.846 against truth and STAPLE scores 1.000, because STAPLE notices that the careful pair agree with each other while the noisy three do not.

See mask consensus for the details and the caveats.