Shipping Segmentation That Runs in a Browser Tab

pip install potato-annotation should give you working segmentation with no GPU, no new Python dependency, and no outbound network at annotation time.

That last constraint is not a nicety. Several research groups deploy Potato air-gapped, where a model fetched at click time is a missing feature rather than a slow one. So both of Potato's image-assist models run in the annotator's browser through ONNX Runtime Web: MobileSAM for click-to-segment, Grounding DINO for text prompting.

Three things that went into it are worth writing down, because they generalise past this project.

The encoder contract admits several plausible readings

SAM's image encoder takes a preprocessed tensor, and the specification leaves room for interpretation: normalization, resize convention, padding, channel order.

We implemented one reading and got masks. They looked right. Centroid error against ground truth was 70 to 148 pixels — the sort of error that reads as a slightly sloppy annotator rather than as a broken pipeline. Three separate plausible readings all produced confident, plausible, wrong masks.

The correct reading lands at 0.1 px.

The lesson is not "read the spec more carefully". It is that a wrong reading here is invisible without ground truth: the masks are the right shape, in roughly the right place, and every downstream check passes. Only a numeric comparison against known truth separates the four candidates, and only one of them is right.

So the contract is pinned by a test against real weights, not documented in a comment. A comment would have been just as true and would not have caught the regression.

The quantization was a real choice

Quantizing a 686 MB model to run in a browser is not optional. Which quantization, though, is usually decided by whatever the export tool defaults to.

Measured against the full-precision Grounding DINO export:

q4f16 holds substantially more of the original geometry and is 50 MB smaller. int8 is the conventional choice and would have shipped a worse model in a larger file.

This took an afternoon to measure and would have been permanent otherwise, because nobody re-examines a quantization choice once boxes are appearing on screen.

Verified live afterwards on the COCO two-cat photo: both cats detected at 0.724 and 0.688, with boxes matching the Python reference implementation to four decimal places.

Writing the tokenizer by hand

Grounding DINO needs its caption tokenized the way BERT does it. The obvious move is to pull in transformers.js.

That is about 2 MB of JavaScript for one function, on a page that already loads a 151 MB model, so we implemented WordPiece directly — roughly 200 lines — and checked it token-for-token against HuggingFace's tokenizers.

The saved bytes are not really the point. The point is that a tokenizer which disagrees with the model's training tokenizer produces subtly wrong detections that look like a threshold problem. You would spend a day tuning box_threshold before suspecting the tokenizer. Having a token-level equivalence test makes that failure mode impossible rather than merely unlikely.

Where the browser stops being the right answer

Video mask propagation runs server-side, and that is a deliberate exception rather than an inconsistency.

The economics are different. Click-to-segment costs one cheap decoder pass per click, against an encoding computed once per image and cached. Propagation costs a full model pass per frame, the model is 181 MB across five graphs, and the video is already on the server. A hundred frames in a browser tab is minutes of a frozen page.

We keep a lighter in-browser carry-forward path that re-prompts frame to frame, and we are careful not to describe the two as the same capability. "SAM 2 propagation runs in your browser" would be a better sentence and a false one.

What we would tell someone building this

1. Get ground truth before you get output. Plausible wrong output is the failure mode, and it survives review.
2. Measure the quantization. The default is a choice someone else made for a different constraint.
3. Test the preprocessing against the reference implementation numerically. Visual inspection cannot distinguish 0.1 px from 148 px of error at a glance.
4. Separate the encoder from the decoder and cache the expensive half. That separation is the entire reason clicking feels interactive.

Further reading

• Interactive segmentation
• Open-vocabulary text prompting
• Model zoo and licences