# Shipping Segmentation That Runs in a Browser Tab

Source: https://www.potatoannotator.com/blog/segmentation-in-the-browser

`pip install potato-annotation` should give you working segmentation with no GPU, no new Python dependency, and no outbound network at annotation time.

That last constraint is not a nicety. Several research groups deploy Potato air-gapped, where a model fetched at click time is a missing feature rather than a slow one. So both of Potato's image-assist models run in the annotator's browser through ONNX Runtime Web: MobileSAM for click-to-segment, Grounding DINO for text prompting.

Three things that went into it are worth writing down, because they generalise past this project.

## The encoder contract admits several plausible readings

SAM's image encoder takes a preprocessed tensor, and the specification leaves room for interpretation: normalization, resize convention, padding, channel order.

We implemented one reading and got masks. They looked right. Centroid error against ground truth was 70 to 148 pixels — the sort of error that reads as a slightly sloppy annotator rather than as a broken pipeline. Three separate plausible readings all produced confident, plausible, wrong masks.

The correct reading lands at **0.1 px**.

The lesson is not "read the spec more carefully". It is that a wrong reading here is **invisible without ground truth**: the masks are the right shape, in roughly the right place, and every downstream check passes. Only a numeric comparison against known truth separates the four candidates, and only one of them is right.

So the contract is pinned by a test against real weights, not documented in a comment. A comment would have been just as true and would not have caught the regression.

## The quantization was a real choice

Quantizing a 686 MB model to run in a browser is not optional. Which quantization, though, is usually decided by whatever the export tool defaults to.

Measured against the full-precision Grounding DINO export:

| Export | Box IoU vs full precision | Size |
|---|---|---|
| `q4f16` | **0.972** | 151 MB |
| `int8` | 0.874 | 201 MB |

`q4f16` holds substantially more of the original geometry **and** is 50 MB smaller. int8 is the conventional choice and would have shipped a worse model in a larger file.

This took an afternoon to measure and would have been permanent otherwise, because nobody re-examines a quantization choice once boxes are appearing on screen.

Verified live afterwards on the COCO two-cat photo: both cats detected at 0.724 and 0.688, with boxes matching the Python reference implementation to four decimal places.

## Writing the tokenizer by hand

Grounding DINO needs its caption tokenized the way BERT does it. The obvious move is to pull in `transformers.js`.

That is about 2 MB of JavaScript for one function, on a page that already loads a 151 MB model, so we implemented WordPiece directly — roughly 200 lines — and checked it token-for-token against HuggingFace's tokenizers.

The saved bytes are not really the point. The point is that a tokenizer which disagrees with the model's training tokenizer produces **subtly wrong detections that look like a threshold problem**. You would spend a day tuning `box_threshold` before suspecting the tokenizer. Having a token-level equivalence test makes that failure mode impossible rather than merely unlikely.

## Where the browser stops being the right answer

Video mask propagation runs server-side, and that is a deliberate exception rather than an inconsistency.

The economics are different. Click-to-segment costs one cheap decoder pass **per click**, against an encoding computed once per image and cached. Propagation costs a full model pass **per frame**, the model is 181 MB across five graphs, and the video is already on the server. A hundred frames in a browser tab is minutes of a frozen page.

We keep a lighter in-browser carry-forward path that re-prompts frame to frame, and we are careful not to describe the two as the same capability. "SAM 2 propagation runs in your browser" would be a better sentence and a false one.

## What we would tell someone building this

1. **Get ground truth before you get output.** Plausible wrong output is the failure mode, and it survives review.
2. **Measure the quantization.** The default is a choice someone else made for a different constraint.
3. **Test the preprocessing against the reference implementation numerically.** Visual inspection cannot distinguish 0.1 px from 148 px of error at a glance.
4. **Separate the encoder from the decoder** and cache the expensive half. That separation is the entire reason clicking feels interactive.

## Further reading

- [Interactive segmentation](/docs/vision-spatial/segmentation)
- [Open-vocabulary text prompting](/docs/vision-spatial/text-prompting)
- [Model zoo and licences](/docs/vision-spatial/model-zoo)