Open-Vocabulary Text Prompting

Type traffic cone and every traffic cone in the image comes back boxed, as suggestions to accept or reject. The model runs in the annotator's browser, so there is no GPU to provision and no network round trip once it is installed.

Setup

potato download-models grounding_dino_tiny   # once per install
potato download-models onnxruntime           # shared with segmentation

annotation_schemes:
  - annotation_type: image_annotation
    name: objects
    description: "Find and label the objects."
    source_field: image
    tools: [bbox, polygon, sam]
    labels: [cat, bus, stop sign]
 
    text_prompt:
      phrases: [cat, bus, stop sign]   # what the box starts with
      box_threshold: 0.3               # how sure before a box is offered
      text_threshold: 0.25             # how a box gets its label
      segment: false                   # true turns accepted boxes into masks

Run examples/image/text-prompt-labeling/ to see it working.

The licence is the interesting part

Text-prompt labelling is widely assumed to require SAM 3 and its non-commercial terms. It does not. The model here is Grounding DINO, an open-vocabulary detector released under Apache-2.0, which is why this could ship to everyone rather than to a paid tier.

Grounding DINO takes an image and a caption and scores 900 candidate boxes against every word of that caption. Potato assembles the caption from your phrases, runs the model through ONNX Runtime Web, and turns the output into suggestions. Accepting one stores a normalized bounding box through the same coordinate contract every other tool uses, so a suggestion and a hand-drawn box are the same kind of object downstream.

Setting segment: true hands each accepted box to the SAM decoder, turning it into a mask.

Quantization chosen by measurement

The shipped model is quantized. Which quantization was not a default that happened to be selected — it was measured against the full-precision export:

q4f16 holds substantially more of the original geometry and is 50 MB smaller than int8. Taking the default would have shipped a worse model in a larger file.

Verified live on the COCO two-cat photo: both cats detected at 0.724 and 0.688 confidence, with bounding boxes matching the Python reference implementation to four decimal places.

BERT tokenization is implemented directly

Grounding DINO needs its caption tokenized the way BERT does it. Potato implements WordPiece directly rather than pulling in about 2 MB of transformers.js for one function, and the implementation is checked token-for-token against HuggingFace's tokenizers.

The point is not the saved bytes. A tokenizer that disagrees with the model's training tokenizer produces subtly wrong detections that look like a threshold problem, so it is worth testing against the reference rather than trusting it.

Where this fits

Suggestions are suggestions. An accepted box is an annotation; an unaccepted one is not, and the two are kept apart all the way through. If you care whether your annotators are reviewing suggestions or rubber-stamping them, the timing is what tells you. See drawing telemetry.

Related

• Interactive segmentation — click instead of type
• Model zoo — licences and what downloads
• Guide: open-vocabulary object detection
• Source documentation