How to Label Objects by Typing Their Name

Open-vocabulary detection means the annotator types a phrase and every match in the image comes back boxed, without the class having been in any training set. It is the fastest way to bootstrap a detection dataset for classes no off-the-shelf detector covers.

How it works

Grounding DINO takes an image and a caption, and scores 900 candidate boxes against every word of that caption. Give it "cat . bus . stop sign" and it returns boxes with per-phrase confidences.

That is a different mechanism from a fixed-class detector, and it changes how you should write your prompts:

• Phrases beat labels. "traffic cone" is a phrase the model has seen in captions; cone_orange_road is not.
• Context helps. "person riding a bicycle" can localise better than "person" on a busy street.
• Terse is worse. The model was trained on natural language, so write natural language.

The licence matters

Text-prompt labelling is widely assumed to require SAM 3 and its non-commercial terms. It does not: Grounding DINO is Apache-2.0.

This is worth checking before you build a dataset rather than after. If your labelling model's licence is non-commercial and your dataset is not, you have a problem that is expensive to discover late. Potato's model zoo states each model's licence and refuses to download a non-commercial one without --accept-licence.

Configuration

annotation_schemes:
  - annotation_type: image_annotation
    name: objects
    description: "Find and label the objects."
    source_field: image
    tools: [bbox, polygon, sam]
    labels: [cat, bus, stop sign]
 
    text_prompt:
      phrases: [cat, bus, stop sign]
      box_threshold: 0.3
      text_threshold: 0.25
      segment: false

The two thresholds do different jobs and should be tuned separately:

• box_threshold decides whether a box is offered at all. Raise it if annotators are drowning in false positives.
• text_threshold decides which phrase labels an offered box. Raise it if boxes are appearing under the wrong label.

Setting segment: true hands each accepted box to the SAM decoder, turning it into a mask.

Quantization is a real choice

If you are deploying one of these yourself, do not accept the default export. Measured against the 686 MB full-precision Grounding DINO export:

q4f16 keeps substantially more geometry and is smaller. Taking int8 because it is the conventional choice ships a worse model in a larger file.

Where it breaks down

• Fine-grained classes. "Golden retriever" versus "labrador" is beyond it; it will box the dog.
• Domain-specific vocabulary. Medical, industrial and scientific terms are usually not in the training captions.
• Counting. It finds instances; it does not reliably find all instances in a crowded scene.
• Negations and relations. "The cup that is not on the table" is not a query it answers.

For those cases the suggestions are still a useful starting point, but expect to draw.

Suggestions are not annotations

Keep them apart all the way through. A box the annotator accepted is ground truth; a box they did not is evidence about the model, and folding the two together destroys the only thing the second is for.

If you want to know whether the accepting is real review, that is a timing question, not an accuracy question.

Further reading

• Text prompting reference
• Interactive segmentation
• Model zoo and licences