VLM Grounding, Pointing and Region Captioning
Bind referring expressions to regions, score pointing by point-in-region hit rate rather than IoU, and count ungroundedness separately instead of as a miss.
Bind each referring expression to the region that answers it, with ungroundedness as an explicit answer rather than a missing one. Grounding is scored by IoU at four thresholds; pointing is scored as a point-in-region hit rate, because a point has no area and every IoU against one is zero.
annotation_schemes:
# grounding_eval owns the phrase-to-region binding but does not draw, so an
# image schema sits beside it to own the canvas.
- annotation_type: image_annotation
name: region
description: "Draw the region for the selected phrase."
source_field: image
tools: [bbox, polygon, landmark]
labels: [{name: referent, color: "#6e56cf"}]
- annotation_type: grounding_eval
name: grounding
description: "What does each phrase refer to?"
region_type: box # box | polygon | mask | point
expressions_field: expressionsItems carry their phrases:
{"id": "img1", "image": "/media/img1.jpg",
"expressions": [{"id": "e1", "text": "the man in the red shirt"}]}A bare list of strings works too, with ids falling back to position — which matters, because reordering the list then re-points existing annotations. A source with stable ids should use them.
Runnable example: examples/ai-assisted/grounding-eval/.
Pointing is not grounding with a small box
Models in the Molmo family emit points rather than boxes. Set region_type: point and the annotator places a landmark.
Points have to be scored differently: a point has no area, so every IoU against it is 0. Scoring points the way boxes are scored reports total failure for a model that is pointing perfectly. The question a point answers is "is it in the thing", which is a hit rate.
mean_miss_distance is computed over the misses only. Averaged over hits as well it would mostly measure how large the objects are, not how badly the model missed.
Agreement between annotators on a pointing task is likewise a distance, in normalized image units, not a coefficient. An overlap measure applied to points saturates: two annotators pointing at opposite corners of the same image still score about 0.86, which any coefficient scale would band as strong agreement. A distance has no ceiling to saturate against, and is named a distance so nothing bands it as a coefficient.
Ungroundedness is counted separately
Three outcomes are tracked apart from each other:
- Correctly declined. The expression refers to nothing present, and the annotator said so.
- Hallucinated a location. A region was given for something that is not there.
- Missed a present referent. Something was there and was not found.
An expression the annotator never answered is excluded, not counted as a miss. Counting it makes a model look worse the more phrases were skipped, which is a statement about the annotator rather than about the model.
Region captions
region_caption adds a free-text description per region, with agreement computed through a semantic distance rather than string overlap, since two annotators describing the same object rarely share a content word.
- annotation_type: region_caption
name: captions
description: "Describe each region you drew."
placeholder: "e.g. a red mug with a chipped handle"
min_length: 10
max_length: 300
agreement_distance: tokenRunnable example: examples/image/region-captioning/, which is configured with two annotators and agreement.
Reviewing a model's predictions
Set predictions_field and the item's predictions travel to the client, adding a verdict control. Predictions are labelled as predictions all the way through and are never merged into the annotator's own regions. A prediction the annotator has not accepted is evidence about the model, and folding it into ground truth destroys the only thing it is for.