How to Evaluate VLM Grounding and Pointing

Grounding evaluation asks whether a model put a region in the right place; pointing evaluation asks whether it put a point on the right thing. They need different measures, and using the grounding measure for pointing reports total failure for a model that is pointing perfectly.

Grounding: report several thresholds

Score grounding by IoU against the human region, at 0.25, 0.5, 0.75 and 0.9.

One threshold cannot distinguish "nearly right" from "nowhere near", and a model tuned to clear 0.5 exactly looks identical to one that is genuinely tight. The shape of the curve across thresholds is the finding.

Pointing is not grounding with a small box

Models in the Molmo family emit points. A point has no area, so every IoU against it is 0. Score a pointing model the way you score a grounding model and it reports complete failure regardless of how well it points.

The question a point answers is "is it in the thing", so the measure is a hit rate: point_in_region.

Report mean_miss_distance over the misses only. Averaged over hits as well, it mostly measures how large the objects are rather than how badly the model missed.

Annotator agreement on points is a distance

The same structural problem appears between annotators. An overlap-derived measure applied to points saturates: two annotators pointing at opposite corners of the same image still score about 0.86, which any coefficient scale bands as strong agreement.

So report the mean and median distance between annotators' points in normalized image units, where 0 is perfect. A distance has no ceiling to saturate against, and calling it a distance stops anyone banding it as a coefficient.

Ungroundedness is three different errors

Track these apart:

Folding them into one "wrong" number hides the difference between a model that over-claims and one that under-claims, which have opposite fixes.

An expression the annotator never answered is excluded, not counted as a miss. Counting it makes a model look worse the more phrases were skipped, which is a statement about the annotator rather than the model.

Region captions need a semantic distance

Two annotators describing the same object rarely share a content word. "A red mug with a chipped handle" and "the damaged cup on the left" describe the same region and have no overlapping content terms.

Compute caption agreement through a semantic distance rather than string overlap. Krippendorff's α accepts any distance function, so the same coefficient machinery applies once the distance is right.

Config

annotation_schemes:
  # grounding_eval owns the binding but does not draw, so an image schema
  # supplies the canvas.
  - annotation_type: image_annotation
    name: region
    description: "Draw the region for the selected phrase."
    source_field: image
    tools: [bbox, polygon, landmark]
    labels: [{name: referent, color: "#6e56cf"}]
 
  - annotation_type: grounding_eval
    name: grounding
    description: "What does each phrase refer to?"
    region_type: box          # use `point` for pointing evaluation
    expressions_field: expressions

Items carry their phrases with stable ids:

{"id": "img1", "image": "/media/img1.jpg",
 "expressions": [{"id": "e1", "text": "the man in the red shirt"}]}

A bare list of strings works, but ids then fall back to position, so reordering the list re-points existing annotations.

Working examples: the RefCOCO and PixMo-Points showcase designs.

Reviewing model predictions

If you are auditing a model rather than collecting ground truth, keep predictions labelled as predictions the whole way through. A prediction the annotator has not accepted is evidence about the model, and merging it into the human regions destroys the only thing it is for.

Further reading

• VLM grounding reference
• Measuring agreement on bounding boxes
• Open-vocabulary object detection