# How to Evaluate VLM Grounding and Pointing

Source: https://www.potatoannotator.com/docs/guides/vlm-grounding-evaluation

**Grounding evaluation asks whether a model put a region in the right place; pointing evaluation asks whether it put a point on the right thing.** They need different measures, and using the grounding measure for pointing reports total failure for a model that is pointing perfectly.

## Grounding: report several thresholds

Score grounding by [IoU](https://en.wikipedia.org/wiki/Jaccard_index) against the human region, at **0.25, 0.5, 0.75 and 0.9**.

One threshold cannot distinguish "nearly right" from "nowhere near", and a model tuned to clear 0.5 exactly looks identical to one that is genuinely tight. The shape of the curve across thresholds is the finding.

## Pointing is not grounding with a small box

Models in the [Molmo](https://arxiv.org/abs/2409.17146) family emit points. **A point has no area, so every IoU against it is 0.** Score a pointing model the way you score a grounding model and it reports complete failure regardless of how well it points.

The question a point answers is "is it *in* the thing", so the measure is a hit rate: `point_in_region`.

Report `mean_miss_distance` over the **misses only**. Averaged over hits as well, it mostly measures how large the objects are rather than how badly the model missed.

### Annotator agreement on points is a distance

The same structural problem appears between annotators. An overlap-derived measure applied to points saturates: two annotators pointing at **opposite corners of the same image** still score about 0.86, which any coefficient scale bands as strong agreement.

So report the **mean and median distance** between annotators' points in normalized image units, where 0 is perfect. A distance has no ceiling to saturate against, and calling it a distance stops anyone banding it as a coefficient.

## Ungroundedness is three different errors

Track these apart:

| Outcome | Meaning |
|---|---|
| Correctly declined | The expression refers to nothing present, and the model said so |
| Hallucinated a location | A region was given for something that is not there |
| Missed a present referent | Something was there and was not found |

Folding them into one "wrong" number hides the difference between a model that over-claims and one that under-claims, which have opposite fixes.

**An expression the annotator never answered is excluded, not counted as a miss.** Counting it makes a model look worse the more phrases were skipped, which is a statement about the annotator rather than the model.

## Region captions need a semantic distance

Two annotators describing the same object rarely share a content word. "A red mug with a chipped handle" and "the damaged cup on the left" describe the same region and have no overlapping content terms.

Compute caption agreement through a **semantic distance** rather than string overlap. Krippendorff's α accepts any distance function, so the same coefficient machinery applies once the distance is right.

## Config

```yaml
annotation_schemes:
  # grounding_eval owns the binding but does not draw, so an image schema
  # supplies the canvas.
  - annotation_type: image_annotation
    name: region
    description: "Draw the region for the selected phrase."
    source_field: image
    tools: [bbox, polygon, landmark]
    labels: [{name: referent, color: "#6e56cf"}]

  - annotation_type: grounding_eval
    name: grounding
    description: "What does each phrase refer to?"
    region_type: box          # use `point` for pointing evaluation
    expressions_field: expressions
```

Items carry their phrases with stable ids:

```json
{"id": "img1", "image": "/media/img1.jpg",
 "expressions": [{"id": "e1", "text": "the man in the red shirt"}]}
```

A bare list of strings works, but ids then fall back to position, so reordering the list re-points existing annotations.

Working examples: the [RefCOCO](/showcase/refcoco-expression) and [PixMo-Points](/showcase/pixmo-points-pointing) showcase designs.

## Reviewing model predictions

If you are auditing a model rather than collecting ground truth, keep predictions labelled as predictions the whole way through. A prediction the annotator has not accepted is evidence about the model, and merging it into the human regions destroys the only thing it is for.

## Further reading

- [VLM grounding reference](/docs/vision-spatial/vlm-grounding)
- [Measuring agreement on bounding boxes](/docs/guides/measuring-agreement-on-bounding-boxes)
- [Open-vocabulary object detection](/docs/guides/open-vocabulary-object-detection)