# VLM Grounding, Pointing and Region Captioning

Source: https://www.potatoannotator.com/docs/vision-spatial/vlm-grounding

**Bind each referring expression to the region that answers it, with ungroundedness as an explicit answer rather than a missing one.** Grounding is scored by IoU at four thresholds; pointing is scored as a point-in-region hit rate, because a point has no area and every IoU against one is zero.

```yaml
annotation_schemes:
  # grounding_eval owns the phrase-to-region binding but does not draw, so an
  # image schema sits beside it to own the canvas.
  - annotation_type: image_annotation
    name: region
    description: "Draw the region for the selected phrase."
    source_field: image
    tools: [bbox, polygon, landmark]
    labels: [{name: referent, color: "#6e56cf"}]

  - annotation_type: grounding_eval
    name: grounding
    description: "What does each phrase refer to?"
    region_type: box            # box | polygon | mask | point
    expressions_field: expressions
```

Items carry their phrases:

```json
{"id": "img1", "image": "/media/img1.jpg",
 "expressions": [{"id": "e1", "text": "the man in the red shirt"}]}
```

A bare list of strings works too, with ids falling back to position — which matters, because reordering the list then re-points existing annotations. A source with stable ids should use them.

Runnable example: [`examples/ai-assisted/grounding-eval/`](https://github.com/davidjurgens/potato/tree/main/examples/ai-assisted/grounding-eval).

## Pointing is not grounding with a small box

Models in the [Molmo](https://arxiv.org/abs/2409.17146) family emit points rather than boxes. Set `region_type: point` and the annotator places a landmark.

Points have to be scored differently: **a point has no area, so every IoU against it is 0**. Scoring points the way boxes are scored reports total failure for a model that is pointing perfectly. The question a point answers is "is it *in* the thing", which is a hit rate.

`mean_miss_distance` is computed over the **misses only**. Averaged over hits as well it would mostly measure how large the objects are, not how badly the model missed.

Agreement between annotators on a pointing task is likewise a **distance**, in normalized image units, not a coefficient. An overlap measure applied to points saturates: two annotators pointing at opposite corners of the same image still score about 0.86, which any coefficient scale would band as strong agreement. A distance has no ceiling to saturate against, and is named a distance so nothing bands it as a coefficient.

## Ungroundedness is counted separately

Three outcomes are tracked apart from each other:

- **Correctly declined.** The expression refers to nothing present, and the annotator said so.
- **Hallucinated a location.** A region was given for something that is not there.
- **Missed a present referent.** Something was there and was not found.

An expression the annotator **never answered** is **excluded**, not counted as a miss. Counting it makes a model look worse the more phrases were skipped, which is a statement about the annotator rather than about the model.

## Region captions

`region_caption` adds a free-text description per region, with agreement computed through a **semantic distance** rather than string overlap, since two annotators describing the same object rarely share a content word.

```yaml
  - annotation_type: region_caption
    name: captions
    description: "Describe each region you drew."
    placeholder: "e.g. a red mug with a chipped handle"
    min_length: 10
    max_length: 300
    agreement_distance: token
```

Runnable example: [`examples/image/region-captioning/`](https://github.com/davidjurgens/potato/tree/main/examples/image/region-captioning), which is configured with two annotators and agreement.

## Reviewing a model's predictions

Set `predictions_field` and the item's predictions travel to the client, adding a verdict control. Predictions are labelled as predictions all the way through and are **never merged into the annotator's own regions**. A prediction the annotator has not accepted is evidence about the model, and folding it into ground truth destroys the only thing it is for.

## Related

- [RefCOCO showcase design](/showcase/refcoco-expression)
- [PixMo-Points showcase design](/showcase/pixmo-points-pointing)
- [Guide: VLM grounding evaluation](/docs/guides/vlm-grounding-evaluation)
- [Source documentation](https://github.com/davidjurgens/potato/blob/main/docs/agent-evaluation/vlm_grounding.md)