# Inter-Annotator Agreement for 3D Objects, Time Segments and Agent Traces

Source: https://www.potatoannotator.com/blog/agreement-for-3d-time-and-agent-traces

**To measure agreement on structured annotations, compare the structures themselves. Match each annotator's 3D boxes or time segments to the other's by IoU, count unmatched objects as disagreement, and score an agent trace one step at a time so that every step is a unit for kappa or alpha.**

[Inter-annotator agreement](/docs/guides/inter-annotator-agreement) is easy to define when each item gets one label. Many current tasks give an item a set of things instead, such as cuboids around the cars in a lidar sweep, segments marking when each speaker talks, or a verdict for every step an agent took. A measure built for one label per item has to reduce those structures to something it can compare, and the easy reduction, to label names, makes every pair of annotators agree.

## Perfect agreement from label names

That reduction keeps the label names and drops everything else. Two annotators who each drew three boxes labeled "car" on a lidar sweep both reduce to "car", and they agree perfectly even if their boxes sit in different places.

Potato's per-scheme agreement route, `/admin/api/agreement`, did this for structured schemes before version 2.11, and on agent traces, segments, regions and 3D objects every pair of annotators came out in perfect agreement. A perfect score on a hard spatial or temporal task usually means the measure lost the structure. In 2.11 that route no longer reports structured schemes and points to the full agreement report at `/admin/iaa`, which compares the structures.

## Matching 3D objects and time segments

Agreement on a set of objects starts with matching. Each object from one annotator is paired with at most one object from the other, using an overlap measure, and a pair counts as the same object only above a threshold. Time segments use temporal IoU ([intersection over union](https://en.wikipedia.org/wiki/Jaccard_index)), the length of the overlap divided by the length of the union, and 3D objects use 3D IoU, the same ratio over volumes. Potato matches segments at 0.5 and 3D objects at 0.25, because a small offset costs more IoU in three dimensions than in two. A box shifted by 10% of its size along each axis keeps a 2D IoU of about 0.68 and a 3D IoU of only about 0.57.

After matching, each matched pair contributes its IoU, a matched pair with different labels contributes half its IoU, and every unmatched object on either side contributes zero. The item's agreement is the total divided by the number of objects, matched pairs counted once. An annotator who misses half the objects therefore scores low even when the objects they did mark are perfect, and two annotators who both mark nothing agree.

`/admin/iaa` reports two numbers per scheme for 3D objects (`spatial_annotation`) and for time segments under `temporal_grounding` and `tiered_annotation`:

- **`mean_agreement`**, the mean of that item-level score over every pair of annotators on every item with two or more annotators; and
- **`mean_object_count_diff`**, the mean difference in how many objects the two annotators marked.

The count difference separates the two ways a set can disagree. A low agreement with a count difference near zero means people found the same objects and placed them differently. A large count difference means one annotator finds objects the other misses, which points at what the guidelines count as an object.

Neither number is chance-corrected. A chance baseline for spatial agreement needs a model of where objects land when annotators disagree at random, and Potato builds one only for 2D boxes and polygons, as [agreement over geometry](/docs/measurement/geometry-agreement) describes. For time segments the same approach fails, because a segment from a different clip is not a meaningful random comparison when clips differ in length, which [agreement over time](/docs/measurement/temporal-agreement) explains. Compare these numbers between pilots of the same task, where the difficulty of placing an object stays fixed.

One edge case changed in 2.11. Two identical zero-length segments, an instant marked by both annotators at the same time, have an IoU of 0 divided by 0. They used to score as complete disagreement, and now they count as agreeing.

## Step-level agreement for agent traces

An agent trace is a sequence of steps, and the trace schemes (`process_reward`, `trajectory_eval`, `tool_call_review`, `failure_attribution`, `gui_trajectory`) ask for a label on each step. Scoring a trace by its set of labels repeats the label-name problem, and scoring it by an overall verdict throws away the steps.

`/admin/iaa` treats each step of each trace as its own unit. Two annotators who rate step 4 of trace 12 contribute one pair, and nominal [Krippendorff's alpha](https://en.wikipedia.org/wiki/Krippendorff%27s_alpha) and [Cohen's kappa](https://en.wikipedia.org/wiki/Cohen%27s_kappa) are computed over all such units, along with percent agreement. Unlike the set measures above, both coefficients are chance-corrected, so a step-level alpha of 0.6 on a process-reward task reads on the same scale as alpha on any categorical task. The [labeling engine](/docs/labeling-engine/contracts) parses step labels the same way when it scores an LLM judge against a person, and its score is the share of shared steps the two labeled differently.

## Assigning two annotators per item

`/admin/iaa` needs no configuration. It scores every item that two or more annotators have answered, so the work is in the assignment. To give every item two annotators, set two annotators per item and hand out the least-annotated items first:

```yaml
assignment_strategy: least_annotated
num_annotators_per_item: 2
```

Two annotators per item is the minimum the report needs. On a small overlap set, a single unusual trace or lidar sweep can move the mean enough to change what you conclude, so plan the overlap before annotation starts rather than after. The older `agreement_metrics` block configures the dashboard's per-scheme figures, which from 2.11 skip these schemes and point to `/admin/iaa`.

## Agreement and count difference across pilot rounds

Read `mean_agreement` together with the count difference and the number of pairs `/admin/iaa` reports beside it. A drop between pilot rounds with a stable count difference points at placement, so the guidelines on box tightness or segment boundaries need work. A rise in the count difference points at what counts as an object or an event. For agent traces, compare step-level alpha with the alpha the same annotators reach on a simpler categorical question in the same study, which helps separate hard step labels from unreliable annotators.

## Further reading

- [Inter-annotator agreement](/docs/guides/inter-annotator-agreement), for the coefficients and when to use each
- [Agreement over geometry](/docs/measurement/geometry-agreement), for chance-corrected agreement on 2D boxes and polygons
- [Agreement over time](/docs/measurement/temporal-agreement), for segments and break-points in audio and video
- [Raw IoU Is Not Agreement](/blog/raw-iou-is-not-agreement), on why mean IoU alone overstates spatial agreement