Agreement over Time
What Potato reports when annotators mark time segments and break-points, why segment agreement is not chance-corrected, and why break-point agreement is a tolerance sweep.
Two annotators marking when an event starts will not choose the same frame, so temporal agreement has to say how close counts as agreeing. Potato answers that differently for time segments and for break-points: segments are matched by temporal IoU above a threshold, and break-points are reported across a sweep of tolerances.
Measures for time segments
For audio and video segments, the agreement report compares every pair of annotators on every item and averages four numbers:
| Question | Measure |
|---|---|
| How well do the two sets of segments agree overall? | Mean agreement, which penalizes loose boundaries, missed segments and mismatched labels |
| How close are the boundaries when both annotators found the segment? | Mean matched temporal IoU |
| Did they find the same segments at all? | Detection F1 at the matching threshold, 0.5 by default |
| Did they mark the same number of segments? | Mean difference in segment count |
Segments are matched on temporal IoU alone. Labels count only in mean agreement, where a matched pair with different labels earns half its IoU, so no number on its own says how often annotators agree on the label. A pair in which neither annotator marked anything is counted but not scored, since two empty answers are not evidence of agreement.
The tolerance sweep for break-points
Break-points are single instants, such as the moment a robot episode goes wrong, and two of them count as the same mark when they fall within a tolerance window. Every number then depends on that window, so the robot episode report computes them at 0.04, 0.25, 0.5, 1 and 2 seconds and shows all five.
Reporting a single threshold invites the reader to assume whichever tolerance supports their conclusion. Agreement at 0.25 s and agreement at 2 s are different claims about the data:
- High at 2 s and low at 0.25 s means annotators agree an event happened but not precisely when. That is usually a guideline problem: the definition of "when it starts" is underspecified.
- Low at both means they disagree about whether it happened at all.
At each tolerance, break-point agreement is decomposed into detection, localization and category, as described under world-model evaluation. Time segments have no such sweep; their boundaries are judged only through temporal IoU.
Coverage is reported alongside
For dense curves, such as a progress-reward trace over a robot episode, agreement is reported as ICC plus Pearson with coverage stated.
A correlation computed over 5% of a timeline says nothing about the other 95%, and a coefficient that does not carry its coverage will be read as though it did.
A known limitation
Time segments get the uncorrected measures only: no σ, no detection α and no label α. σ's chance baseline is built from distances between objects on different items, and "a segment from a different clip" is not a meaningful comparison when clips differ in length. Getting that right needs its own design, and claiming chance correction here before it exists would be worse than saying so.
To measure label agreement on segments, or boundary agreement at several tolerances, paste the segments into the span and box agreement calculator, which reports boundary F1 across a tolerance sweep and α over matched segments' labels.