Agreement over Time
Temporal IoU for segment boundaries, boundary α, and why break-point agreement is reported as a tolerance sweep rather than at a single threshold.
Two annotators marking when an event starts will not choose the same frame, so temporal agreement has to say how close counts as agreeing — and that choice changes the answer. Potato reports the sweep rather than picking one threshold for you.
What is measured
| Question | Measure |
|---|---|
| Do the segments overlap? | Temporal IoU |
| Do the boundaries land in the same place? | Boundary α |
| Did they mark the same kind of event? | Nominal α over labels |
| Did they agree there was an event at all? | Detection α |
The tolerance sweep
For break-point and boundary annotations, agreement depends on a matching tolerance: how far apart can two marks be and still count as the same mark?
Reporting a single threshold invites the reader to assume whichever tolerance supports their conclusion. Agreement at 0.25 s and agreement at 2 s are different claims about the data:
- High at 2 s and low at 0.25 s means annotators agree an event happened but not precisely when. That is usually a guideline problem: the definition of "when it starts" is underspecified.
- Low at both means they disagree about whether it happened at all.
So the report sweeps the tolerance and shows the curve. See world-model evaluation, where break-point agreement is additionally decomposed into detection, localization and category.
Coverage is reported alongside
For dense curves — a progress-reward trace over a robot episode, for instance — agreement is reported as ICC plus Pearson with coverage stated.
A correlation computed over 5% of a timeline says nothing about the other 95%, and a coefficient that does not carry its coverage will be read as though it did.
A known limitation
Temporal segments currently get the uncorrected measures only. σ's chance baseline is built from between-item distances, and "a segment from a different clip" is not a meaningful comparison when clips differ in length. Getting that right needs its own design, and claiming chance correction here before it exists would be worse than saying so.