# Potato 2.8: Annotate Anything, Then Measure It Source: https://www.potatoannotator.com/blog/potato-2-8-release Potato was a text annotation platform with an image mode. As of 2.8 it covers images, video, gigapixel scans, 3D point clouds, depth maps, robot episodes, generative-video rollouts and vision-language grounding, and for every one of them it reports whether the annotators agreed. That second half is the part worth reading twice. Drawing tools are not scarce: CVAT, Label Studio, Roboflow, V7 and Supervisely all draw boxes and polygons, several of them better than Potato does. What is scarce is a chance-corrected reliability statistic over spatial labels. We have not found another annotation platform that reports one. The standard is raw IoU or percent match. ![Potato 2.8: annotate anything, then measure whether the annotations agree](/images/blog/potato-2-8-banner.svg "Potato 2.8") ## Upgrade if you installed from PyPI Before anything else. Wheels up to 2.7.1 declared their package data with a single-level glob, so **27 templates stored in subdirectories never made it into the published package**. Solo mode's setup routes, the admin dashboard, judge calibration and the corpus map were all broken for anyone who ran `pip install`. Anyone running from a git checkout never saw it, which is also why the test suite never caught it. ```bash pip install --upgrade potato-annotation ``` Reported and fixed in [PR #164](https://github.com/davidjurgens/potato/pull/164) by [@0x6362](https://github.com/0x6362). ## Why raw IoU is not agreement Take a corpus where every image holds one large, centred object. Two annotators who both draw a box in the middle will overlap heavily, so mean IoU comes out around 0.95. So will an annotator who drew a box in the middle without looking at the image. The number is measuring how easy the task is, not how much the annotators agree, and that is exactly the problem chance correction was invented to solve for categorical labels. Potato reports σ, which is α's own `1 − D_o/D_e` form generalized to an arbitrary distance, with the chance baseline estimated empirically by comparing annotations of *different* items. σ = 0 means annotators agree no more than they would on unrelated images. The obvious alternative — Krippendorff's α over 1 − IoU — turns out not to work, and it is worth saying why, because it was our original plan. IoU distance saturates: two randomly paired boxes almost always have IoU 0, so expected disagreement collapses to about 1 and α degenerates into `1 − mean distance` with no chance correction left in it. Braylan, Alonso and Lease ([WWW 2022](https://dl.acm.org/doi/10.1145/3485447.3512242)) found the same thing empirically, reporting that for bounding boxes α ranks plain L2 above IoU and GIoU, inverting the ordering practitioners give. Spatial agreement is also reported as three numbers rather than one: did annotators find the same objects, call them the same thing, and put them in the same place. An annotator who finds everything and mislabels it has a different problem from one who labels correctly and boxes loosely, and the two have different fixes. ## The bug that made this necessary Adjudication compared the *keys* of an annotation. An image schema stores everything under one key named `_data`. Every image pair therefore scored 1.0 agreement. Two annotators who agreed on nothing looked unanimous, and no image was ever routed for review. That is fixed, and it is the reason the agreement layer is built into each annotation type rather than added on top. ## Segmentation and text prompting, in the browser Click an object, get a mask, in about 130 ms with no GPU and no network call per click. MobileSAM under ONNX Runtime Web. Type a phrase and every match comes back boxed. That one is Grounding DINO, and the licence is the interesting part: text-prompt labelling is widely assumed to need SAM 3's non-commercial terms, and it does not. Grounding DINO is **Apache-2.0**, which is why this could ship to everyone rather than to a paid tier. Two details from building it that generalise beyond Potato: **The encoder contract was checked against real weights.** SAM's input specification admits several plausible readings, and three of them produce confident, plausible, *wrong* masks: 70 to 148 pixels of centroid error, which looks like a slightly sloppy annotator rather than a broken pipeline. The correct reading lands at 0.1 px. Only a check against real weights and real ground truth separates them. **The quantization was chosen by measurement.** Against the 686 MB full-precision Grounding DINO export, `q4f16` holds box IoU 0.972 where `int8` holds 0.874 — and `q4f16` is 50 MB *smaller*. Taking the conventional choice would have shipped a worse model in a larger file. ## Video, and the value of declining to answer Draw a mask on one frame, press Track forward, and SAM 2 follows the object through the frames that follow. Measured against known ground truth: per-frame IoU of 0.974 to 0.979, with no decay from the first frame to the last, at roughly 1.32 s per frame on CPU. This one runs **server-side**, deliberately. The cost is per frame rather than per prompt, and a hundred frames of a video model in a browser tab would be minutes of a frozen page. Potato does have a lighter in-browser carry-forward path, and the two should not be conflated. The property that matters most in an annotation loop is that the model **decides for itself when the object is occluded and returns an empty frame** rather than guessing. A plausible wrong mask on a hidden frame is work to undo; an empty frame is immediately readable, and is also the correct answer. ## 3D, depth and robots `spatial_annotation` reads PCD, PLY, LAS, KITTI `.bin` and `.xyz`, with octree level of detail, orthographic slab panels, and calibration that projects every 3D box into every camera image so annotators can verify in 2D while editing in 3D. Rotation is stored as a quaternion rather than a yaw angle. That is what makes KITTI import lossless, including the roughly 0.85° camera-to-lidar mounting tilt a yaw-only field discards silently — and export back to KITTI reports *how much* orientation it had to drop rather than flattening the box quietly. Depth maps read 16-bit PNG and TIFF, NPY, PFM and EXR, with a metres readout under the cursor. Zero is painted magenta rather than rendered as "very close", because zero is the near-universal no-return code and a stereo rig facing a textureless wall really does return 80% holes. `episode_annotation` puts N synchronized video streams and M robot time-series lanes on one timeline. Three outcomes, not two: **partial is the modal result in real robot data**, and forcing it into a binary destroys the signal that makes the dataset worth annotating. Lane downsampling preserves minima and maxima, so a one-frame force spike survives the trip to a 300-pixel lane — which matters, because a missed grasp is visible in the force trace several frames before it is obvious on camera. ## Marking where the world breaks `rollout_evaluation` shows 2 to N generated videos on one clock and asks the annotator to mark the frame at which the world stops making sense, then tag which physical or causal property broke. A rating of 3 out of 5 for "physical plausibility" cannot be checked, cannot be localised, and cannot be used to fix anything. A frame index plus a category can be all three, and two annotators' answers are two points on a line, so a real agreement statistic applies. Break-point agreement is reported as detection, localization and category, with the matching tolerance shown as a **sweep** rather than one number, because agreement at 0.25 s and agreement at 2 s are different claims about the same data. ## The feature we expect to be under-appreciated Pre-labelling makes annotators faster. It also makes rubber-stamping frictionless: a suggestion appears, the annotator clicks accept, and the dataset becomes a record of a model agreeing with itself. Nothing in the annotation distinguishes that from careful review. The geometry is identical, and every quality measure looks *better*, inter-annotator agreement included, because annotators who all accept the same pre-label are unanimous by construction. The only place the difference shows up is the timing. Drawing telemetry records time per shape, stroke dynamics, revision counts and AI-suggestion accept latency — and records no coordinates at all, ever, as a structural property of the event record rather than a policy. As one-click auto-labelling becomes the norm, this is the only signal that separates review from rubber-stamping. ## Also in 2.8 - **Threaded conversations.** The `dialogue` display renders reply structure from each turn's `reply_to`, and `potato convokit` imports any [ConvoKit](https://convokit.cornell.edu/) corpus and exports back, with no `convokit` dependency. - **15 import and 29 export formats**, 11 round-tripping. Darwin works both directions, which matters if you are leaving a platform rather than joining one. - **Live database ingestion.** Rows created after startup become annotatable within `poll_interval_seconds` with no restart. - **Machine-readable specs.** A JSON Schema covering 159 config keys, 61 annotation types and 24 display types, plus an OpenAPI document with 419 paths. Both generated from the code, both checked in CI. - **No outbound requests.** Every asset serves from the install across all 14 templates. `styles.css` had been opening with an `@import` of Google Fonts, which sent every annotator's IP address to a third party on every page load — from a tool people self-host precisely so their data does not leave their infrastructure. It had survived three previous air-gap audits, because every guard read `