Новости, руководства и обновления от команды Potato.
Potato 2.9 imports projects from brat, doccano, Prodigy, CoNLL and REFI-QDA, transcribes and diarizes audio locally, and records whether an image shape came from a person, a model or an import. Read the upgrade notes first.
Mean IoU between annotators reads about 0.95 on a typical detection corpus, including for annotators who never looked at the image. What a chance-corrected alternative looks like, and why Krippendorff's alpha over IoU distance does not work.
Potato 2.8 adds computer vision, gigapixel deep zoom, 3D point clouds, depth maps, robot episodes, generative-video evaluation and VLM grounding — and reports chance-corrected agreement over all of them.
Get notified about new tutorials, releases, and community highlights.
Annotation teams already run norming sessions and already wish they knew how their annotators think. Multiplayer Rooms measure the calibration meeting with a live agreement meter; Think-Aloud Mode records the reasoning out loud, locally, with no LLM.
Gold standards are expensive, leaky, and only test the items you thought to plant. Two alternatives, peer-prediction scoring and counterfactual boundary probes, catch inattentive annotators and broken guidelines without a single planted item.
Our paper on Potato 2.0 is in the ACL 2026 System Demonstrations. Here is what it covers and how to cite Potato in your work.
A walkthrough of human evaluation for spoken, video, and document agents in Potato: scoring turn-taking on a dual-track timeline, grounding video events with live IoU, tagging speech errors, and marking table structure.
Annotation pipelines are built to erase disagreement, but on subjective tasks the disagreement is the data. A guide to telling genuine label variation from error, and keeping it in Potato.
Potato 2.6 lets you assign one annotator to most items and three to a stratified sample, raise coverage on items where annotators disagree, and route the contested ones to an adjudicator.
Оценивать выводы моделей с помощью LLM легко. Сложное — понять, верить ли ей. Разбор слепой человеческой калибровки в Potato 2.6: голосование по k выборкам, каппа Коэна и Фляйса и ожидаемая ошибка калибровки.
A look at QDA Mode, the upcoming Potato 2.6 workspace for qualitative data analysis: a living codebook, in-vivo coding, analyst memos, cases, and full-text search over a whole corpus.
Deciding annotator count and overlap for an annotation project: rules of thumb for objective and subjective tasks, the coverage-versus-overlap tradeoff, and how to set it in Potato.
A practical guide to evaluating retrieval-augmented generation: score retrieval relevance and answer faithfulness separately, and mark unsupported claims with span annotation in Potato.
Use Potato's trajectory_eval schema for per-step error localization with hierarchical error taxonomies, severity scoring, and running score tracking across agent traces.
Настройка разметки ревью кода в стиле пул-реквестов GitHub в Potato: комментарии прямо в диффе, оценки качества по файлам и вердикты о принятии или отклонении вывода кодового агента.
Как выстроить человеческую оценку вывода ИИ-агента в Potato: от импорта трасс через устройство схем на уровне трассы и шагов до разбора выгруженных результатов.
Step-by-step tutorial on Potato's Solo Mode: label a seed set yourself, let an LLM label the rest, and use the accuracy checks to decide when it is good enough to stop.
Migrate from Label Studio to Potato, convert project configs, annotation schemas, and exported data with this step-by-step guide covering common annotation types.
Integrate Potato with Prolific for crowdsourced annotation, completion URL setup, participant ID tracking, attention checks, payment configuration, and quality filtering.
Настройка разметки ограничивающими рамками для компьютерного зрения в Potato: цвета меток, минимальный размер рамки, несколько классов, правила проверки и экспорт в COCO/YOLO.
Convert Potato annotations to HuggingFace Datasets format, covering JSON export structure, dataset card generation, Hub upload, and integration with transformers training.
Build a complete sentiment classification task with radio buttons, tooltips, and keyboard shortcuts for efficient labeling.
Configure Potato for toxicity detection, hate speech classification, and sensitive content labeling with annotator wellbeing in mind.
Set up annotation for detecting specific sounds like speech, music, applause, or environmental noises with timestamp spans.
Set up your first Potato annotation project in 5 minutes, from pip install to a running server with a YAML config, sample data, and your first labeled instances.
Create a music annotation task in Potato with waveform visualization, 30-second preview playback, hierarchical genre label trees, and pairwise preference comparisons.
Deploy Potato to production with Docker, nginx, SSL, and cloud platforms including AWS, GCP, and Azure. Includes systemd service configuration and scaling tips.
Configure an audio transcription review task in Potato with waveform visualization, variable-speed playback, and inline text correction interfaces for ASR quality evaluation.
Our paper on Potato was accepted at EMNLP 2022. Learn about the research behind the tool and how to cite it in your work.