Potato 2.3: Agentic Annotation, Solo Mode, and Best-Worst Scaling

Potato 2.3.0 adds Solo Mode, a human-LLM labeling workflow that runs through ten documented phases, plus agent trace annotation with eleven registered trace converters, Best-Worst Scaling with three scoring methods, SSO and OAuth logins for Google, GitHub and generic OIDC providers, Parquet export, and 15 demo projects. The release notes record 50 or more commits since 2.2.0 and fixes for 12 security vulnerabilities across 8 modules.

The release follows a change in what people annotate. Researchers who were labeling sentiment and named entities two years ago are now picking apart multi-step agent traces, comparing model outputs in bulk, and building datasets for tasks that did not exist when Potato's schema set was designed. Solo Mode and the agent trace work are both aimed at that shift.

Upgrading

pip install --upgrade potato-annotation==2.3.0

Parquet export needs PyArrow 12.0.0 or newer, which is a separate install:

pip install "pyarrow>=12.0.0"

Agent trace annotation

An agent run is a sequence of tool calls, reasoning steps, and intermediate outputs, and flattening it into plain text throws away the structure an evaluator needs to see. Potato 2.3 renders traces through the display system instead, so the trace is laid out as a conversation with ratings attached to individual turns.

The display is declared in instance_display, with per-turn rating schemes nested under the field that holds the conversation:

instance_display:
  layout:
    direction: vertical
    gap: 16px
  fields:
    - key: task_description
      type: text
      label: "Task"
    - key: conversation
      type: dialogue
      label: "Agent Trace"
      span_target: true
      display_options:
        show_turn_numbers: true
        alternating_shading: true

Two display types carry agent work in 2.3. The dialogue display renders a trace as turns and supports span annotation over them, and interactive_chat covers conversations the annotator takes part in rather than reads back. Setting span_target: true is what lets annotators highlight a passage inside a turn, which is how hallucinations and unsupported claims get marked at the place they occur.

Traces from other frameworks come in through the converter, which registers eleven formats at this release: openai, anthropic, swebench, otel, mcp, multi_agent, langchain, langfuse, react, webarena, and atif.

python -m potato.trace_converter -i traces.json -f react -o data/traces.jsonl

Solo Mode

Multi-annotator labeling needs people hired, trained, and coordinated, and on many research projects that logistics is the bottleneck rather than the annotation interface. Solo Mode pairs one domain expert with an LLM instead. The human labels, the LLM proposes, each correction feeds back into the prompt, and the human hands over more of the queue as the two converge.

The workflow runs through ten documented phases: setup, prompt review, edge case synthesis, edge case labeling, prompt validation, parallel annotation, disagreement resolution, periodic review, autonomous labeling, and final validation. Not every project visits every phase.

solo_mode:
  enabled: true
 
  labeling_models:
    - endpoint_type: "ollama"
      model: "qwen3:4b"
      endpoint_url: "http://localhost:11434"
      max_tokens: 256
      temperature: 0.1
 
  revision_models:
    - endpoint_type: "ollama"
      model: "qwen3:4b"
      endpoint_url: "http://localhost:11434"
 
  uncertainty:
    strategy: "direct_confidence"
 
  thresholds:
    end_human_annotation_agreement: 0.90
    minimum_validation_sample: 20
    confidence_low: 0.5
    periodic_review_interval: 50
 
  instance_selection:
    low_confidence_weight: 0.4
    diversity_weight: 0.3
    random_weight: 0.2
    disagreement_weight: 0.1
 
  state_dir: "solo_state"

The four instance_selection weights decide which items are worth a human's attention. Weighting low confidence at 0.4 and diversity at 0.3 sends most of the human's time to instances the model is unsure about while keeping enough coverage of the rest of the distribution to notice a systematic error. The 0.2 on random sampling is what makes the agreement estimate in end_human_annotation_agreement meaningful, since an agreement rate computed only on hard instances is not the rate you get on the corpus.

Solo Mode ships with 85 integration and end-to-end tests, a developer guide, and two example projects under examples/advanced/solo-mode/ and examples/advanced/simple-solo-mode/. Start from the simple one, which runs against a local Ollama model and needs no API key, so the first thing you evaluate is whether the workflow suits your task rather than whether the model is worth its cost.

Best-Worst Scaling

Best-Worst Scaling shows an annotator a tuple of items, usually four, and asks only for the best and the worst against one criterion. Those two choices imply a set of pairwise comparisons, and scores across many overlapping tuples come out on a real scale from judgments that are quicker to make than a rating.

annotation_schemes:
  - annotation_type: bws
    name: sentiment_bws
    description: "Sentiment Intensity"
    best_description: "Which sentence expresses the MOST positive sentiment?"
    worst_description: "Which sentence expresses the LEAST positive sentiment?"
    tuple_size: 4
    sequential_key_binding: true

Three scoring methods run over the collected tuples. counting is the transparent baseline at (best_count - worst_count) / appearances, needs no extra dependency, and is deterministic. bradley_terry fits a pairwise comparison model and produces log-scale strength parameters. plackett_luce fits a full ranking model. Both model-based methods need pip install choix.

python -m potato.bws_scoring --config config.yaml --method bradley_terry

The scores land in bws_scores.tsv in the output directory, with each item's score, rank, best and worst counts, and appearances. The admin dashboard has a BWS Scoring tab that runs the same computation from a button and shows the results in a table.

SSO and OAuth

A deployment that annotators reach over the internet needs real accounts, and 2.3 adds OAuth 2.0 and OpenID Connect so they can use the identity they already have. Google, GitHub, and any generic OIDC provider are supported, and access can be restricted by email domain or GitHub organization.

authentication:
  method: "oauth"
  providers:
    google:
      client_id: ${GOOGLE_CLIENT_ID}
      client_secret: ${GOOGLE_CLIENT_SECRET}
  user_identity_field: "email"
 
secret_key: ${POTATO_SECRET_KEY}

Add this block to your config. Two requirements come with it. OAuth needs an HTTPS deployment, with localhost exempted for development, and it needs a stable secret_key, because a signing key that changes on every restart invalidates the sessions the login has already issued. Register the redirect URI with your provider as https://your-domain.com/auth/callback/google, substituting the provider name.

Parquet export

Annotation data increasingly ends up in analytics tools that want a columnar format, so 2.3 adds Parquet to the export registry. An export produces up to three tables: annotations.parquet with one row per instance and annotator, spans.parquet with one row per span annotation, and items.parquet with the source data.

python -m potato.export -c config.yaml -f parquet -o ./export/

Options are passed on the command line rather than configured in the project, and the compression codec is one of snappy (the default), gzip, zstd, lz4, or none:

python -m potato.export -c config.yaml -f parquet -o ./export/ \
    --option compression=gzip --option include_items=false

The tables join on the instance key, though the column name differs between them: annotations.parquet and spans.parquet carry instance_id, while items.parquet carries item_id. That is what makes the export usable from DuckDB without loading anything into pandas first:

import duckdb
duckdb.sql("""
  SELECT i.item_id, a.user_id, a.*
  FROM './export/annotations.parquet' a
  JOIN './export/items.parquet' i ON a.instance_id = i.item_id
""")

annotations.parquet is one row per instance and annotator, with each annotation scheme flattened into its own column, so there is no generic value column to select. A scheme that stores several answers at once expands into dotted sub-columns named <scheme>.<part>.

Security fixes

The release fixes 12 security vulnerabilities across 8 modules, along with XSS, robustness, and data integrity issues across 25 files, and a span-delete cascade that iterated dictionary keys instead of values and left link records behind. A dependency audit brought the Python and JavaScript dependencies to current versions.

Two of the other fixes could have changed data you already collected. A has_free_response boolean crash (issue #111) and an all-phases navigation loop both affected whether annotations were recorded as intended, and dynamic multirate quote handling affected how those values were written. If you ran a study on 2.2 using free-response options or multirate schemes, check the output before treating it as final.

Demo projects

The release ships 15 demo projects under demo/, numbered in the order they build on each other, from 01-sentiment-admin through 15-agentic-chat, and covering spans, custom moderation layouts, image classification, audio emotion, video, multimodal items, adjudication, sliders, bounding boxes, dependency trees, tiered ELAN annotation, and agentic chat. Each demo directory holds a config.yaml and its data/, and the suite shares one script that generates the media the media-based demos need.

# From the repository root: generates the demo data, then starts all 15 servers
bash demo/launch.sh

For the full changelog, including every fix named above, see the v2.3.0 release notes in the repository. File issues on GitHub or start a thread in GitHub Discussions.

Links

• Installation Guide
• What's New in v2
• Agentic Annotation
• Solo Mode
• Best-Worst Scaling
• SSO and OAuth
• Parquet Export
• GitHub Repository
• PyPI Package