Skip to content
Эта страница пока недоступна на вашем языке. Показана английская версия.

Web Agent Annotation

Review web-browsing agent traces in Potato with filmstrip navigation, SVG overlays (clicks, bounding boxes, mouse paths), and per-step annotation controls.

New in v2.4.0

Evaluating web-browsing AI agents requires seeing exactly what the agent saw, where it clicked, and whether each step made sense. Potato provides a dedicated Web Agent Trace Viewer with two modes: reviewing pre-recorded traces and creating new ones by browsing live websites.

Overview

ModeUse When
Review ModeYou have pre-recorded traces from WebArena, Mind2Web, Anthropic Computer Use, or Potato's own recorder
Creation ModeYou want annotators to browse websites and record new interaction traces

Review Mode

Annotators step through screenshots of an agent's browsing session. SVG overlays render click markers, bounding boxes, mouse paths, and scroll indicators on top of each screenshot.

Configuration

yaml
instance_display:
  fields:
    - key: steps
      type: web_agent_trace
      label: "Agent Browsing Trace"
      display_options:
        show_overlays: true
        show_filmstrip: true
        show_thought: true
        show_observation: true
        show_element_info: true
        screenshot_max_width: 800
        screenshot_max_height: 600
        filmstrip_size: 80

Data Format

Each instance needs a steps array with per-step data:

json
{
  "id": "trace_001",
  "task_description": "Find and add a blue wool sweater to cart",
  "site": "amazon.com",
  "steps": [
    {
      "step_index": 0,
      "screenshot_url": "screenshots/step_000.png",
      "action_type": "click",
      "element": {
        "tag": "input",
        "text": "Search",
        "bbox": [340, 45, 680, 75]
      },
      "coordinates": {"x": 510, "y": 60},
      "mouse_path": [[200, 300], [350, 200], [510, 60]],
      "thought": "I need to search for blue wool sweaters",
      "observation": "Search box is focused",
      "timestamp": 1.2,
      "viewport": {"width": 1280, "height": 720}
    }
  ]
}

Supported Action Types

ActionOverlay
clickRed circle with crosshair and pulse animation
typeYellow highlight on target element
scrollGreen directional arrow
hoverPurple circle
selectBlue bounding box
navigateNo overlay
waitNo overlay
doneNo overlay

Keyboard Shortcuts

KeyAction
← / →Previous / Next step
1Toggle click marker overlays
2Toggle bounding box overlays
3Toggle mouse path overlays
4Toggle scroll indicators
AShow all overlays
NHide all overlays

Per-Step Annotations

Per-step judgments come from a trajectory_eval scheme rather than a flag on an ordinary scheme. It walks the steps in order and records correctness, an error category, and a severity for each one:

yaml
annotation_schemes:
  - annotation_type: trajectory_eval
    name: step_correctness
    description: "Mark each step correct, incorrect, or unnecessary"
    steps_key: steps
    step_text_key: action_type
    correctness_options:
      - correct
      - incorrect
      - unnecessary
    show_score: true

The result is stored as a steps list, one entry per step index, alongside the computed score. See per-step error localization for the error taxonomy and severity weights.

Creation Mode

Annotators browse a live website inside the Potato interface and interactions are automatically recorded as an annotation-ready trace.

Configuration

yaml
instance_display:
  fields:
    - key: browsing_session
      type: web_agent_trace
      display_options:
        show_overlays: true
        show_filmstrip: true
        show_observation: true
        show_thought: true

Proxy Modes

ModeDescription
auto (default)Detects whether the target site allows iframe embedding and chooses the best mode automatically
iframeForces iframe proxy — works for ~90% of sites with under 100ms overhead
playwrightForces server-side Playwright — works for 100% of sites, requires the playwright package

To enable Playwright mode:

bash
pip install playwright
playwright install chromium

Converting Existing Traces

Use the trace converter CLI to normalize traces from other frameworks into Potato's format:

bash
# Convert from a specific format
python -m potato.trace_converter -i traces.json -f web_agent -o output.jsonl
 
# Auto-detect format
python -m potato.trace_converter -i traces.json --auto-detect -o output.jsonl

Supported input formats: WebArena, VisualWebArena, Mind2Web, Anthropic Computer Use, and raw Potato recordings.

Full Example

yaml
annotation_task_name: "Web Agent Evaluation"
task_dir: "."
 
data_files:
  - "traces.jsonl"
 
instance_display:
  fields:
    - key: task_description
      type: text
      label: "Task"
    - key: steps
      type: web_agent_trace
      label: "Agent Trace"
      display_options:
        show_overlays: true
        show_filmstrip: true
        show_thought: true
 
annotation_schemes:
  - annotation_type: radio
    name: task_success
    description: "Did the agent complete the task successfully?"
    labels:
      - name: "Yes"
      - name: "Partially"
      - name: "No"
  - annotation_type: radio
    name: step_correctness
    description: "Was this step correct?"
    labels:
      - name: correct
      - name: incorrect
      - name: unnecessary
  - annotation_type: text
    name: error_description
    description: "Describe any errors in the agent's behavior"
 
output_annotation_dir: "output/"
export_annotation_format: jsonl

Running Example Projects

bash
# Review Mode
python potato/flask_server.py start examples/agent-traces/web-agent-review/config.yaml -p 8000
 
# Creation Mode
python potato/flask_server.py start examples/agent-traces/web-agent-creation/config.yaml -p 8000

API Reference

These REST endpoints are available when using Creation Mode:

EndpointMethodDescription
/api/web_agent/start_sessionPOSTBegin a recording session
/api/web_agent/save_stepPOSTSave a recorded interaction step
/api/web_agent/save_screenshotPOSTUpload a screenshot for a step
/api/web_agent/end_sessionPOSTEnd session and persist the trace
/api/web_agent/proxy/{url}GETProxy an external URL through the server
/api/web_agent/check_frameableGETTest whether a URL allows iframe embedding

Further Reading

For implementation details, see the source documentation.