Skip to content

Live Coding Agent Observation

Watch coding agents work in real time with pause, rollback, and branching. Three backends supported: Ollama for local models, Anthropic API, and Claude Agent SDK.

New in v2.4.0

Static trace annotation tells you what an agent did. Live observation tells you what an agent does in response to human guidance. Potato's live coding agent mode lets annotators watch a coding agent work in real time -- reading files, editing code, running tests -- and intervene at any point. Pause the agent, send new instructions, rollback to a previous checkpoint, or branch the trajectory to explore alternative approaches.

This produces richer annotation data than static traces alone. You get the full trajectory with timestamps, the annotator's interventions, branching decision points, and comparative data from alternative paths. This data is directly useful for training process reward models, preference models, and instruction-following evaluators.

Requirements

  • Python 3.10+
  • Git (the checkpoint system uses git commits)
  • One of the following agent backends:
    • Ollama for local model inference (no API key required)
    • ANTHROPIC_API_KEY for Anthropic API access
    • Claude Agent SDK for the full Claude Code agent experience

Backends

Potato supports three backends for running coding agents. Each backend runs the agent in a subprocess and streams its actions to the annotation interface in real time.

1. Ollama (Local Models)

Run coding agents locally with no API key required. Ollama provides fast inference for open-weight models. Best for development, testing, and situations where data cannot leave the local machine.

Setup:

bash
# Install Ollama
curl -fsSL https://ollama.com/install.sh | sh
 
# Pull a coding-capable model
ollama pull qwen2.5-coder:7b
 
# Or a larger model for better performance
ollama pull deepseek-coder-v2:16b

Configuration:

yaml
live_agent:
  enabled: true
  backend: ollama
  model: qwen2.5-coder:7b
 
  ollama:
    host: "http://localhost:11434"    # Ollama server URL
    temperature: 0.2
    num_ctx: 8192                     # context window size
    num_predict: 2048                 # max tokens per response
    keep_alive: "5m"                  # keep model loaded in memory
 
  # Agent capabilities
  tools:
    - read_file
    - edit_file
    - write_file
    - bash
    - glob
    - grep
  max_steps: 50
  step_timeout_seconds: 60

2. Anthropic API

Use Claude models via the Anthropic API. Provides strong coding performance with tool use capabilities. Requires an API key.

Setup:

bash
# Set your API key
export ANTHROPIC_API_KEY="sk-ant-..."
 
# Or add to .env file
echo "ANTHROPIC_API_KEY=sk-ant-..." >> .env

Configuration:

yaml
live_agent:
  enabled: true
  backend: anthropic
  model: claude-sonnet-4-20250514
 
  anthropic:
    api_key: ${ANTHROPIC_API_KEY}
    max_tokens: 4096
    temperature: 0.2
    system_prompt: |
      You are a coding assistant working on a software project.
      Read files before editing them. Run tests after making changes.
      Explain your reasoning before each action.
 
  # Agent capabilities
  tools:
    - read_file
    - edit_file
    - write_file
    - bash
    - glob
    - grep
  max_steps: 100
  step_timeout_seconds: 120

3. Claude Agent SDK

The Claude Agent SDK provides the full Claude Code agent experience, including automatic tool orchestration, context management, and multi-file reasoning. This is the most capable backend but requires the SDK to be installed.

Setup:

bash
# Install the Claude Agent SDK
pip install claude-agent-sdk
 
# Set your API key
export ANTHROPIC_API_KEY="sk-ant-..."

Configuration:

yaml
live_agent:
  enabled: true
  backend: claude_agent_sdk
 
  claude_agent_sdk:
    api_key: ${ANTHROPIC_API_KEY}
    model: claude-sonnet-4-20250514
    max_turns: 100
    permission_mode: auto           # auto-approve tool use
    enable_thinking: true           # show extended thinking
 
  max_steps: 100
  step_timeout_seconds: 180

Controls

The annotation interface provides four control actions that let annotators guide the agent's behavior.

Pause / Resume

Click Pause to halt the agent between steps. The agent finishes its current step and waits. The annotator can review the current state, examine files, and decide whether to let the agent continue or intervene. Click Resume to let the agent proceed.

yaml
live_agent:
  controls:
    pause_resume:
      enabled: true
      auto_pause_on_error: true      # pause when a command fails
      auto_pause_after_steps: 0      # pause after N steps (0 = disabled)
      key_value: "Space"

Send Instructions

While the agent is paused, annotators can send new instructions that redirect the agent. This is useful when the agent is going down the wrong path or when the annotator wants to test how the agent responds to guidance.

yaml
live_agent:
  controls:
    send_instructions:
      enabled: true
      placeholder: "Type instructions for the agent..."
      inject_as: system_message      # "system_message" or "user_message"
      key_value: "Enter"
      presets:
        - "Try a different approach"
        - "Read the error message more carefully"
        - "Check the test file for expected behavior"
        - "Revert your last change and try again"

Instructions are injected into the agent's conversation context. The inject_as option controls whether they appear as a system message (authoritative instruction) or a user message (conversational guidance).

Rollback

Rollback reverts the project to a previous git checkpoint. Every file change the agent makes is automatically committed, so the annotator can click any previous step in the timeline and roll back to that exact state. The agent's conversation context is also truncated to match.

yaml
live_agent:
  controls:
    rollback:
      enabled: true
      show_checkpoint_diff: true     # show what will be undone
      require_confirmation: true     # "Are you sure?" dialog
      key_value: "Ctrl+Z"

Branch and Replay

Branch and replay combines rollback with instruction sending. The annotator rolls back to a checkpoint and sends different instructions, creating a branching trajectory. This helps when collecting preference data: you can explore two different approaches from the same starting point and compare outcomes.

yaml
live_agent:
  controls:
    branch:
      enabled: true
      max_branches: 5                # maximum branches from any checkpoint
      branch_naming: auto            # "auto" or "manual"
      compare_view: true             # side-by-side branch comparison
      key_value: "Ctrl+B"

The branch comparison view shows two branches side by side, highlighting where they diverge. Annotators can rate which branch produced better results, generating preference pairs for DPO training.

Git Checkpoint System

The live agent mode uses git to track every file change. This provides reliable rollback, branching, and full change history.

How It Works

  1. Before the agent starts, Potato creates a new git branch named potato-session-{session_id}
  2. After every file change (edit, write, create, delete), Potato automatically commits with a descriptive message
  3. Each commit is tagged as a checkpoint that appears in the timeline
  4. Rollback uses git checkout to restore the working directory to any checkpoint
  5. Branching creates a new git branch from the checkpoint commit

Configuration

yaml
live_agent:
  git_checkpoints:
    enabled: true
    branch_prefix: "potato-session"
    commit_message_format: "Step {step}: {tool} {file_path}"
    auto_commit: true
    cleanup_on_complete: false       # delete session branches when done
    require_clean_working_dir: true  # fail if there are uncommitted changes

Manual Checkpoint Management

bash
# List all Potato session branches
git branch | grep potato-session
 
# View checkpoints for a session
git log potato-session-abc123 --oneline
 
# Clean up old session branches
git branch --list 'potato-session-*' | xargs -r git branch -D

Data Format

Input data for live coding agent tasks specifies the task description and optionally a starting file or directory:

json
{
  "id": "task_001",
  "task_description": "Fix the bug in src/parser.py where empty input causes a crash",
  "project_dir": "/path/to/project",
  "start_file": "src/parser.py",
  "test_command": "python -m pytest tests/test_parser.py -v",
  "context_files": [
    "src/parser.py",
    "tests/test_parser.py"
  ]
}
FieldRequiredDescription
idYesUnique task identifier
task_descriptionYesWhat the agent should do
project_dirYesPath to the project directory
start_fileNoFile to show the agent initially
test_commandNoCommand to verify the fix
context_filesNoFiles to pre-load into the agent's context

Configuration Reference

Complete configuration for a live coding agent observation task:

yaml
annotation_task_name: "Live Coding Agent Observation"
task_dir: "."
 
data_files:
  - "data/coding_tasks.jsonl"
 
item_properties:
  id_key: id
  text_key: task_description
 
instance_display:
  fields:
    - key: structured_turns
      type: coding_trace
      label: "Agent session"
      display_options:
        diff_view: unified
        terminal_theme: dark
        collapse_long_outputs: true
        max_output_lines: 50
        show_file_tree: true
        show_step_numbers: true
        show_reasoning: true
 
live_agent:
  enabled: true
  backend: anthropic
  model: claude-sonnet-4-20250514
 
  anthropic:
    api_key: ${ANTHROPIC_API_KEY}
    max_tokens: 4096
    temperature: 0.2
 
  tools:
    - read_file
    - edit_file
    - write_file
    - bash
    - glob
    - grep
 
  max_steps: 100
  step_timeout_seconds: 120
 
  controls:
    pause_resume:
      enabled: true
      auto_pause_on_error: true
      key_value: "Space"
    send_instructions:
      enabled: true
      inject_as: system_message
      presets:
        - "Try a different approach"
        - "Read the error message carefully"
        - "Run the tests first"
    rollback:
      enabled: true
      require_confirmation: true
    branch:
      enabled: true
      max_branches: 5
      compare_view: true
 
  git_checkpoints:
    enabled: true
    branch_prefix: "potato-session"
    auto_commit: true
    cleanup_on_complete: false
 
annotation_schemes:
  # Per-step ratings during observation
  - annotation_type: trajectory_eval
    name: step_quality
    description: "Rate each agent step as you observe it"
    steps_key: agentic_steps
    correctness_options:
      - "Good"
      - "Acceptable"
      - "Unnecessary"
      - "Incorrect"
 
  # Overall task completion after agent finishes
  - annotation_type: radio
    name: task_completion
    description: "Did the agent complete the task?"
    labels:
      - "Fully Complete"
      - "Partially Complete"
      - "Failed"
 
  # Branch comparison (when branching is used)
  - annotation_type: radio
    name: branch_preference
    description: "Which branch produced a better result?"
    labels:
      - "Branch A"
      - "Branch B"
      - "Both Equal"
      - "Both Failed"
 
  # Notes on the observation
  - annotation_type: text
    name: observation_notes
    description: "Describe what you observed and any interventions you made"
    label_requirement:
      required: false
 
output_annotation_dir: "output/"
export_annotation_format: "jsonl"

Branching Trajectory Export

When annotators roll back and take a different path, the branch they kept and the branch they abandoned make a preference pair: the kept path is chosen, the original is rejected.

json
{
  "prompt": "Fix the failing test in tests/test_parser.py",
  "chosen": "...the corrected trajectory...",
  "rejected": "...the original trajectory...",
  "trace_id": "task_001"
}

That is one line of trajectory_dpo.jsonl. trajectory_sft.jsonl carries the same corrected path as {prompt, completion, trace_id}, and trajectory_corrections.json holds the full records with counts of how many were edited.

Export branching trajectories for preference learning:

bash
python -m potato.export \
  -c config.yaml \
  -f trajectory_correction \
  -o results/

The exporter takes the config and writes into the directory you name: trajectory_corrections.json, trajectory_sft.jsonl with the corrected paths as SFT targets, and trajectory_dpo.jsonl with the branch comparisons as DPO preference pairs. There is no separate trajectory-tree format; the branch structure travels inside those records.

Security

The live agent runs in the project directory specified in the task data. It has access to read, write, and execute files within that directory. Consider the following security practices:

  • Sandboxing: For untrusted code or untrusted agent models, run Potato inside a Docker container or VM. The agent can execute arbitrary shell commands, so isolation is important.
  • Read-only mode: Disable the bash and write_file tools if you only want the agent to analyze code without modifying it.
  • Network restrictions: Use Docker's --network none flag to prevent the agent from making network requests.
  • Resource limits: Set max_steps and step_timeout_seconds to prevent runaway agents.
yaml
# Restricted tool set for analysis-only tasks
live_agent:
  tools:
    - read_file
    - glob
    - grep
  # No edit_file, write_file, or bash

Troubleshooting

Ollama Not Running

text
Error: Connection refused at http://localhost:11434

Start the Ollama server:

bash
ollama serve

Verify it is running:

bash
ollama list

API Key Missing

text
Error: ANTHROPIC_API_KEY environment variable not set

Set the environment variable:

bash
export ANTHROPIC_API_KEY="sk-ant-..."

Or add it to your project's .env file. Potato loads .env files automatically.

Git Not Initialized

text
Error: Project directory is not a git repository

The checkpoint system requires git. Initialize a repository in the project directory:

bash
cd /path/to/project
git init
git add -A
git commit -m "Initial commit"

Agent Stuck in a Loop

If the agent repeats the same action multiple times, it may be stuck. Potato detects loops when the same tool call with the same arguments is repeated 3 times and automatically pauses the agent. You can configure this threshold:

yaml
live_agent:
  loop_detection:
    enabled: true
    threshold: 3                     # pause after N identical consecutive steps
    action: pause                    # "pause" or "terminate"

Session Branch Cleanup

Over time, session branches accumulate. Potato ships no cleanup command for them, so they are ordinary git branches you remove with git:

bash
# See what is there, newest last
git for-each-ref --sort=committerdate \
  --format='%(committerdate:short)  %(refname:short)' 'refs/heads/potato-session-*'
 
# Remove one session's branch
git branch -D potato-session-abc123
 
# Remove all of them
git branch --list 'potato-session-*' | xargs -r git branch -D

Check the listing before the last one. It deletes unmerged branches without asking, which is the point here and a mistake anywhere else.

See Also

For implementation details, see the source documentation.