Skip to content
advancedsurvey

How to annotate AgentBoard Progress Scoring

Assess multi-turn LLM agent progress by identifying achieved milestones, scoring overall progress, categorizing the agent environment, and noting partial progress observations.

Try it live — no install

Boot the real Potato server in your browser (WebAssembly) and annotate with this exact config. Nothing leaves your machine.

Run live in your browser

Get This Design

View on GitHub(opens in a new tab)

Clone or download from the repository

Quick start:

bash
git clone https://github.com/davidjurgens/potato-showcase.git
cd potato-showcase/agentic/agentboard-progress-scoring
potato start config.yaml

Configuration File

This Potato config reproduces the annotation task. Save it as config.yaml and run potato start config.yaml to try it.

yaml
# AgentBoard Progress Scoring
# Based on "AgentBoard: An Analytical Evaluation Board of Multi-turn LLM Agents" (Ma et al., NeurIPS 2024)
# Task: Evaluate multi-turn agent progress through milestone tracking and progress scoring

annotation_task_name: "AgentBoard Progress Scoring"
task_dir: "."

data_files:
  - sample-data.json
item_properties:
  id_key: "id"
  text_key: "text"

output_annotation_dir: "annotation_output/"
export_annotation_format: "jsonl"

annotation_schemes:
  - name: milestones_reached
    description: "Select all subtask milestones that the agent successfully achieved."
    annotation_type: multiselect
    labels:
      - "Milestone 1"
      - "Milestone 2"
      - "Milestone 3"
      - "Milestone 4"
      - "Milestone 5"

  - name: progress_score
    description: "Rate the overall progress of the agent toward completing the task."
    annotation_type: likert
    min_label: "1 - No Progress"
    max_label: "5 - Complete"
    size: 5

  - name: agent_category
    description: "Which environment category does this agent task belong to?"
    annotation_type: radio
    labels:
      - name: "Web Shopping"
        key_value: "1"
      - name: "Web Browsing"
        key_value: "2"
      - name: "Tool Use"
        key_value: "3"
      - name: "Game"
        key_value: "4"
      - name: "Embodied"
        key_value: "5"
      - name: "Reasoning"
        key_value: "6"

  - name: progress_notes
    description: "Notes on partial progress, missed milestones, or interesting agent behavior."
    annotation_type: text

user_config:
  allow_all_users: true
max_annotations_per_user: 50
num_annotators_per_item: 2

Sample Data

json
[
  {
    "id": "ab-001",
    "text": "Purchase a red cotton t-shirt in size Medium from an online clothing store, applying any available discount code.",
    "environment": "Web Shopping",
    "milestones": "1. Navigate to clothing store\n2. Search for red cotton t-shirt\n3. Select size Medium\n4. Apply discount code\n5. Complete checkout",
    "trajectory": "Turn 1: Navigate to shop.example.com\nTurn 2: Search 'red cotton t-shirt'\nTurn 3: Filter by color: Red, material: Cotton\nTurn 4: Select first result 'Classic Cotton Tee - Red' ($24.99)\nTurn 5: Choose size: Medium\nTurn 6: Add to cart\nTurn 7: Go to checkout\nTurn 8: Look for discount code field\nTurn 9: Try code 'SAVE10' - Invalid\nTurn 10: Try code 'WELCOME' - 15% off applied\nTurn 11: Confirm order details\nTurn 12: Click 'Place Order'",
    "final_state": "Order confirmed. Red cotton t-shirt (M) purchased for $21.24 after 15% discount. Order #SH-90234."
  },
  {
    "id": "ab-002",
    "text": "Use the Wikipedia API to find the population of the 5 largest cities in Japan and calculate their total population.",
    "environment": "Tool Use",
    "milestones": "1. Identify the 5 largest cities in Japan\n2. Query population for each city\n3. Retrieve correct population figures\n4. Calculate total population\n5. Present results clearly",
    "trajectory": "Turn 1: Call wikipedia_search('largest cities in Japan')\nTurn 2: Parse results - identified Tokyo, Yokohama, Osaka, Nagoya, Sapporo\nTurn 3: Call wikipedia_page('Tokyo') - Population: 13,960,000\nTurn 4: Call wikipedia_page('Yokohama') - Population: 3,749,000\nTurn 5: Call wikipedia_page('Osaka') - Population: 2,753,000\nTurn 6: Call wikipedia_page('Nagoya') - Population: 2,296,000\nTurn 7: Call wikipedia_page('Sapporo') - Population: 1,973,000\nTurn 8: Calculate total: 24,731,000",
    "final_state": "Total population of 5 largest Japanese cities: 24,731,000. All data retrieved successfully from Wikipedia API."
  }
]

// ... and 6 more items

Dataset & paper

Ma et al., NeurIPS 2024

Official dataset(opens in a new tab)Read the paper(opens in a new tab)

Citation (BibTeX)

bibtex
@inproceedings{ma2024agentboard,
  title={AgentBoard: An Analytical Evaluation Board of Multi-turn LLM Agents},
  author={Ma, Chang and Zhang, Junlei and Zhu, Zhihao and Yang, Cheng and Yang, Yujiu and Jin, Yaohui and Lan, Zhenzhong and Kong, Lingpeng and He, Junxian},
  booktitle={Advances in Neural Information Processing Systems},
  year={2024}
}

Details

Annotation Types

multiselectlikertradiotext

Domain

Agentic AILLM AgentsBenchmarking

Use Cases

Agent EvaluationProgress Assessment

Tags

multi-turn-agentsprogress-scoringmilestonesllm-benchmarking

Found an issue or want to improve this design?

Open an Issue(opens in a new tab)

Related Designs

tau-bench Agent Evaluation

Evaluate tool-agent-user interactions in customer service domains by judging task success, conversation quality, tool use correctness, and providing evaluation rationale.

radiolikert

OSWorld: Desktop Agent Task Evaluation

Evaluation of multimodal agents performing open-ended tasks in real desktop environments. Annotators assess task success, identify OS-level actions, rate efficiency, and analyze failures across Ubuntu, Windows, and macOS environments.

radiomultiselect

RefactorBench Multi-File Evaluation

Evaluate multi-file refactoring operations generated by coding agents. Annotators assess whether refactorings preserve behavior, identify the types of refactoring applied, rate code improvement, and provide detailed review comments.

radiomultiselect