TRAIL Agent Trace Issue Localization

# TRAIL Agent Trace Issue Localization
#
# Dataset:  TRAIL (Trace Reasoning and Agentic Issue Localization), Patronus AI
# Paper:    Deshpande, D., Gangal, V., Mehta, H., Krishnan, J., Kannappan, A., and
#           Qian, R. (2025). TRAIL: Trace Reasoning and Agentic Issue Localization.
#           arXiv:2505.08638 (no published version recorded on arXiv).
#
#   @misc{deshpande2025trail,
#     title     = {TRAIL: Trace Reasoning and Agentic Issue Localization},
#     author    = {Deshpande, Darshan and Gangal, Varun and Mehta, Hersh and
#                  Krishnan, Jitin and Kannappan, Anand and Qian, Rebecca},
#     year      = {2025},
#     publisher = {arXiv},
#     doi       = {10.48550/ARXIV.2505.08638},
#     url       = {https://arxiv.org/abs/2505.08638}
#   }
#
# DOI:      10.48550/arXiv.2505.08638
# Dataset:  https://huggingface.co/datasets/PatronusAI/TRAIL (gated; access must be requested)
# Code:     https://github.com/patronus-ai/trail-benchmark
# Licence:  The dataset is gated. Every trace in sample-data.json is ILLUSTRATIVE:
#           it was written for this showcase in the span and step shape that TRAIL
#           describes, and no TRAIL trace, span, or error annotation is copied.
#
# Task: Walk an agent trace span by span. Mark each span correct or incorrect and,
# for an incorrect span, choose the error category from TRAIL's taxonomy, an impact
# level, and a written description.
#
# TRAIL error taxonomy, quoted from the taxonomy tree in the paper's prompt appendix
# (Appendix A.11). Parenthetical glosses are the paper's own.
#   Reasoning Errors
#     Hallucinations: Language-only; Tool-related (fabricating tool outputs / capabilities)
#     Information Processing: Poor Information Retrieval (Tried to find information that
#       was not relevant to the task); Tool Output Misinterpretation (Made assumptions
#       about the tool output or used the tool output in an incorrect context)
#     Decision Making: Incorrect Problem Identification (Misunderstood the overall task
#       or the local task); Tool Selection Errors (Used the wrong tool for the task)
#     Output Generation: Formatting Errors (Errors with formatting and execution of code
#       or structuring of output in a specific format); Instruction Non-compliance
#       (Failed to perform the task provided and instead did something else)
#   System Execution Errors
#     Configuration: Tool Definition Issues (The tool was not defined correctly by
#       the user or contains some errors that make it inconsistent with its
#       description. For example, web search tool was defined as a calculator tool);
#       Environment Setup Errors (includes
#       permission problems and inability to access resources or API keys)
#     API Issues: Rate Limiting (Like 429); Authentication Errors (Like 401/403);
#       Service Errors (Like 500); Resource Not Found (Like 404)
#     Resource Management: Resource Exhaustion (includes memory overflow); Timeout Issues
#       (The system took too long to respond)
#   Planning and Coordination Errors
#     Context Management: Context Handling Failures (includes window overflow and state
#       tracking or forgetting important context); Resource Abuse (Called the tool
#       excessively due to memory issues)
#     Task Management: Goal Deviation (The system deviated from the task or the subtask);
#       Task Orchestration (includes subtask coordination between agents and progress
#       monitoring)
#   Domain Specific Errors (Errors that are specific to the domain of the task)
# The body text of the paper spells three leaf names slightly differently ("Tool
# Selection Error", "Task Orchestration Errors", "Incorrect Tool Definition"). This
# config follows the taxonomy tree.
#
# Impact: the paper records an impact of Low, Medium, or High for every error
# (Appendix A.7) and does not define the three levels in the text read for this
# design. Annotators apply their own reading of the levels, and a project would
# write its own definitions into annotation_instructions.
#
# Adaptations:
#  - All ten traces are illustrative, short (3 to 7 spans), and invented. The one
#    web address in them (example.com/p/4471) is fictitious, on the reserved
#    example.com domain, and is not meant to resolve to a paper. TRAIL's
#    traces come from GAIA and SWE Bench Lite runs and average far more spans.
#  - TRAIL records each error as {category, location (span id), evidence, description,
#    impact}, and one span can carry several errors. trajectory_eval stores one entry per
#    step: correctness, one error type, one severity, and one rationale. A span with
#    two errors takes the more consequential one in the type field and names the other
#    in the rationale. The location is the step number. An evidence quote is not a
#    separate field, so annotators paste it into the rationale.
#  - TRAIL's three severities are Low, Medium, and High impact. They are mapped to
#    the scheme's severities. The weights (Low -1, Medium -3, High -5) are required by the scheme and are
#    not from TRAIL, so show_score is false and no trace score is computed. The
#    renderer still prints each weight beside its severity name, so annotators
#    see "LOW (-1)", "MEDIUM (-3)" and "HIGH (-5)".
#  - TRAIL also rates each trace for reliability, security, instruction adherence, and
#    plan optimality on 0-5 and 1-5 scales (Appendix A.7.1). Those four trace-level
#    ratings are not included here.
#  - Each step is shown as a card from the steps list. A step with correctness
#    "correct" carries no error fields.
#
# No model_training block: Potato's trainers consume specific schemes, and none of
# them reads trajectory_eval. The prm-reward trainer reads process_reward, which is
# a different scheme, so there is nothing to train from these labels.
#
# Run:
#   python -m potato.flask_server start config.yaml -p 8000

annotation_task_name: "TRAIL Agent Trace Issue Localization"
task_dir: "."

data_files:
  - sample-data.json

item_properties:
  id_key: id
  text_key: task

output_annotation_dir: "annotation_output/"
export_annotation_format: "jsonl"

instance_display:
  layout:
    direction: vertical
    gap: 12px
  fields:
    - key: task
      type: text
      label: "Task given to the agent"
    - key: trace_source
      type: text
      label: "Trace"

annotation_schemes:
  - annotation_type: trajectory_eval
    name: trace_spans
    description: "Read each span in order. Mark it correct or incorrect. For an incorrect span, choose the error category, the impact, and describe the error with a short quote from the span as evidence."
    steps_key: steps
    step_text_key: content
    correctness_options:
      - correct
      - incorrect
    error_types:
      - name: "Reasoning Errors / Hallucinations"
        subtypes:
          - "Language-only"
          - "Tool-related"
      - name: "Reasoning Errors / Information Processing"
        subtypes:
          - "Poor Information Retrieval"
          - "Tool Output Misinterpretation"
      - name: "Reasoning Errors / Decision Making"
        subtypes:
          - "Incorrect Problem Identification"
          - "Tool Selection Errors"
      - name: "Reasoning Errors / Output Generation"
        subtypes:
          - "Formatting Errors"
          - "Instruction Non-compliance"
      - name: "System Execution Errors / Configuration"
        subtypes:
          - "Tool Definition Issues"
          - "Environment Setup Errors"
      - name: "System Execution Errors / API Issues"
        subtypes:
          - "Rate Limiting"
          - "Authentication Errors"
          - "Service Errors"
          - "Resource Not Found"
      - name: "System Execution Errors / Resource Management"
        subtypes:
          - "Resource Exhaustion"
          - "Timeout Issues"
      - name: "Planning and Coordination Errors / Context Management"
        subtypes:
          - "Context Handling Failures"
          - "Resource Abuse"
      - name: "Planning and Coordination Errors / Task Management"
        subtypes:
          - "Goal Deviation"
          - "Task Orchestration"
      - name: "Domain Specific Errors"
    severities:
      - name: LOW
        weight: -1
      - name: MEDIUM
        weight: -3
      - name: HIGH
        weight: -5
    show_score: false

user_config:
  allow_all_users: true

max_annotations_per_user: 10
num_annotators_per_item: 2

annotation_instructions: |
  Each item is a short agent trace made of spans. A span is one model call or one tool call.
  Judge every span in order. Mark a span incorrect only when that span contains the error,
  not when it merely follows an earlier error. For an incorrect span, pick the most specific
  category in the taxonomy, set the impact to LOW, MEDIUM, or HIGH, and write what went wrong.
  Quote the evidence from the span in the description. If one span holds two errors, choose
  the one that matters more and name the other in the description.
  All traces in this design are illustrative and were written for the showcase.