Skip to content
Documentation
Docs/Annotation Types

Conversation Trees

Annotate hierarchical conversation structures with node rating and path selection.

Conversation Trees

The tree annotation schema enables annotation of hierarchical conversation structures such as chatbot response trees, dialogue systems, or branching narratives. Annotators can rate individual nodes, select preferred paths, and compare branches at decision points.

Use Cases

  • Chatbot evaluation - Rating quality of multiple response options
  • Dialogue systems - Selecting preferred conversation paths
  • A/B testing - Comparing different response strategies
  • Interactive fiction - Evaluating branching story paths

Quick Start

yaml
annotation_schemes:
  - annotation_type: tree_annotation
    name: response_quality
    description: Evaluate the conversation tree
    node_scheme:
      annotation_type: likert
      min_label: "Poor"
      max_label: "Excellent"
      size: 5
    path_selection:
      enabled: true
      description: Select the best response path through the tree

Configuration Options

FieldTypeDefaultDescription
annotation_typestringRequiredMust be "tree_annotation"
namestringRequiredUnique identifier for this schema
descriptionstringRequiredInstructions displayed to annotators
node_schemeobject{}Annotation scheme config for per-node annotation
path_selection.enabledbooleanfalseEnable path selection through the tree
path_selection.descriptionstring"Select the best response path"Instructions for path selection
branch_comparison.enabledbooleanfalseEnable branch comparison mode

Examples

Node Rating

Rate each response in the conversation tree:

yaml
annotation_schemes:
  - annotation_type: tree_annotation
    name: response_rating
    description: Rate each response in the conversation
    node_scheme:
      annotation_type: likert
      min_label: "Very Bad"
      max_label: "Very Good"
      size: 5

Path Selection

Select the best path through the conversation:

yaml
annotation_schemes:
  - annotation_type: tree_annotation
    name: best_path
    description: Navigate the conversation tree
    path_selection:
      enabled: true
      description: Click on responses to build the best conversation path

Multi-Criteria Node Rating

Rate nodes on multiple dimensions:

yaml
annotation_schemes:
  - annotation_type: tree_annotation
    name: multi_criteria
    description: Evaluate each response on multiple criteria
    node_scheme:
      annotation_type: multirate
      options:
        - Relevance
        - Fluency
        - Helpfulness
      labels: ["1", "2", "3", "4", "5"]

Branch Comparison Mode

Compare sibling branches at decision points:

yaml
annotation_schemes:
  - annotation_type: tree_annotation
    name: branch_compare
    description: Compare response options at each decision point
    branch_comparison:
      enabled: true
    node_scheme:
      annotation_type: radio
      labels: ["Better", "Same", "Worse"]

Data Format

Input

Tree data should be provided in JSON format with a hierarchical structure:

json
{
  "id": "conv_001",
  "tree": {
    "id": "root",
    "role": "user",
    "content": "Hello, I need help with my order",
    "children": [
      {
        "id": "resp_a",
        "role": "assistant",
        "content": "I'd be happy to help! Can you provide your order number?",
        "children": [
          {
            "id": "user_2",
            "role": "user",
            "content": "It's ORDER-12345",
            "children": []
          }
        ]
      },
      {
        "id": "resp_b",
        "role": "assistant",
        "content": "Sure, what seems to be the problem?",
        "children": []
      }
    ]
  }
}

Configure the tree data key:

yaml
item_properties:
  id_key: id
  tree_key: tree

Output

Tree annotations are saved with both node-level and path-level data:

json
{
  "response_quality": {
    "node_annotations": {
      "resp_a": {
        "rating": 4
      },
      "resp_b": {
        "rating": 2
      }
    },
    "selected_path": ["root", "resp_a", "user_2"]
  }
}

User Interface

Tree Visualization

The conversation tree is displayed with nodes representing messages/responses, edges connecting parent-child relationships, and branching points where multiple responses exist.

Node Selection

Click on any node to view the full message content, access the annotation panel, and add the node to the selected path (if path selection is enabled).

Path Selection

When path selection is enabled, click nodes to add them to your path. The selected path is highlighted, and you can use "Clear Path" to start over.

Best Practices

  1. Keep node schemes simple - complex annotation interfaces on each node can slow annotation
  2. Provide context - ensure annotators can see parent messages when rating a response
  3. Use path selection wisely - works best for smaller trees
  4. Train annotators - tree navigation requires practice; provide training examples

Further Reading

For implementation details, see the source documentation.

PreviousTriageNextCoreference Chains