Skip to content
Эта страница пока недоступна на вашем языке. Показана английская версия.

Automation Rules

Route incoming agent traces automatically with filter, sample, and action rules. Potato runs each rule over every incoming trace to route it to the annotation queue, curate it into a dataset, run an evaluator, fire a webhook, or notify annotators.

Automation rules close the production-to-evaluation loop: a programmable filter → sampling rate → actions pipeline runs over every item entering Potato, whether loaded from data files or ingested at runtime via the trace webhook or tracing SDK. Each matching rule can route the item to the annotation queue, curate it into an eval dataset, run an evaluator, fire an outbound webhook, or notify annotators.

Enabling

yaml
automation:
  enabled: true
  rules:
    - name: route-errors
      when: {field: status, in: [error, failed]}
      sample_rate: 1.0          # 0.0–1.0 (default 1.0 = every match)
      actions:
        - {type: add_to_queue, priority: 100, reason: "Agent errored"}
        - {type: add_to_dataset, dataset: errors-to-fix}
        - {type: run_evaluator, evaluator: trajectory_match}
        - {type: fire_webhook, url: "https://example.com/hook"}
        - {type: notify, message: "New error trace"}

How a rule fires

A rule fires when both hold:

  1. when matches — the shared condition grammar (same as triage): equals, in, contains, exists, lt/lte/gt/gte, dotted field paths (metadata.score). A list of conditions is AND-ed; an empty when matches everything.
  2. sample_rate selects it — deterministic sampling on a hash of (item id, rule name), so re-processing the same item yields the same decision (idempotent, replay-safe).

Actions

ActionWhen it runsEffect
add_to_queueinline (fast)Boost the item's triage priority so it surfaces first
add_to_datasetinline (fast)Append the item as an example to a dataset
notifyinline (fast)Notify connected annotators via SSE
run_evaluatorbackground workerScore the item with an evaluator; store the score on the item
fire_webhookbackground workerPOST {rule, item_id, item_data} to an external URL

Fast actions run inline in the ingestion path. Heavy actions (run_evaluator, fire_webhook) go to a background worker, so ingestion never waits on an external call. Every action records an outcome, and failures are caught rather than raised, so a bad rule cannot break ingestion.

Inspecting

The admin dashboard links to Automation (/admin/automation), showing configured rules, activity counters, and recent action outcomes (also available as JSON at /admin/automation/status and /admin/automation/outcomes).