# Automation Rules

Source: https://www.potatoannotator.com/docs/agent-evaluation/automation-rules

**Automation rules close the production-to-evaluation loop: a programmable `filter → sampling rate → actions` pipeline runs over every item entering Potato** — whether loaded from data files or ingested at runtime via the [trace webhook](/docs/features/agentic-annotation) or [tracing SDK](/docs/agent-evaluation/tracing-sdk). Each matching rule can route the item to the annotation queue, curate it into an [eval dataset](/docs/agent-evaluation/datasets-and-experiments), run an [evaluator](/docs/agent-evaluation/programmatic-evaluators), fire an outbound webhook, or notify annotators.

## Enabling

```yaml
automation:
  enabled: true
  rules:
    - name: route-errors
      when: {field: status, in: [error, failed]}
      sample_rate: 1.0          # 0.0–1.0 (default 1.0 = every match)
      actions:
        - {type: add_to_queue, priority: 100, reason: "Agent errored"}
        - {type: add_to_dataset, dataset: errors-to-fix}
        - {type: run_evaluator, evaluator: trajectory_match}
        - {type: fire_webhook, url: "https://example.com/hook"}
        - {type: notify, message: "New error trace"}
```

## How a rule fires

A rule fires when **both** hold:

1. **`when` matches** — the shared condition grammar (same as [triage](/docs/agent-evaluation/triage-queue)): `equals`, `in`, `contains`, `exists`, `lt`/`lte`/`gt`/`gte`, dotted field paths (`metadata.score`). A list of conditions is AND-ed; an empty `when` matches everything.
2. **`sample_rate` selects it** — **deterministic** sampling on a hash of `(item id, rule name)`, so re-processing the same item yields the same decision (idempotent, replay-safe).

## Actions

| Action | When it runs | Effect |
|--------|--------------|--------|
| `add_to_queue` | inline (fast) | Boost the item's triage priority so it surfaces first |
| `add_to_dataset` | inline (fast) | Append the item as an example to a dataset |
| `notify` | inline (fast) | Notify connected annotators via SSE |
| `run_evaluator` | background worker | Score the item with an evaluator; store the score on the item |
| `fire_webhook` | background worker | POST `{rule, item_id, item_data}` to an external URL |

Fast actions run inline in the ingestion path; heavy actions (`run_evaluator`, `fire_webhook`) are dispatched to a background worker so ingestion never blocks. Every action records an outcome, and failures are caught — automation never breaks ingestion.

## Inspecting

The admin dashboard links to **Automation** (`/admin/automation`), showing configured rules, activity counters, and recent action outcomes (also available as JSON at `/admin/automation/status` and `/admin/automation/outcomes`).

## Related

- [Full reference on Read the Docs](https://potatoannotator.readthedocs.io/en/latest/agent-evaluation/automation_rules/) — all action types and the condition grammar, version-matched
- [Datasets & Experiments](/docs/agent-evaluation/datasets-and-experiments) — `add_to_dataset` targets
- [Programmatic Evaluators](/docs/agent-evaluation/programmatic-evaluators) — `run_evaluator`
- [Triage Queue](/docs/agent-evaluation/triage-queue) — shares the condition grammar
- [Tracing SDK](/docs/agent-evaluation/tracing-sdk) — a source of incoming traces