Agent Evaluation
Human and LLM-judge evaluation of agent output: judge calibration, trace review, trajectory editing, programmatic evaluators, and CI evaluation.
This section treats a model's output as something to be reviewed rather than something to be labeled. The unit of work is usually a trace: one run of an agent, with its tool calls, intermediate steps and final answer.
LLM-as-Judge Calibration and Judge ↔ Human Alignment cover the case where a model does the grading and you need to know how far to trust it. Three-Pane Trace Evaluation covers the case where a person does. CI Evaluation covers running either on every commit.
Potato does not train the agents or judges you evaluate. Everything here produces labels, scores, and agreement statistics, and any training on those labels happens outside the tool.