Potato collects the human judgments that LLM evaluation and post-training need, including pairwise preferences with ties for RLHF and DPO, rubric scores, error spans in generated text, and the human labels for checking an LLM-as-a-judge. Potato is free and open source and runs on your own server, so unreleased model outputs and private prompts stay on your infrastructure.
Human judgments are still the reference against which models and automatic metrics are measured, and collecting those judgments well is an annotation problem. A team has to choose a response format and the number of raters per item, keep raters from favoring the longer answer, and show that the labels are reliable. Potato handles the study side, which covers randomized side-by-side presentation, several raters per item, agreement as the study runs, and exports that keep every rater's answer.
LLM evaluation tasks in Potato
Each row is an annotation type or tool that ships with Potato, and the linked page has its configuration.
| Task | How Potato does it | Docs |
|---|---|---|
| Pairwise preference for reward models and DPO | pairwise with binary, scale, or multi-dimension modes, ties, and randomized A/B order | Pairwise comparison |
| Rubric grading | rubric_eval with named criteria and a score per criterion | Rubric evaluation |
| Ranking several outputs | Best-worst scaling, which turns best and worst picks from small sets of outputs into scores | Best-worst scaling |
| Finding errors and hallucinations | Error spans marked inside the generated text, MQM style | Error span |
| Checking an LLM judge | Human labels on a sample, compared with the judge's on the same items | LLM judge vs. human agreement |
| Comparing models live | One prompt sent to several models side by side, with a win-rate leaderboard | Model Arena |
| RAG and grounded answers | Answer quality and support judged against the retrieved passages | RAG evaluation |
A preference-collection task
The annotation_schemes block below asks raters for a binary preference with a tie option and a one-sentence reason. The full task, with the prompt and the two responses on the page, is in RLHF and preference data.
annotation_schemes:
- annotation_type: pairwise
name: preference
description: "Which response better follows the instruction and is more helpful and harmless?"
mode: binary
allow_tie: true
- annotation_type: text
name: rationale
description: "One sentence on why you chose it."
label_requirement:
required: falseThe rationale is optional because writing one slows raters down, though the rationale is worth collecting on at least a sample. With rationales, you can audit the preference data for raters who rewarded length or formatting instead of quality.
Working alongside Label Studio, Argilla, Langfuse, and LMArena
Label Studio has a pairwise template for human preference collection, and Argilla has rating and ranking questions for building preference datasets. Langfuse lets a team score traces from production in annotation queues. Chatbot Arena, now LMArena, set the pattern for blind side-by-side voting ranked with a Bradley-Terry model (Zheng et al., 2023). Potato's emphasis is the evaluation as a study, with several raters per item, agreement measured on every scheme, attention and gold checks for crowd raters, and statistics that run in the browser. The Bradley-Terry calculator ranks models from pairwise judgments, and the judge agreement calculator tests an LLM judge against people.
Frequently asked questions
How do I check whether my LLM-as-a-judge agrees with human raters?
Have at least two people label a sample of items without seeing the judge's answers, then compare the judge's agreement with those people to the people's agreement with each other. A judge whose Cohen's kappa with the humans is close to the humans' kappa with each other is doing about as well as another annotator. LLM judge vs. human agreement gives the procedure, and the judge agreement calculator runs the comparison on a CSV.
Should I use pairwise comparisons, Likert ratings, or rubrics?
Use pairwise comparisons when you need preference data or a ranking of models, because raters compare two outputs more reliably than they assign stable absolute scores. Use a rubric when you need to know why one output is better, since a score per criterion separates helpfulness from accuracy. Likert ratings of single outputs suit quick quality checks. Pairwise and best-worst scaling and Rating scales compare the formats.
Can I run a blind side-by-side comparison without sending data to a third party?
Yes. Potato runs on your own server and shows the two outputs in randomized order, so raters cannot tell which model produced which. Nothing leaves your server unless you configure a hosted AI endpoint, and the Model Arena can call local models through Ollama or vLLM.
How many raters per item do I need for preference labels?
The number depends on how much raters disagree on your items. Two raters on every item let you measure agreement directly, and an odd number of raters gives a majority on a two-way choice when ties are not allowed. How many annotators do you need? and the annotation power planner help size a study for a target precision.
Further reading
- LLM human evaluation tools compared, for Argilla, Label Studio, Prodigy, Arena, and managed rater services
- Human evaluation of generated text, for designing the evaluation
- Detecting hallucinations, for marking unsupported claims with span annotation
- Trajectory editing for SFT and DPO, for corrected agent runs as training pairs
- AI agent evaluation and NLP and text, for related pages on this site
References
- Zheng et al. (2023). Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. NeurIPS 2023 Datasets and Benchmarks. https://arxiv.org/abs/2306.05685