LLM Human Evaluation and Preference Data Tools Compared

Teams collect human preference data and LLM ratings with annotation platforms such as Labelbox and Label Studio, with managed services such as Scale AI and Surge AI that also provide the raters, or with free tools such as Argilla, Label Studio Community Edition, and Potato. Potato is a free, self-hosted alternative to Labelbox, Label Studio Enterprise, and the software side of a managed service, for pairwise preferences, Likert and rubric scores, and best-worst scaling with several raters per item and agreement reported on every pair. Potato provides no raters, so the raters are your own people or participants recruited on Prolific.

Human evaluation of a language model asks people to judge the model's outputs. In a pairwise preference judgment, a rater sees one prompt and two responses and picks the better response or calls a tie. In reinforcement learning from human feedback (RLHF), a reward model learns from preference pairs. Direct preference optimization (DPO) trains a model on prompt, chosen, and rejected triples without a reward model. The Bradley-Terry model turns many pairwise outcomes into one strength score per model, and public leaderboards use the model to rank models from votes. A rubric scores one response on several named criteria, often on a Likert scale, and best-worst scaling asks raters for the best and the worst output in a small set. Red teaming labels record whether adversarial prompts got harmful answers. An LLM-as-a-judge scores outputs automatically, and human labels on a sample show whether the judge can be trusted (Zheng et al., 2023).

This page covers human judgments of model outputs. AI Agent Evaluation Tools Compared covers human review of agent traces in LangSmith, Langfuse, Arize Phoenix, and Potato, and LLM evaluation in Potato lists Potato's evaluation tasks in one place.

LLM evaluation tools worth comparing

Licenses and prices come from each project's own pages, checked on 2026-10-29.

The Argilla repository states that the original authors have moved on and that the project will publish bug fixes and patches but no new features. The latest Argilla release, 2.8.0, came out in March 2025. Label Studio's red-teaming and production-RLHF chat templates require Label Studio Enterprise, and the pairwise RLHF and side-by-side templates have no such mark. Arena counts only votes cast while the two models are anonymous, and Arena publishes de-identified conversations in public research datasets.

Which tool for which task

The managed services provide raters as part of the service. The other tools are software, and the team that runs the software brings the raters.

Free Labelbox and Scale AI alternatives

Labelbox sells an annotation platform for preference data, and Scale AI sells the data itself. Labelbox's multimodal chat editor sends one prompt to up to 10 models, live or from imported conversations, and raters rank, select, and rate the responses for model comparison or RLHF. Scale AI sells human feedback and preference ranking with a network of vetted experts and targeted red-teaming, and Scale's RLHF page offers a demo rather than a price. Surge AI describes domain-expert labeling teams, RLHF interfaces with APIs, and red-teaming, and Prolific runs a self-serve participant pool plus managed AI data collection.

Three free tools cover the collection software. Argilla runs on a free Hugging Face Space and sets the number of responses each record needs with min_submitted, and the Argilla documentation describes progress counts and no agreement statistic. Label Studio Community Edition includes the pairwise RLHF template and the side-by-side comparison template, and Label Studio keeps agreement metrics, the overlap setting for several annotators per task, and LLM-as-a-judge features in paid plans. Potato includes agreement, several raters per item, Prolific recruitment, attention checks, and gold items at no cost, and what Potato does not do lists the gaps.

None of the free tools provides raters. A free tool replaces the software a managed service runs, and the rater work still costs participant pay. Prolific's minimum pay is $8.00 an hour, and Prolific recommends $12.00, before the platform fee.

Agreement and recruitment of raters by tool

Preference labels often disagree, because two good answers can differ in style more than in quality. A study therefore sends each pair to several raters and reports how often the raters agree. The table gives each tool's way of sending one item to several people and the agreement measure that the tool's documentation describes.

A dash means we found nothing for the cell in the tool's own documentation, checked on 2026-10-29. If a cell is wrong, tell us.

In Potato, the admin dashboard at /admin/iaa reports agreement for each scheme while the study runs. A pairwise scheme in binary mode is scored as a nominal choice of A, B, or tie, and a scheme in scale mode is scored as a continuous rating. From version 2.13.1, /admin/iaa scores a rubric_eval grid for each criterion and pooled over every criterion, and the reliability export covers pairwise schemes. How many annotators do you need? helps size the number of raters per pair.

Position bias and blinding

LLM judges often favor the response in one position (Zheng et al., 2023), and human raters can show a first-position preference as well. A study that always shows the same model first measures that preference along with quality. Arena hides both model names until the vote and reveals the names afterward. Label Studio's side-by-side template shows the two answers in the order of the input data, and Prodigy's ab.openai.prompts recipe runs a blind comparison of two prompts. Potato's Model Arena is an admin tool that shows each model's label on the response card. The Model Arena suits a quick look at a prompt by the study team.

For a blind study in Potato, list the fields that name the models under item_properties.private_keys. From version 2.13.1, Potato leaves those fields out of everything it sends to the rater's browser and keeps them in the exports. In earlier versions every field of an item reaches the page source, so keep the model names in a separate key file there. A pairwise scheme also accepts randomize_order: true, which gives each rater an order derived from the rater, the item, and the scheme. From version 2.13.1, the exports and the agreement dashboard read each answer in data order, so A always means the first response in the data file. In earlier versions the stored answer is the position shown, and a study with randomize_order maps each answer back to a response from user_state.json before analysis.

A preference study with three raters per pair

The config below runs a blind preference study for raters recruited on Prolific. num_annotators_per_item: 3 sends every pair to three raters, and the dashboard reports agreement on every pair. Three raters give a majority on any pair that no rater calls a tie. The tie option keeps a rater from being forced to choose between two equally good answers, and the optional reason boxes let you audit whether raters rewarded length or formatting. The config is complete and passes Potato's validator.

annotation_task_name: Chat response preference, three raters per pair
task_dir: .
output_annotation_dir: annotation_output/
data_files:
  - data/pairs.jsonl
item_properties:
  id_key: id
  text_key: prompt
user_config:
  allow_all_users: true
  users: []
 
# Raters arrive from Prolific with the participant ID in the URL
login:
  type: url_direct
  url_argument: PROLIFIC_PID
completion_code: C1ABCDEF
auto_redirect_on_completion: true
 
# Three raters per pair, so agreement can be measured on every pair
assignment_strategy: random
num_annotators_per_item: 3
max_annotations_per_user: 40
 
annotation_schemes:
  - annotation_type: pairwise
    name: preference
    description: Which response better answers the prompt?
    items_key: responses
    labels: [Response A, Response B]
    allow_tie: true
    tie_label: About the same
    justification:
      reason_categories: [More accurate, Follows the instruction, Safer, Better written]
      required: false
    label_requirement:
      required: true

Each line of data/pairs.jsonl contains one prompt and two responses, already shuffled, with no model names:

{"id": "p002", "prompt": "Write a one-line commit message for fixing a typo in the README.", "responses": ["This commit fixes a small typographical error that was present in the README file of the repository.", "Fix typo in README"]}

Start the server with potato start config.yaml -p 8000 and give Prolific the study URL https://your-server.com/?PROLIFIC_PID={{%PROLIFIC_PID%}}. A rater who finishes sees the completion code and is sent back to Prolific's completion page. Raters answer with the keys 1, 2, and 0 for a tie. For a graded preference instead of a choice, set mode: scale and put the end captions under scale.labels, as Pairwise comparison shows. A pairwise scheme has no options: key.

From preference labels to DPO data

Potato exports one row per item and rater as CSV, TSV, JSONL, or Parquet, and the huggingface export pushes the annotations and items to a Hugging Face dataset repository. Each row keeps the rater's choice under preference.selection. Version 2.13.1 has no exporter that writes DPO triples from a pairwise scheme. A short script joins the export to the data file instead. Export the annotations first:

python -m potato.export -c config.yaml -f jsonl -o export/

The script below keeps the majority choice for each pair, skips ties and pairs without a majority, and writes the prompt, chosen, and rejected fields that Hugging Face TRL reads for DPOTrainer and RewardTrainer:

import json
from collections import Counter
 
items = {r["id"]: r for r in map(json.loads, open("data/pairs.jsonl"))}
votes = {}
for row in map(json.loads, open("export/annotations.jsonl")):
    choice = row["labels"].get("preference", {}).get("selection")
    votes.setdefault(row["instance_id"], []).append(choice)
 
with open("dpo.jsonl", "w") as out:
    for item_id, choices in votes.items():
        winner, count = Counter(choices).most_common(1)[0]
        if winner not in ("A", "B") or count <= len(choices) / 2:
            continue  # skip ties and pairs without a majority
        item = items[item_id]
        chosen = 0 if winner == "A" else 1
        out.write(json.dumps({"prompt": item["prompt"],
                              "chosen": item["responses"][chosen],
                              "rejected": item["responses"][1 - chosen]}) + "\n")

Keep the per-rater export beside the DPO file. A reward model trained on majority labels discards the disagreement, and the disagreement shows which pairs were hard. The Model Arena's Export DPO button writes the same triples from admin picks in the arena, one per winner and loser. RLHF and preference data covers multi-dimensional preferences and the pitfalls of preference data.

Checking an LLM judge against the raters

An LLM judge replaces raters only after the judge's labels have been compared with blind human labels on the same items. Potato's judge calibration samples each judge several times per item, has people label a sample without seeing the judge's answers, and reports kappa between the judge and the people, kappa between the people, expected calibration error, and the Brier score. Judge calibration covers radio, select, likert, multiselect, and span schemes. To calibrate a preference judge, ask the preference as a radio scheme with A, B, and tie labels. The admin endpoint /admin/api/judge-position-bias runs each sampled item twice, once with the labels reversed, and reports how often the judge's answer flips. LLM judge vs. human agreement gives the procedure, and the judge agreement calculator runs the comparison on a CSV.

What Potato does not do for LLM evaluation

• No raters. Recruit raters on Prolific, a SONA subject pool, or another panel, or invite your own team.
• No DPO export from pairwise schemes. The script above builds the triples from the JSONL export.
• No agreement on rubric grids. Ask each criterion as a likert scheme when agreement per criterion matters.
• A Model Arena for quick checks only. The arena is an admin page that shows model labels, keeps at most 1,000 picks in memory, and loses the picks and the leaderboard when the server restarts.
• No public leaderboard. Arena collects votes from the public, and Potato collects judgments from the raters you invite.
• No production tracing. Score live traffic in an observability tool, then bring a sample to Potato, as AI Agent Evaluation Tools Compared describes.
• Needs a server. Someone installs Potato with pip and runs the server where the raters can reach the server.

Checked against each project's own documentation and pricing pages in October 2026.

Frequently asked questions

Is there a free alternative to Scale AI for RLHF preference data?

Yes, Potato, Argilla, and Label Studio Community Edition are free tools for collecting pairwise preferences from raters you recruit yourself. Potato adds agreement on every pair and Prolific recruitment. No free tool provides trained raters the way Scale AI and Surge AI do. A team that uses a free tool pays the raters directly, for example through Prolific at a minimum of $8.00 an hour plus the platform fee.

How does Chatbot Arena rank models from pairwise votes?

Arena, formerly LMArena and Chatbot Arena, fits a Bradley-Terry model to anonymous pairwise votes. A voter sees two answers from unnamed models and picks one. Only votes cast before the names are revealed count toward the ranking. The open-source Arena-Rank package implements the Bradley-Terry fit and an extension that controls for answer style, such as length and formatting. Potato's Model Arena and the Bradley-Terry calculator rank models the same way from your own raters' choices.

How do I randomize which response appears first in a pairwise comparison?

Shuffle the two responses for each item when you write the data file, and keep a separate key of which model wrote each response. Each model then appears first on about half of the pairs, and every rater sees the same order. The exports and the agreement figures need no correction. Potato also randomizes per rater with randomize_order: true, and from version 2.13.1 the exports and agreement figures read each answer in data order.

Further reading

• LLM evaluation in Potato, for Potato's evaluation tasks in one place
• RLHF and preference data and Rubric-based LLM evaluation
• Pairwise and best-worst scaling and Best-worst scaling
• Human evaluation of generated text
• Running a Crowdsourced Study
• AI Agent Evaluation Tools Compared and Text Annotation Tools Compared
• Model Arena and Inter-Annotator Agreement

References

1. Rafailov, R., Sharma, A., Mitchell, E., Ermon, S., Manning, C. D., & Finn, C. (2023). Direct Preference Optimization: Your Language Model is Secretly a Reward Model. Advances in Neural Information Processing Systems 36 (NeurIPS 2023). https://arxiv.org/abs/2305.18290
2. Zheng, L., Chiang, W.-L., Sheng, Y., et al. (2023). Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. NeurIPS 2023 Datasets and Benchmarks. https://arxiv.org/abs/2306.05685