Skip to content

Rubric-Based LLM Evaluation

How to score LLM outputs on several defined criteria at once with Potato's rubric_eval grid, MT-Bench style, and how to write criteria that raise agreement.

A rubric breaks a vague judgment ("is this a good answer?") into specific scored criteria (helpfulness, accuracy, completeness, tone, safety), each rated on a scale. It makes LLM evaluation repeatable and reveals why one answer beats another. Benchmarks such as MT-Bench score model answers this way.

A rubric turns subjective quality into a grid of defined criteria and scale points, which raises agreement and makes results interpretable.

When to use a rubric

  • The output is rich enough that a single score loses information.
  • You need to know which dimension is weak (accuracy vs. tone), rather than an overall verdict alone.
  • You want criteria stakeholders agree on up front.

If you only need "which is better", a pairwise comparison is cheaper. Rubrics fit better when you need an absolute, multi-dimensional profile.

Setting it up in Potato

rubric_eval carries the criteria and the scale in one scheme, so an annotator fills the whole grid in a single pass instead of answering a separate question per criterion. Naming the scale points as well as the criteria is what stops two annotators from reading the same number differently:

yaml
annotation_schemes:
  - annotation_type: rubric_eval
    name: answer_quality
    description: "Rate the answer on each criterion."
    scale_points: 5
    scale_labels: ["Poor", "Below average", "Average", "Good", "Excellent"]
    criteria:
      - {name: Helpfulness, description: "Does it address the user's actual need?"}
      - {name: Accuracy,    description: "Is it factually correct?"}
      - {name: Completeness, description: "Does it cover the important points?"}
      - {name: Tone,        description: "Is the style appropriate?"}

Potato renders the scheme as a table with one row per criterion and one column per scale point. Annotators pick one cell in every row, and show_overall: true adds a summary row at the bottom when you want a single number alongside the breakdown.

Writing good criteria

  • Make them independent. Overlapping criteria ("helpful" and "useful") get scored together and add noise.
  • Anchor the scale. Describe what a 1 and a 5 look like for each criterion, rather than only the ends of the scale as a whole.
  • Keep it short. Four to six criteria is a common range, and long rubrics fatigue annotators and lower agreement.

Rubrics and LLM-as-judge

The same rubric you give humans can prompt an "LLM judge" for cheap pre-scoring, then have humans verify, as in LLM pre-annotation. Keep a human-scored sample to check the judge against, and watch for the judge's own biases (length, formatting, self-preference).

Further reading