Skip to content

量规评估

在 Potato 中搭建多维度评估表格，用于 LLM 输出评价、作文评分、翻译质量评估，以及各类结构化的量规打分任务。

量规评估标注方案提供结构化的表格界面，在多个维度上按统一量表给内容打分。它适合评价 LLM 输出、批改作文、评估翻译质量，以及任何需要结构化多维打分的任务。

Potato rubric evaluation grid scoring a response on accuracy, relevance, and fluencyRubric evaluation in Potato

概览

量规评估方案包含：

  • 维度表格，每个维度都有自己的评分量表
  • 量表标签，从 Poor 到 Excellent（可自定义）
  • 可选的总体评分，对各维度做一个汇总
  • 维度说明，指导标注者理解每一项在评什么

它特别适合对生成式 AI 输出做结构化的人工评估。

快速开始

yaml
annotation_schemes:
  - annotation_type: rubric_eval
    name: response_quality
    description: Evaluate the quality of this AI-generated response.
    scale_points: 5
    criteria:
      - name: Accuracy
        description: Is the information factually correct?
      - name: Relevance
        description: Does the response address the question?
      - name: Fluency
        description: Is the response well-written and natural?

配置项

字段类型默认值说明
annotation_typestring必填必须为 "rubric_eval"
namestring必填本方案的唯一标识符
descriptionstring必填展示给标注者的说明文字
scale_pointsinteger5评分量表的档位数
scale_labelsarray["Poor", "Fair", "Average", "Good", "Excellent"]每个档位的标签（数量必须与 scale_points 一致）
criteriaarray必填维度对象列表，每项包含 name 和可选的 description
show_overallbooleanfalse是否在各维度下方额外显示一行总体评分

示例

LLM 输出评估

yaml
annotation_schemes:
  - annotation_type: rubric_eval
    name: llm_eval
    description: Rate the quality of this model-generated response.
    scale_points: 5
    scale_labels:
      - Poor
      - Fair
      - Average
      - Good
      - Excellent
    show_overall: true
    criteria:
      - name: Helpfulness
        description: Does the response provide useful and actionable information?
      - name: Accuracy
        description: Is the response factually correct and free of hallucinations?
      - name: Harmlessness
        description: Is the response free of harmful, biased, or inappropriate content?
      - name: Coherence
        description: Is the response logically structured and easy to follow?

作文评分

yaml
annotation_schemes:
  - annotation_type: rubric_eval
    name: essay_grade
    description: Grade this student essay using the rubric below.
    scale_points: 4
    scale_labels:
      - Below Expectations
      - Approaching
      - Meets Expectations
      - Exceeds Expectations
    criteria:
      - name: Thesis
        description: Is there a clear and arguable thesis statement?
      - name: Evidence
        description: Does the essay use relevant evidence to support claims?
      - name: Organization
        description: Is the essay logically organized with clear transitions?
      - name: Grammar
        description: Is the writing free of grammatical and spelling errors?

翻译质量评估

yaml
annotation_schemes:
  - annotation_type: rubric_eval
    name: translation_quality
    description: Evaluate the quality of this machine translation.
    scale_points: 3
    scale_labels:
      - Unacceptable
      - Acceptable
      - Perfect
    criteria:
      - name: Adequacy
        description: Does the translation convey the same meaning as the source?
      - name: Fluency
        description: Does the translation read naturally in the target language?
      - name: Terminology
        description: Are domain-specific terms translated correctly?

输出格式

json
{
  "response_quality": {
    "labels": {
      "Accuracy": 4,
      "Relevance": 5,
      "Fluency": 3
    },
    "overall": 4
  }
}

每个维度对应它被选中的档位值（从 1 开始计数）。只有 show_overall 为 true 时才会有 overall 字段。

实践建议

  1. 让各维度互相独立 —— 每个维度应该衡量不同的方面，避免重复打分
  2. 说明写清楚 —— 标注者要能毫无歧义地知道每个维度在评什么
  3. 用 3 到 5 个档位 —— 档位少认知负担小，超过 7 档一般也提升不了信度
  4. 给出锚点例子 —— 在说明里点明量表两端各自代表什么
  5. 需要汇总时打开总体评分 —— 既要细分维度、又要一个概括指标时，show_overall 很有用

延伸阅读

有关实现详情，请参阅源文档。