Skip to content

量規評估

在 Potato 中搭建多維度評估表格，用於 LLM 輸出評價、作文評分、翻譯品質評估，以及各類結構化的量規打分任務。

量規評估標註方案提供結構化的表格介面，在多個維度上按統一量表給內容打分。它適合評價 LLM 輸出、批改作文、評估翻譯品質，以及任何需要結構化多維打分的任務。

Potato rubric evaluation grid scoring a response on accuracy, relevance, and fluencyRubric evaluation in Potato

概覽

量規評估方案包含：

  • 維度表格，每個維度都有自己的評分量表
  • 量表標籤，從 Poor 到 Excellent（可自定義）
  • 可選的總體評分，對各維度做一個彙總
  • 維度說明，指導標註者理解每一項在評什麼

它特別適合對生成式 AI 輸出做結構化的人工評估。

快速開始

yaml
annotation_schemes:
  - annotation_type: rubric_eval
    name: response_quality
    description: Evaluate the quality of this AI-generated response.
    scale_points: 5
    criteria:
      - name: Accuracy
        description: Is the information factually correct?
      - name: Relevance
        description: Does the response address the question?
      - name: Fluency
        description: Is the response well-written and natural?

配置項

欄位類型預設值說明
annotation_typestring必填必須為 "rubric_eval"
namestring必填本方案的唯一識別符號
descriptionstring必填展示給標註者的說明文字
scale_pointsinteger5評分量表的檔位數
scale_labelsarray["Poor", "Fair", "Average", "Good", "Excellent"]每個檔位的標籤（數量必須與 scale_points 一致）
criteriaarray必填維度物件列表，每項包含 name 和可選的 description
show_overallbooleanfalse是否在各維度下方額外顯示一行總體評分

示例

LLM 輸出評估

yaml
annotation_schemes:
  - annotation_type: rubric_eval
    name: llm_eval
    description: Rate the quality of this model-generated response.
    scale_points: 5
    scale_labels:
      - Poor
      - Fair
      - Average
      - Good
      - Excellent
    show_overall: true
    criteria:
      - name: Helpfulness
        description: Does the response provide useful and actionable information?
      - name: Accuracy
        description: Is the response factually correct and free of hallucinations?
      - name: Harmlessness
        description: Is the response free of harmful, biased, or inappropriate content?
      - name: Coherence
        description: Is the response logically structured and easy to follow?

作文評分

yaml
annotation_schemes:
  - annotation_type: rubric_eval
    name: essay_grade
    description: Grade this student essay using the rubric below.
    scale_points: 4
    scale_labels:
      - Below Expectations
      - Approaching
      - Meets Expectations
      - Exceeds Expectations
    criteria:
      - name: Thesis
        description: Is there a clear and arguable thesis statement?
      - name: Evidence
        description: Does the essay use relevant evidence to support claims?
      - name: Organization
        description: Is the essay logically organized with clear transitions?
      - name: Grammar
        description: Is the writing free of grammatical and spelling errors?

翻譯品質評估

yaml
annotation_schemes:
  - annotation_type: rubric_eval
    name: translation_quality
    description: Evaluate the quality of this machine translation.
    scale_points: 3
    scale_labels:
      - Unacceptable
      - Acceptable
      - Perfect
    criteria:
      - name: Adequacy
        description: Does the translation convey the same meaning as the source?
      - name: Fluency
        description: Does the translation read naturally in the target language?
      - name: Terminology
        description: Are domain-specific terms translated correctly?

輸出格式

json
{
  "response_quality": {
    "labels": {
      "Accuracy": 4,
      "Relevance": 5,
      "Fluency": 3
    },
    "overall": 4
  }
}

每個維度對應它被選中的檔位值（從 1 開始計數）。只有 show_overall 為 true 時才會有 overall 欄位。

實踐建議

  1. 讓各維度互相獨立 —— 每個維度應該衡量不同的方面，避免重複打分
  2. 說明寫清楚 —— 標註者要能毫無歧義地知道每個維度在評什麼
  3. 用 3 到 5 個檔位 —— 檔位少認知負擔小，超過 7 檔一般也提升不了信度
  4. 給出錨點例子 —— 在說明裡點明量表兩端各自代表什麼
  5. 需要彙總時開啟總體評分 —— 既要細分維度、又要一個概括指標時，show_overall 很有用

延伸閱讀

有關實現詳情，請參閱源文件。