Skip to content
Showcase/Best-Worst Scaling
beginnercomparison

Best-Worst Scaling

MaxDiff annotation where annotators select the best and worst items from a set for relative comparison.

Submit

ملف الإعدادconfig.yaml

# Best-Worst Scaling (MaxDiff) Configuration
# Annotators select the best and worst items from a set

task_dir: "."
annotation_task_name: "Best-Worst Scaling"

data_files:
  - "data/comparison_sets.json"

item_properties:
  id_key: "id"
  text_key: "prompt"
  text_display_key: "prompt"
  list_display_key: "options"

user_config:
  allow_all_users: true

annotation_schemes:
  - annotation_type: "radio"
    name: "best_response"
    description: "Select the BEST response to the given prompt"
    labels:
      - "Option A"
      - "Option B"
      - "Option C"
      - "Option D"

  - annotation_type: "radio"
    name: "worst_response"
    description: "Select the WORST response to the given prompt"
    labels:
      - "Option A"
      - "Option B"
      - "Option C"
      - "Option D"

  - annotation_type: "text"
    name: "reasoning"
    description: "Briefly explain your choices (optional)"
    required: false

  - annotation_type: "radio"
    name: "difficulty"
    description: "How difficult was this comparison?"
    labels:
      - name: "Easy"
        tooltip: "Clear difference between options"
      - name: "Moderate"
        tooltip: "Some thought required"
      - name: "Hard"
        tooltip: "Very similar quality, difficult to choose"

output: "annotation_output/"

output_annotation_dir: "annotation_output/"
output_annotation_format: "json"

بيانات نموذجيةsample-data.json

[
  {
    "id": "bws_001",
    "prompt": "Write a haiku about spring",
    "options": [
      "Cherry blossoms fall / Gentle breeze carries petals / Nature awakens",
      "Spring is here now / Flowers bloom in the garden / Birds sing songs today",
      "Pink flowers bloom / The sun is warm and nice / I like spring a lot",
      "Melting snow reveals / Hidden crocuses emerge / Life begins anew"
    ]
  },
  {
    "id": "bws_002",
    "prompt": "Explain photosynthesis to a 10-year-old",
    "options": [
      "Plants eat sunlight and make food from it.",
      "Photosynthesis is like cooking for plants! They use sunlight as their stove, water and air as ingredients, and make their own food called glucose. The bonus? They release oxygen for us to breathe!",
      "Plants absorb light energy through chlorophyll in their leaves, converting carbon dioxide and water into glucose and oxygen through a complex biochemical process.",
      "Plants are green because they have chlorophyll. This helps them make food from the sun."
    ]
  }
]

// ... and 1 more items

احصل على هذا التصميم

View on GitHub

Clone or download from the repository

بدء سريع:

git clone https://github.com/davidjurgens/potato-showcase.git
cd potato-showcase/templates/comparison/best-worst-scaling
potato start config.yaml

التفاصيل

أنواع التوسيم

best-worst

المجال

nlpevaluation

حالات الاستخدام

preference-learningrankingquality-assessment

الوسوم

comparisonmaxdiffbest-worstrankingpreference

وجدت مشكلة أو تريد تحسين هذا التصميم؟

افتح مشكلة

تصاميم ذات صلة

Ranking Task

Drag-and-drop ranking interface to order items from best to worst.

ranking

Dialogue Act Labeling

Classify utterances in conversations by their communicative function (question, statement, request, etc.).

radio

Fact Verification

Verify claims as supported, refuted, or not enough information based on provided evidence.

radiotext