Ranking Task

Drag-and-drop ranking interface to order items from best to worst.

⚖️

comparison annotation

Configuration Fileconfig.yaml

# Ranking Task Configuration
# Drag-and-drop interface to order items

annotation_task_name: "Response Ranking"

data_files:
  - "data/ranking_sets.json"

item_properties:
  id_key: "id"
  text_display_key: "prompt"
  list_display_key: "responses"

user_config:
  allow_all_users: true

annotation_schemes:
  - annotation_type: "ranking"
    name: "response_ranking"
    description: "Rank the responses from BEST (1) to WORST"
    allow_ties: false
    show_numbers: true

  - annotation_type: "radio"
    name: "ranking_criterion"
    description: "What criterion did you primarily use for ranking?"
    labels:
      - name: "Accuracy"
        tooltip: "Correctness of information"
      - name: "Clarity"
        tooltip: "How easy to understand"
      - name: "Helpfulness"
        tooltip: "How useful the response is"
      - name: "Engagement"
        tooltip: "How interesting or engaging"
      - name: "Multiple criteria"
        tooltip: "Combination of factors"

  - annotation_type: "likert"
    name: "confidence"
    description: "How confident are you in this ranking?"
    size: 5
    min_label: "Not confident"
    max_label: "Very confident"

  - annotation_type: "text"
    name: "notes"
    description: "Any notes about your ranking decision"
    required: false

output: "annotation_output/"

Sample Datasample-data.json

[
  {
    "id": "rank_001",
    "prompt": "What are the benefits of regular exercise?",
    "responses": [
      "Exercise is good for you and you should do it.",
      "Regular exercise offers numerous benefits including improved cardiovascular health, better mood through endorphin release, weight management, stronger muscles and bones, and reduced risk of chronic diseases like diabetes.",
      "Working out helps with: 1) Heart health 2) Mental wellness 3) Weight control 4) Energy levels. Even 30 minutes daily makes a big difference!",
      "Exercise benefits include health improvements, better sleep, stress reduction, and increased longevity based on numerous scientific studies."
    ]
  },
  {
    "id": "rank_002",
    "prompt": "How do I make scrambled eggs?",
    "responses": [
      "Crack eggs, cook them, stir.",
      "Beat 2-3 eggs with a splash of milk. Heat butter in a pan over medium-low heat. Pour in eggs and gently stir with a spatula, pushing curds from edges to center. Remove when slightly wet - they'll finish cooking off heat. Season with salt and pepper.",
      "Making scrambled eggs: First crack some eggs into a bowl. Add milk if you want. Put butter in a pan and heat it up. Pour the eggs in and stir them around until they're cooked. Add salt.",
      "For perfect scrambled eggs: whisk eggs vigorously, cook low and slow, remove from heat early, and let residual heat finish the job."
    ]
  }
]

// ... and 1 more items

Get This Design

View on GitHub

Clone or download from the repository

Quick start:

git clone https://github.com/davidjurgens/potato-showcase.git
cd potato-showcase/ranking-task
potato start config.yaml

Details

Annotation Types

ranking

Domain

nlpevaluation

Use Cases

rankingpreference-learningordering

Related Designs

Best-Worst Scaling

MaxDiff annotation where annotators select the best and worst items from a set for relative comparison.

best-worst

Dialogue Act Labeling

Classify utterances in conversations by their communicative function (question, statement, request, etc.).

radio

Fact Verification

Verify claims as supported, refuted, or not enough information based on provided evidence.

radiotext