MMLU-Pro - Tiered Multi-Subject Evaluation
Tiered evaluation for multi-subject question answering, based on MMLU-Pro (Wang et al., NeurIPS 2024). Annotators verify answers to challenging 10-option multiple choice questions across STEM and humanities subjects, using a tiered annotation scheme for topic and subtopic categorization.
About this dataset
MMLU-Pro is a multiple-choice benchmark that extends the original MMLU across academic and professional subjects. It was built by Yubo Wang and colleagues (University of Waterloo and collaborators) and presented as a spotlight at the NeurIPS 2024 Datasets and Benchmarks track. It exists because leading models had begun to saturate MMLU, leaving little room to distinguish strong systems.
The questions were drawn and revised from MMLU and other sources, with the answer set expanded from four options to ten and trivial or noisy items removed. Each question has a single correct answer among up to ten options (A through J). The annotation task is to categorize a question by subject and select the correct option.
The dataset holds 12,032 questions across 14 disciplines: math, physics, chemistry, law, engineering, economics, health, psychology, business, biology, philosophy, computer science, history, and an 'other' category. Adding options and pruning weak items lowered model accuracy by 16 to 33 percent relative to MMLU and made scores more stable across prompt wordings (about 2 percent variation versus 4 to 5 percent on MMLU).
The Potato config below reproduces this task with a tiered annotation scheme that captures a topic and a dependent subtopic, plus a radio scheme for the ten answer options (A-J). Use it to sort questions into a subject hierarchy while verifying the correct answer.
- Released
- NeurIPS 2024 D&B (spotlight)
- Questions
- 12,032
- Disciplines
- 14
- Answer options
- Up to 10 (A-J)
- Accuracy drop vs MMLU
- 16-33%
- Prompt sensitivity
- ~2% (vs 4-5% in MMLU)
Configuration Fileconfig.yaml
This Potato config reproduces the annotation task. Save it as config.yaml and run potato start config.yaml to try it.
# MMLU-Pro - Tiered Multi-Subject Evaluation
# Based on Wang et al., NeurIPS 2024
# Paper: https://arxiv.org/abs/2406.01574
# Dataset: https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro
#
# MMLU-Pro extends MMLU with 10 answer options (A-J) instead of 4,
# making it significantly more challenging. This task uses a tiered
# annotation scheme to categorize questions by topic and subtopic,
# alongside answer selection.
#
# The tiered annotation allows organizing questions hierarchically:
# - Topic: The broad subject area (e.g., Biology, Physics, History)
# - Subtopic: A more specific area within the topic
#
# Answer options: A through J (10 choices per question)
annotation_task_name: "MMLU-Pro: Tiered Multi-Subject Evaluation"
task_dir: "."
data_files:
- sample-data.json
item_properties:
id_key: "id"
text_key: "text"
output_annotation_dir: "annotation_output/"
output_annotation_format: "json"
port: 8000
server_name: localhost
annotation_schemes:
- annotation_type: tiered_annotation
name: subject_classification
description: "Classify the question by topic and subtopic using the tiered hierarchy"
source_field: "audio_url"
media_type: "audio"
tiers:
- name: "topic"
tier_type: "independent"
- name: "subtopic"
tier_type: "dependent"
parent_tier: "topic"
constraint_type: "symbolic_association"
- annotation_type: radio
name: correct_answer
description: "Select the correct answer from the 10 options (A-J)"
labels:
- "A"
- "B"
- "C"
- "D"
- "E"
- "F"
- "G"
- "H"
- "I"
- "J"
keyboard_shortcuts:
"A": "1"
"B": "2"
"C": "3"
"D": "4"
"E": "5"
"F": "6"
"G": "7"
"H": "8"
"I": "9"
"J": "0"
annotation_instructions: |
You will evaluate challenging multiple-choice questions from MMLU-Pro.
1. Read the question and all 10 answer options carefully.
2. Classify the question by topic and subtopic using the tiered scheme.
3. Select the single correct answer (A through J).
4. These questions are intentionally difficult and may require expert knowledge.
html_layout: |
<div style="padding: 15px; max-width: 800px; margin: auto;">
<div style="background: #f0f9ff; border: 1px solid #bae6fd; border-radius: 8px; padding: 16px; margin-bottom: 16px;">
<span style="display: inline-block; background: #0369a1; color: white; padding: 2px 10px; border-radius: 12px; font-size: 13px; margin-bottom: 8px;">{{subject}}</span>
<p style="font-size: 16px; font-weight: 600; line-height: 1.6; margin: 8px 0 0 0;">{{text}}</p>
</div>
<div style="background: #f8fafc; border: 1px solid #e2e8f0; border-radius: 8px; padding: 16px; margin-bottom: 16px;">
<strong style="color: #475569;">Answer Options:</strong>
<p style="font-size: 15px; line-height: 1.8; margin: 8px 0 0 0; white-space: pre-line;">{{options}}</p>
</div>
</div>
allow_all_users: true
instances_per_annotator: 50
annotation_per_instance: 3
allow_skip: true
skip_reason_required: false
Sample Datasample-data.json
[
{
"id": "mmlu_pro_001",
"text": "Which of the following best describes the role of topoisomerase II in DNA replication?",
"options": "A. It unwinds the double helix ahead of the replication fork\nB. It synthesizes RNA primers for Okazaki fragments\nC. It relieves positive supercoiling by making transient double-strand breaks\nD. It joins Okazaki fragments on the lagging strand\nE. It proofreads newly synthesized DNA\nF. It degrades RNA primers after replication\nG. It adds telomeric sequences to chromosome ends\nH. It methylates newly synthesized DNA strands\nI. It prevents re-replication by licensing origins\nJ. It stabilizes single-stranded DNA at the replication fork",
"subject": "Biology",
"audio_url": ""
},
{
"id": "mmlu_pro_002",
"text": "A projectile is launched at an angle of 60 degrees above the horizontal with an initial speed of 50 m/s. Ignoring air resistance, what is the maximum height reached by the projectile?",
"options": "A. 45.9 m\nB. 55.7 m\nC. 63.8 m\nD. 76.5 m\nE. 85.3 m\nF. 95.7 m\nG. 102.4 m\nH. 110.2 m\nI. 127.6 m\nJ. 143.1 m",
"subject": "Physics",
"audio_url": ""
}
]
// ... and 8 more itemsGet This Design
Clone or download from the repository
Quick start:
git clone https://github.com/davidjurgens/potato-showcase.git cd potato-showcase/preference-learning/mmlu-pro-tiered-eval potato start config.yaml
Dataset & paper
Wang et al., NeurIPS 2024 (Datasets & Benchmarks)
Citation (BibTeX)
@inproceedings{wang2024mmlupro,
title = "{MMLU}-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmark",
author = "Wang, Yubo and Ma, Xueguang and Zhang, Ge and Ni, Yuansheng and Chandra, Abhranil and Guo, Shiguang and Ren, Weiming and Arulraj, Aaran and He, Xuan and Jiang, Ziyan and Li, Tianle and Ku, Max and Wang, Kai and Zhuang, Alex and Fan, Rongqi and Yue, Xiang and Chen, Wenhu",
booktitle = "Advances in Neural Information Processing Systems (NeurIPS), Datasets and Benchmarks Track",
year = "2024",
url = "https://arxiv.org/abs/2406.01574"
}Details
Annotation Types
Domain
Use Cases
Tags
Found an issue or want to improve this design?
Open an IssueRelated Designs
MathDial: Math Tutoring Dialogue Dataset
MathDial is a dataset of 2,861 one-to-one math tutoring dialogues grounded in GSM8K word problems, annotated with a four-category teacher-move taxonomy (Focus, Probing, Telling, Generic). This Potato config reproduces that teacher-move labeling task.
Student Essay Discourse Element Classification
Discourse element annotation of student essays based on Song et al. (EMNLP 2020). Annotators identify argumentative discourse units, classify essay types, and tag rhetorical strategies used in student writing.
#HashtagWars - Learning a Sense of Humor
Humor ranking of tweets submitted to Comedy Central's @midnight #HashtagWars, classifying comedic quality. Based on SemEval-2017 Task 6.