Math Question Answering and Category Classification
Answering SAT-style mathematical questions with category classification, covering closed-vocabulary algebra, open-vocabulary algebra, and geometry. Based on SemEval-2019 Task 10 (Math QA).
Configuration Fileconfig.yaml
This Potato config reproduces the annotation task. Save it as config.yaml and run potato start config.yaml to try it.
# Math Question Answering and Category Classification
# Based on Hopkins et al., SemEval 2019
# Paper: https://aclanthology.org/S19-2153/
# Dataset: https://competitions.codalab.org/competitions/20013
#
# This task asks annotators to solve SAT-style mathematical questions and
# classify them by category. Annotators provide the answer to the question
# (numeric or multiple-choice) and identify its mathematical category.
#
# Category Labels (following the shared task's three-way taxonomy):
# - Closed-vocabulary Algebra: symbolic algebra questions using a restricted
# vocabulary (equations, expressions, functions, polynomials)
# - Open-vocabulary Algebra: algebra word problems requiring reasoning over
# free-form natural language
# - Geometry: shapes, areas, angles, and spatial reasoning (may include a diagram)
annotation_task_name: "Math Question Answering"
task_dir: "."
data_files:
- sample-data.json
item_properties:
id_key: "id"
text_key: "text"
output_annotation_dir: "annotation_output/"
output_annotation_format: "json"
port: 8000
server_name: localhost
annotation_schemes:
- annotation_type: text
name: answer
description: "Provide the answer to the question (a number, or the letter of the correct multiple-choice option)."
- annotation_type: radio
name: math_category
description: "Which category does this question belong to?"
labels:
- "Closed-vocabulary Algebra"
- "Open-vocabulary Algebra"
- "Geometry"
keyboard_shortcuts:
"Closed-vocabulary Algebra": "1"
"Open-vocabulary Algebra": "2"
"Geometry": "3"
tooltips:
"Closed-vocabulary Algebra": "Symbolic algebra questions using a restricted vocabulary (equations, expressions, functions, polynomials)"
"Open-vocabulary Algebra": "Algebra word problems requiring reasoning over free-form natural language"
"Geometry": "Questions involving shapes, areas, angles, or spatial reasoning (may include a diagram)"
annotation_instructions: |
You will be shown a SAT-style math question along with its difficulty level. Your task is to:
1. Solve the question and provide the answer (a number, or the letter of the correct multiple-choice option) in the text field.
2. Classify the question into one of three categories: Closed-vocabulary Algebra, Open-vocabulary Algebra, or Geometry.
html_layout: |
<div style="padding: 15px; max-width: 800px; margin: auto;">
<div style="background: #fefce8; border: 1px solid #fde68a; border-radius: 8px; padding: 12px; margin-bottom: 12px;">
<strong style="color: #a16207;">Difficulty:</strong>
<span style="font-size: 15px;">{{difficulty}}</span>
</div>
<div style="background: #f0f9ff; border: 1px solid #bae6fd; border-radius: 8px; padding: 16px; margin-bottom: 16px;">
<strong style="color: #0369a1;">Problem:</strong>
<p style="font-size: 16px; line-height: 1.7; margin: 8px 0 0 0;">{{text}}</p>
</div>
</div>
allow_all_users: true
instances_per_annotator: 50
annotation_per_instance: 2
allow_skip: true
skip_reason_required: false
Sample Datasample-data.json
[
{
"id": "mathqa_001",
"text": "If 3x + 7 = 22, what is the value of x?",
"difficulty": "Easy"
},
{
"id": "mathqa_002",
"text": "Suppose 3x + y = 15, where x is a positive integer. What is the difference between the largest possible value of y and the smallest possible value of x, assuming that y is also a positive integer?",
"difficulty": "Medium"
}
]
// ... and 8 more itemsGet This Design
Clone or download from the repository
Quick start:
git clone https://github.com/davidjurgens/potato-showcase.git cd potato-showcase/semeval/2019/task10-math-qa potato start config.yaml
Dataset & paper
Hopkins et al., SemEval 2019
Citation (BibTeX)
@inproceedings{hopkins-etal-2019-semeval,
title = "{S}em{E}val-2019 Task 10: Math Question Answering",
author = "Hopkins, Mark and Le Bras, Ronan and Petrescu-Prahova, Cristian and Stanovsky, Gabriel and Hajishirzi, Hannaneh and Koncel-Kedziorski, Rik",
booktitle = "Proceedings of the 13th International Workshop on Semantic Evaluation",
month = jun,
year = "2019",
address = "Minneapolis, Minnesota, USA",
publisher = "Association for Computational Linguistics",
url = "https://aclanthology.org/S19-2153",
doi = "10.18653/v1/S19-2153",
pages = "893--899"
}Details
Annotation Types
Domain
Use Cases
Tags
Found an issue or want to improve this design?
Open an IssueRelated Designs
BRAINTEASER - Commonsense-Defying QA
Lateral thinking and commonsense-defying question answering task requiring annotators to select answers to brain teasers that defy default commonsense assumptions and provide explanations. Based on SemEval-2024 Task 9 (BRAINTEASER).
QA over Tabular Data
Question answering over tabular data: annotators answer natural-language questions from a table by providing the value or list derived from it, and label the answer type (boolean, category, number, or list). Based on SemEval-2025 Task 8 (DataBench).
R2VQ - Recipe Question Answering
Question answering over recipe texts, where annotators answer questions about recipes and classify whether the answer is explicitly stated, inferable, or not answerable from the recipe. A simplified adaptation of SemEval-2022 Task 9 (R2VQ, Tu et al.), whose original competence-based protocol answers template-based question families over recipes annotated with cooking-role and semantic-role layers.