Skip to content
Potato
Why Potato
Docs
Blog
Showcase
Playground
Tools
Community
Get Started
Open main menu
Showcase
/
Evaluation Tasks
Evaluation Tasks
39 annotation designs you can copy and run. Each ships a
config.yaml
and sample data.
All evaluation tasks designs
#HashtagWars - Learning a Sense of Humor
AgentRewardBench Trajectory Scoring
AITW Mobile Trajectory Review
AndroidWorld: Mobile Agent Task Evaluation
API-Bank Tool Call Review
ArgSciChat Scientific Argumentation Dialogue
Argument Quality Assessment
Belebele Multilingual Reading Comprehension Benchmark
Bias Benchmark for QA (BBQ)
BIG-Bench Task Evaluation
Code Review Annotation (CodeReviewer)
Community Question Answering
Cross-lingual Causal Commonsense Reasoning (XCOPA)
DevBench Repository Evaluation
DiaSafety Dialogue Safety Annotation
Do-Not-Answer: LLM Safety Refusal Dataset
EA-MT - Entity-Aware Machine Translation
Everyday Knowledge Across Diverse Languages and Cultures (BLEnD)
Fact-Checking in Community Question Answering Forums
FActScore: Atomic Factual Precision Evaluation for LLMs
FAVA: Fine-grained Hallucination Detection and Editing for Language Models
Godspeed Questionnaire: AI Assistant Perception
GPQA - Graduate-Level Expert QA Evaluation
HumanEval Code Correctness Evaluation
HumanEval: Python Code Generation Benchmark
IFEval: Instruction-Following Evaluation Benchmark for LLMs
MathDial: Math Tutoring Dialogue Dataset
MMLU Knowledge Evaluation
MMLU-Pro - Tiered Multi-Subject Evaluation
MQM Machine Translation Error Annotation
MultiAgentBench Collaboration Scorecard
Red Team Adversarial Evaluation
SODA: Social Dialogue Quality Evaluation
SORRY-Bench: LLM Safety Refusal Evaluation
SWE-bench: Code Agent Solution Evaluation
TrajEval Staged Evaluation
Triage
WebArena: Realistic Web Agent Evaluation Benchmark
XNLI - Cross-Lingual Natural Language Inference
Other categories
Text Annotation (212)
Image Annotation (41)
Video Annotation (32)
Audio Annotation (33)
Preference Learning (32)
Comparison Tasks (4)
Surveys (63)