Detecting Machine-Generated Code
SemEval-2026 Task 13: detecting machine-generated code across multiple programming languages, generators, and application scenarios. Annotators judge whether a code snippet is human-written or machine-generated (Subtask A), attribute the generator family (Subtask B), and give a fine-grained origin label (Subtask C: fully human, fully AI, human-AI hybrid, or AI tuned to look human-like). Training code is in C++/Python/Java, with unseen languages and domains at test time.
Configuration Fileconfig.yaml
This Potato config reproduces the annotation task. Save it as config.yaml and run potato start config.yaml to try it.
# Detecting Machine-Generated Code
# Based on Orel et al., SemEval-2026 Task 13
# Paper: https://aclanthology.org/2026.semeval-1.445/
# Task repo: https://github.com/mbzuai-nlp/SemEval-2026-Task13
#
# Given a code snippet, decide whether it was written by a human or
# generated by a model, across many languages and usage scenarios. The
# shared task has three subtasks:
# - Subtask A: binary human vs. machine-generated
# - Subtask B: generator-family attribution (which model family)
# - Subtask C: fine-grained origin (fully human, fully AI, human-AI
# hybrid, or a model tuned/prompted to look human-like)
# Training code is C++/Python/Java; test adds unseen languages
# (Go, PHP, C#, C, JavaScript) and unseen domains (research, production).
annotation_task_name: "Machine-Generated Code Detection"
task_dir: "."
data_files:
- sample-data.json
item_properties:
id_key: "id"
text_key: "code"
output_annotation_dir: "annotation_output/"
output_annotation_format: "json"
port: 8000
server_name: localhost
annotation_schemes:
- annotation_type: radio
name: origin
description: "Subtask A: Was this code written by a human or generated by a machine?"
labels:
- "Human-written"
- "Machine-generated"
keyboard_shortcuts:
"Human-written": "1"
"Machine-generated": "2"
- annotation_type: radio
name: fine_grained_origin
description: "Subtask C: fine-grained origin"
labels:
- "Fully human-written"
- "Fully AI-generated"
- "Human-AI hybrid"
- "AI tuned/prompted to look human-like"
- annotation_type: text
name: generator_family
description: "Subtask B: if machine-generated, which model family do you think produced it? (optional)"
textarea: false
required: false
placeholder: "e.g. GPT, Llama, Qwen, DeepSeek..."
annotation_instructions: |
Read the code snippet. First decide whether it was written by a human or
generated by a machine. Then give a fine-grained origin label, and if you
believe it is machine-generated, optionally note the likely generator
family.
html_layout: |
<div style="padding: 15px; max-width: 860px; margin: auto;">
<div style="background: #f1f5f9; border: 1px solid #cbd5e1; border-radius: 8px; padding: 10px 14px; margin-bottom: 12px;">
<strong style="color: #334155;">Language:</strong>
<span style="margin-left: 8px;">{{language}}</span>
</div>
<pre style="background: #0f172a; color: #e2e8f0; border-radius: 8px; padding: 16px; overflow-x: auto; font-size: 14px; line-height: 1.5;">{{code}}</pre>
</div>
allow_all_users: true
instances_per_annotator: 40
annotation_per_instance: 3
allow_skip: true
Sample Datasample-data.json
[
{
"id": "mgc_001",
"language": "Python",
"code": "def two_sum(nums, target):\n seen = {}\n for i, n in enumerate(nums):\n if target - n in seen:\n return [seen[target - n], i]\n seen[n] = i\n return []"
},
{
"id": "mgc_002",
"language": "Python",
"code": "# quick hack, fix later\ndef f(a):\n x=0\n for i in a:\n x+=i\n return x/len(a) if a else 0"
}
]
// ... and 8 more itemsGet This Design
Clone or download from the repository
Quick start:
git clone https://github.com/davidjurgens/potato-showcase.git cd potato-showcase/semeval/2026/task13-machine-generated-code potato start config.yaml
Dataset & paper
Orel et al., SemEval 2026 (Task 13)
Citation (BibTeX)
@inproceedings{orel-etal-2026-semeval,
title = "{S}em{E}val-2026 Task 13: Detecting Machine-Generated Code with Multiple Programming Languages, Generators, and Application Scenarios",
author = "Orel, Daniil and Azizov, Dilshod and Paul, Indraneil and Wang, Yuxia and Gurevych, Iryna and Nakov, Preslav",
booktitle = "Proceedings of the 20th International Workshop on Semantic Evaluation (SemEval-2026)",
year = "2026",
publisher = "Association for Computational Linguistics",
url = "https://aclanthology.org/2026.semeval-1.445",
pages = "3640--3658"
}Details
Annotation Types
Domain
Use Cases
Tags
Found an issue or want to improve this design?
Open an IssueRelated Designs
BRAINTEASER - Commonsense-Defying QA
Lateral thinking and commonsense-defying question answering task requiring annotators to select answers to brain teasers that defy default commonsense assumptions and provide explanations. Based on SemEval-2024 Task 9 (BRAINTEASER).
CLARITY: Unmasking Political Question Evasions
SemEval-2026 Task 6 (CLARITY): detecting how directly a politician answers a question. Given a question-answer pair from a U.S. presidential interview, annotators classify the reply into one of three clarity levels (Clear Reply, Ambiguous, Clear Non-Reply) and, when the answer is evasive, identify the evasion technique used. Multi-part questions are first decomposed into single sub-questions so each judgment captures how well one specific inquiry is addressed.
Clickbait Spoiling
Classification and extraction of spoilers for clickbait posts, including spoiler type identification and span-level spoiler detection. Based on SemEval-2023 Task 5 (Fröbe et al.).