Skip to content
advancedtext

Detecting Machine-Generated Code

SemEval-2026 Task 13: detecting machine-generated code across multiple programming languages, generators, and application scenarios. Annotators judge whether a code snippet is human-written or machine-generated (Subtask A), attribute the generator family (Subtask B), and give a fine-grained origin label (Subtask C: fully human, fully AI, human-AI hybrid, or AI tuned to look human-like). Training code is in C++/Python/Java, with unseen languages and domains at test time.

Try it live — no install

Boot the real Potato server in your browser (WebAssembly) and annotate with this exact config. Nothing leaves your machine.

Run live in your browser

Get This Design

View on GitHub(opens in a new tab)

Clone or download from the repository

Quick start:

bash
git clone https://github.com/davidjurgens/potato-showcase.git
cd potato-showcase/semeval/2026/task13-machine-generated-code
potato start config.yaml

Configuration File

This Potato config reproduces the annotation task. Save it as config.yaml and run potato start config.yaml to try it.

yaml
# Detecting Machine-Generated Code
# Based on Orel et al., SemEval-2026 Task 13
# Paper: https://aclanthology.org/2026.semeval-1.445/
# Task repo: https://github.com/mbzuai-nlp/SemEval-2026-Task13
#
# Given a code snippet, decide whether it was written by a human or
# generated by a model, across many languages and usage scenarios. The
# shared task has three subtasks:
#   - Subtask A: binary human vs. machine-generated
#   - Subtask B: generator-family attribution (which model family)
#   - Subtask C: fine-grained origin (fully human, fully AI, human-AI
#                hybrid, or a model tuned/prompted to look human-like)
# Training code is C++/Python/Java; test adds unseen languages
# (Go, PHP, C#, C, JavaScript) and unseen domains (research, production).

annotation_task_name: "Machine-Generated Code Detection"
task_dir: "."

data_files:
  - sample-data.json

item_properties:
  id_key: "id"
  text_key: "code"

output_annotation_dir: "annotation_output/"
export_annotation_format: "jsonl"

port: 8000

annotation_schemes:
  - annotation_type: radio
    name: origin
    description: "Subtask A: Was this code written by a human or generated by a machine?"
    labels:
      - name: "Human-written"
        key_value: "1"
      - name: "Machine-generated"
        key_value: "2"

  - annotation_type: radio
    name: fine_grained_origin
    description: "Subtask C: fine-grained origin"
    labels:
      - "Fully human-written"
      - "Fully AI-generated"
      - "Human-AI hybrid"
      - "AI tuned/prompted to look human-like"

  - annotation_type: text
    name: generator_family
    description: "Subtask B: if machine-generated, which model family do you think produced it? (optional)"
    textarea: false
    required: false
    placeholder: "e.g. GPT, Llama, Qwen, DeepSeek..."

annotation_instructions: |
  Read the code snippet. First decide whether it was written by a human or
  generated by a machine. Then give a fine-grained origin label, and if you
  believe it is machine-generated, optionally note the likely generator
  family.

user_config:
  allow_all_users: true
max_annotations_per_user: 40
num_annotators_per_item: 3

Sample Data

json
[
  {
    "id": "mgc_001",
    "language": "Python",
    "code": "def two_sum(nums, target):\n    seen = {}\n    for i, n in enumerate(nums):\n        if target - n in seen:\n            return [seen[target - n], i]\n        seen[n] = i\n    return []"
  },
  {
    "id": "mgc_002",
    "language": "Python",
    "code": "# quick hack, fix later\ndef f(a):\n    x=0\n    for i in a:\n        x+=i\n    return x/len(a) if a else 0"
  }
]

// ... and 8 more items

Dataset & paper

Orel et al., SemEval 2026 (Task 13)

Official dataset(opens in a new tab)Read the paper(opens in a new tab)

Citation (BibTeX)

bibtex
@inproceedings{orel-etal-2026-semeval,
    title = "{S}em{E}val-2026 Task 13: Detecting Machine-Generated Code with Multiple Programming Languages, Generators, and Application Scenarios",
    author = "Orel, Daniil  and Azizov, Dilshod  and Paul, Indraneil  and Wang, Yuxia  and Gurevych, Iryna  and Nakov, Preslav",
    booktitle = "Proceedings of the 20th International Workshop on Semantic Evaluation (SemEval-2026)",
    year = "2026",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2026.semeval-1.445",
    pages = "3640--3658"
}

Details

Annotation Types

radiotext

Domain

NLPCodeSemEval

Use Cases

Machine-Generated Text DetectionCode ProvenanceAI Content Attribution

Tags

semevalsemeval-2026shared-taskmachine-generated-codeai-detectioncode

Found an issue or want to improve this design?

Open an Issue(opens in a new tab)

Related Designs

BRAINTEASER - Commonsense-Defying QA

Lateral thinking and commonsense-defying question answering task requiring annotators to select answers to brain teasers that defy default commonsense assumptions and provide explanations. Based on SemEval-2024 Task 9 (BRAINTEASER).

radiotext

CLARITY: Unmasking Political Question Evasions

SemEval-2026 Task 6 (CLARITY): detecting how directly a politician answers a question. Given a question-answer pair from a U.S. presidential interview, annotators classify the reply into one of three clarity levels (Clear Reply, Ambiguous, Clear Non-Reply) and, when the answer is evasive, identify the evasion technique used. Multi-part questions are first decomposed into single sub-questions so each judgment captures how well one specific inquiry is addressed.

radiotext

Clickbait Spoiling

Classification and extraction of spoilers for clickbait posts, including spoiler type identification and span-level spoiler detection. Based on SemEval-2023 Task 5 (Fröbe et al.).

textradio