Skip to content
advancedtext

PsyCoMark: Psycholinguistic Conspiracy Marker Extraction and Detection

SemEval-2026 Task 10 (PsyCoMark): studying how conspiracy theories are expressed in social-media (Reddit) statements. Annotators make a document-level conspiracy judgment (Yes / No / Can't tell) and extract five psycholinguistic conspiracy markers as spans: Actor, Action, Effect, Victim, and Evidence.

Try it live — no install

Boot the real Potato server in your browser (WebAssembly) and annotate with this exact config. Nothing leaves your machine.

Run live in your browser

Get This Design

View on GitHub(opens in a new tab)

Clone or download from the repository

Quick start:

bash
git clone https://github.com/davidjurgens/potato-showcase.git
cd potato-showcase/semeval/2026/task10-psycomark-conspiracy
potato start config.yaml

Configuration File

This Potato config reproduces the annotation task. Save it as config.yaml and run potato start config.yaml to try it.

yaml
# PsyCoMark: Psycholinguistic Conspiracy Marker Extraction and Detection
# Based on Samory, Soldner, and Batzdorfer, SemEval-2026 Task 10
# Paper: https://aclanthology.org/2026.semeval-1.444/
# Task page: https://hide-ous.github.io/semeval2026-psycomark
#
# The task pairs binary conspiracy detection with the extraction of five
# psycholinguistic markers that characterize conspiracy narratives. Data
# are English Reddit submission statements.
#
# Document-level conspiracy label: Yes / No / Can't tell.
# Five psycholinguistic markers (extracted as spans):
# - Actor: who is claimed to be secretly acting (the alleged conspirators)
# - Action: the secret/malicious action they are said to carry out
# - Effect: the claimed outcome or consequence of the action
# - Victim: who is claimed to be harmed
# - Evidence: text the author offers as proof of the claim

annotation_task_name: "PsyCoMark - Conspiracy Markers"
task_dir: "."

data_files:
  - sample-data.json

item_properties:
  id_key: "id"
  text_key: "text"

output_annotation_dir: "annotation_output/"
export_annotation_format: "jsonl"

port: 8000

annotation_schemes:
  - annotation_type: radio
    name: conspiracy
    description: "Does this statement express a conspiracy theory?"
    labels:
      - name: "Yes"
        key_value: "1"
      - name: "No"
        key_value: "2"
      - name: "Can't tell"
        key_value: "3"

  - annotation_type: span
    name: markers
    description: "Highlight the psycholinguistic conspiracy markers"
    labels:
      - "Actor"
      - "Action"
      - "Effect"
      - "Victim"
      - "Evidence"

annotation_instructions: |
  Read the statement. First decide whether it expresses a conspiracy theory
  (Yes / No / Can't tell). Then, if relevant, highlight the psycholinguistic
  markers: the Actor (alleged conspirators), the Action they secretly take,
  the Effect of that action, the Victim harmed, and any Evidence offered as
  proof.

user_config:
  allow_all_users: true
max_annotations_per_user: 40
num_annotators_per_item: 3

Sample Data

json
[
  {
    "id": "psy_001",
    "text": "The pharmaceutical companies are hiding the real cure so they can keep selling us expensive treatments forever."
  },
  {
    "id": "psy_002",
    "text": "I planted tomatoes last weekend and they're already starting to sprout in the garden."
  }
]

// ... and 8 more items

Dataset & paper

Samory et al., SemEval 2026 (Task 10)

Official dataset(opens in a new tab)Read the paper(opens in a new tab)

Citation (BibTeX)

bibtex
@inproceedings{samory-etal-2026-semeval,
    title = "{S}em{E}val-2026 Task 10: {P}sy{C}o{M}ark -- Psycholinguistic Conspiracy Marker Extraction and Detection",
    author = "Samory, Mattia  and Soldner, Felix  and Batzdorfer, Veronika",
    booktitle = "Proceedings of the 20th International Workshop on Semantic Evaluation (SemEval-2026)",
    year = "2026",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2026.semeval-1.444",
    pages = "3628--3639"
}

Details

Annotation Types

radiospan

Domain

NLPComputational Social ScienceSemEval

Use Cases

Conspiracy DetectionPsycholinguisticsSpan Extraction

Tags

semevalsemeval-2026shared-taskconspiracypsycholinguisticssocial-media

Found an issue or want to improve this design?

Open an Issue(opens in a new tab)

Related Designs

Aspect-Based Sentiment Analysis

Identification of aspect terms in review text with sentiment polarity classification for each aspect. Based on SemEval-2016 Task 5 (ABSA).

spanradio

Character Identification on Multiparty Dialogues

Identification and linking of character mentions in TV show dialogue, combining span annotation with entity resolution for the main cast of Friends. Based on SemEval-2018 Task 4.

spanradio

Clickbait Spoiling

Classification and extraction of spoilers for clickbait posts, including spoiler type identification and span-level spoiler detection. Based on SemEval-2023 Task 5 (Fröbe et al.).

textradio