Skip to content
intermediatesurvey

Rating Plausibility of Word Senses through Narrative Understanding

SemEval-2026 Task 5: rating how plausible a particular sense of an ambiguous word is within a short narrative. Each item is a 4-5 sentence story containing a homonym in a target sentence, with sparse contextual cues. Annotators judge, on a 1-5 scale, how plausible a given candidate sense of the homonym is in that narrative context. Built on the Ambistory dataset.

Try it live — no install

Boot the real Potato server in your browser (WebAssembly) and annotate with this exact config. Nothing leaves your machine.

Run live in your browser

Get This Design

View on GitHub(opens in a new tab)

Clone or download from the repository

Quick start:

bash
git clone https://github.com/davidjurgens/potato-showcase.git
cd potato-showcase/semeval/2026/task05-word-sense-plausibility
potato start config.yaml

Configuration File

This Potato config reproduces the annotation task. Save it as config.yaml and run potato start config.yaml to try it.

yaml
# Rating Plausibility of Word Senses through Narrative Understanding
# Based on Gehring, Meyer, and Roth, SemEval-2026 Task 5
# Paper: https://aclanthology.org/2026.semeval-1.448/
# Task page: https://nlu-lab.github.io/semeval.html
#
# Each item is a short (4-5 sentence) story containing an ambiguous word
# (a homonym) in a target sentence. The story deliberately offers sparse
# cues. Given one candidate sense of the homonym, annotators rate how
# plausible that sense is in the narrative on a 1-5 Likert scale
# (1 = very implausible, 5 = very plausible). Built on the Ambistory dataset.

annotation_task_name: "Word-Sense Plausibility in Narratives"
task_dir: "."

data_files:
  - sample-data.json

item_properties:
  id_key: "id"
  text_key: "story"

output_annotation_dir: "annotation_output/"
export_annotation_format: "jsonl"

port: 8000

annotation_schemes:
  - annotation_type: likert
    name: plausibility
    description: "How plausible is the candidate sense of the target word in this story?"
    size: 5
    min_label: "Very implausible"
    max_label: "Very plausible"

annotation_instructions: |
  Read the short story and note the target (ambiguous) word and the
  candidate sense provided. Using only the narrative context, rate how
  plausible it is that the target word carries that candidate sense, on a
  scale from 1 (very implausible) to 5 (very plausible).

user_config:
  allow_all_users: true
max_annotations_per_user: 50
num_annotators_per_item: 5

Sample Data

json
[
  {
    "id": "wsp_001",
    "story": "Mara had been saving for months. She finally walked down to the bank and waited in line. When it was her turn, she slid the envelope across the counter. The teller smiled and counted the bills.",
    "target_word": "bank",
    "candidate_sense": "a financial institution"
  },
  {
    "id": "wsp_002",
    "story": "The afternoon was hot and still. They followed the trail until it opened onto the river. Tom sat down on the bank and dipped his feet in the cool water. Dragonflies skimmed the surface.",
    "target_word": "bank",
    "candidate_sense": "a financial institution"
  }
]

// ... and 8 more items

Dataset & paper

Gehring, Meyer & Roth, SemEval 2026 (Task 5)

Official dataset(opens in a new tab)Read the paper(opens in a new tab)

Citation (BibTeX)

bibtex
@inproceedings{gehring-etal-2026-semeval,
    title = "{S}em{E}val-2026 Task 5: Rating Plausibility of Word Senses in Ambiguous Stories through Narrative Understanding",
    author = "Gehring, Janosch  and Meyer, Selina  and Roth, Michael",
    booktitle = "Proceedings of the 20th International Workshop on Semantic Evaluation (SemEval-2026)",
    year = "2026",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2026.semeval-1.448",
    pages = "3691--3703"
}

Details

Annotation Types

likert

Domain

NLPWord SenseSemEval

Use Cases

Word Sense DisambiguationNarrative UnderstandingPlausibility Rating

Tags

semevalsemeval-2026shared-taskword-sensenarrativeplausibility

Found an issue or want to improve this design?

Open an Issue(opens in a new tab)

Related Designs

Assessing Humor in Edited News Headlines

Rate the funniness of edited news headlines on a 0-3 scale and judge which of two edited versions is funnier, based on SemEval-2020 Task 7 (Hossain et al.). Headlines are minimally edited by replacing a single word to create humorous effect.

likertradio

Fine-Grained Sentiment Analysis on Financial Microblogs and News

Graded sentiment analysis of financial text with topic classification, rating market sentiment from very bearish to very bullish on a 7-point scale. Based on SemEval-2017 Task 5.

likertradio

Graded Word Similarity in Context

Rate the graded semantic similarity of a PAIR of target words that appear together within a shared context, based on SemEval-2020 Task 3 (Armendariz et al.). The same word pair is judged within two different contexts; Subtask 1 predicts the change in similarity between contexts and Subtask 2 predicts the absolute rating. Ratings follow the SimLex-style 0-6 similarity scale (later mapped to 0-10). Similarity is distinct from relatedness (e.g., coffee and cup are related but not similar).

likertslider