Capturing Discriminative Attributes

Binary classification of whether a semantic attribute discriminates between two words, testing understanding of fine-grained word meaning differences. Based on SemEval-2018 Task 10.

Configuration Fileconfig.yaml

This Potato config reproduces the annotation task. Save it as config.yaml and run potato start config.yaml to try it.

yaml

# Capturing Discriminative Attributes
# Based on Krebs et al., SemEval 2018
# Paper: https://aclanthology.org/S18-1117/
# Dataset: https://competitions.codalab.org/competitions/17326
#
# This task asks annotators to determine whether a given attribute
# (property or feature) discriminates between two words -- that is,
# whether it is a characteristic of one word but not the other.
#
# Classification Labels:
# - Discriminative: The attribute distinguishes between the two words
# - Not Discriminative: The attribute applies equally to both words or neither

annotation_task_name: "Capturing Discriminative Attributes"
task_dir: "."

data_files:
  - sample-data.json

item_properties:
  id_key: "id"
  text_key: "text"

output_annotation_dir: "annotation_output/"
output_annotation_format: "json"

port: 8000
server_name: localhost

annotation_schemes:
  - annotation_type: radio
    name: discriminative_judgment
    description: "Does this attribute discriminate between the two words?"
    labels:
      - "Discriminative"
      - "Not Discriminative"
    keyboard_shortcuts:
      "Discriminative": "1"
      "Not Discriminative": "2"
    tooltips:
      "Discriminative": "The attribute is a characteristic of one word but not the other"
      "Not Discriminative": "The attribute applies to both words equally, or to neither"

annotation_instructions: |
  You will be shown two words and an attribute. Your task is to determine whether
  the attribute discriminates between the two words. An attribute is discriminative
  if it is characteristic of one word but not the other.
  Example: "has wings" is discriminative for (airplane, car) but "has wheels" is not
  discriminative for (car, truck).

html_layout: |
  <div style="padding: 15px; max-width: 800px; margin: auto;">
    <div style="display: flex; justify-content: center; align-items: center; gap: 30px; margin-bottom: 16px;">
      <div style="background: #f0f9ff; border: 1px solid #bae6fd; border-radius: 8px; padding: 16px; text-align: center; min-width: 120px;">
        <span style="color: #64748b; font-size: 13px;">Word 1</span>
        <p style="font-size: 20px; font-weight: bold; margin: 4px 0;">{{word_1}}</p>
      </div>
      <span style="font-size: 24px; color: #94a3b8;">vs</span>
      <div style="background: #f0f9ff; border: 1px solid #bae6fd; border-radius: 8px; padding: 16px; text-align: center; min-width: 120px;">
        <span style="color: #64748b; font-size: 13px;">Word 2</span>
        <p style="font-size: 20px; font-weight: bold; margin: 4px 0;">{{word_2}}</p>
      </div>
    </div>
    <div style="background: #fefce8; border: 1px solid #fde68a; border-radius: 8px; padding: 16px; text-align: center; margin-bottom: 16px;">
      <strong style="color: #a16207;">Attribute:</strong>
      <p style="font-size: 18px; font-weight: bold; margin: 4px 0;">{{text}}</p>
    </div>
  </div>

allow_all_users: true
instances_per_annotator: 50
annotation_per_instance: 2
allow_skip: true
skip_reason_required: false

Sample Datasample-data.json

json

[
  {
    "id": "discattr_001",
    "text": "has wings",
    "word_1": "airplane",
    "word_2": "car"
  },
  {
    "id": "discattr_002",
    "text": "is round",
    "word_1": "ball",
    "word_2": "globe"
  }
]

// ... and 8 more items

Get This Design

View on GitHub

Clone or download from the repository

Quick start:

git clone https://github.com/davidjurgens/potato-showcase.git
cd potato-showcase/semeval/2018/task10-discriminative-attributes
potato start config.yaml

Dataset & paper

Krebs et al., SemEval 2018

Official dataset ↗Read the paper ↗

Citation (BibTeX)

bibtex

@inproceedings{krebs-etal-2018-semeval,
    title = "{S}em{E}val-2018 Task 10: Capturing Discriminative Attributes",
    author = "Krebs, Alicia and Lenci, Alessandro and Paperno, Denis",
    booktitle = "Proceedings of the 12th International Workshop on Semantic Evaluation",
    month = jun,
    year = "2018",
    address = "New Orleans, Louisiana",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/S18-1117",
    pages = "732--740"
}

Details

Annotation Types

radio

Domain

SemEvalNLPLexical SemanticsWord Attributes

Use Cases

Attribute ClassificationLexical SemanticsWord Discrimination

Related Designs

CODWOE - Comparing Dictionaries and Word Embeddings

Definition modeling for words in context, adapting SemEval-2022 Task 1 (CODWOE), which compared dictionary definitions with distributional word representations across two subtasks: definition modeling (embedding to definition) and reverse dictionary (definition to embedding). Based on Mickus et al.

textradio

Hypernym Discovery

Discovery of hypernyms (broader terms) for a given target term in context, with classification of the hypernym relationship type. Based on SemEval-2018 Task 9.