Skip to content
Showcase/ECB+ Cross-Document Event Coreference
advancedtext

ECB+ Cross-Document Event Coreference

Cross-document event coreference annotation based on the ECB+ corpus (Cybulska and Vossen, LREC 2014). Annotators mark event mentions — actions with their times, locations, and human and non-human participants — across multiple news documents describing similar but distinct event instances, and build shared event records that link coreferent event descriptions across documents by citing evidence spans. Complements the within-document MAVEN-ERE task (text/coreference/maven-ere-event-coreference): MAVEN-ERE links event mentions inside a single document, whereas ECB+ requires deciding whether descriptions in different documents refer to the same real-world event instance or merely to the same event type.

About this dataset

ECB+ extends the EventCorefBank (ECB) of Bejan and Harabagiu (2010) to make cross-document event coreference genuinely hard. The authors observed that each ECB topic covered a single seminal event, so systems could solve coreference by little more than topic classification and lemma matching. They added 502 new texts describing different instances of the same event types — for example, a second celebrity's rehab check-in alongside the original one — so that annotators and systems must distinguish same-type-but-different-instance events.

The annotation is event-centric: events are modeled as a combination of four components — an action (what happens or holds true), a time slot anchoring the action, a location, and participants, divided into human and non-human. Component mentions are marked from the point of view of the event action: only participants, times, and locations of the action count, not every entity or time expression in the sentence. Two action mentions corefer if they refer to the same instance of an action happening at the same time, in the same place, with the same participants.

The corpus was built in three rounds by two trained student annotators: component mentions and intra-document coreference in the new ECB+ texts, re-annotation of the original ECB, and finally cross-document coreference links per topic (using the CROMER tool, in which annotators create named event instances and attach coreferent mentions). Inter-annotator agreement reached Cohen's kappa of 0.74 on the full 30-tag set and 0.76 on cross-document coreference assignment.

The Potato config reproduces the cross-document stage with the multi_document_event scheme: annotators create one shared event record per real-world event instance, attach the member documents, and fill the action, time, location, and participant slots by citing evidence spans from the texts. The sample data mirrors the ECB+ design, pairing documents about the same instance with documents about a distinct instance of the same event type. ECB+'s token-level 30-tag inventory and separate intra-document layer are deliberately not reproduced.

Released
LREC 2014
Topics
43
Texts
982 (502 newly added ECB+ texts)
Action mentions
14,884
Time / location mentions
2,392 / 2,255
Participant mentions
9,577 human; 2,963 non-human
Coreference chains
7,671 intra-document; 2,204 cross-document
Agreement
Cohen's kappa 0.74 (mentions), 0.76 (cross-document coreference)
Submit

Configuration Fileconfig.yaml

This Potato config reproduces the annotation task. Save it as config.yaml and run potato start config.yaml to try it.

yaml
# ECB+: Cross-Document Event Coreference Annotation
# Based on: Agata Cybulska and Piek Vossen (2014), "Using a Sledgehammer to
#   Crack a Nut? Lexical Diversity and Event Coreference Resolution."
#   Proceedings of the Ninth International Conference on Language Resources
#   and Evaluation (LREC 2014), pp. 4545-4552.
#   Paper: https://aclanthology.org/L14-1646/
#   Dataset: https://github.com/cltl/ecbPlus
#
# Task: annotate event mentions — actions together with their times, locations,
# and (human and non-human) participants — across multiple news documents that
# describe similar but distinct event instances (e.g. two different celebrity
# rehab check-ins), and establish which event descriptions corefer across
# documents. Following the ECB+ event-centric design, annotators build one
# shared event record per real-world event instance, attach the documents that
# describe it, and fill the component slots by citing evidence spans from the
# member documents; documents describing a different instance of the same event
# type belong to a separate event record. Deliberate simplification: ECB+'s
# token-level tagset (30 tags, including seven action classes and entity
# subtypes) and its separate intra-document coreference layer are not
# reproduced; cross-document event identity is captured through the shared
# event records and their cited evidence.
#
# Annotation instructions reproduced verbatim (event-component definitions,
# event-centric marking rules, and coreference criteria) from Sections 1, 4.2,
# and 4.3 of the paper. The complete ECB+ guidelines are published separately
# as Cybulska & Vossen (2014), "Guidelines for ECB+ annotation of events and
# their coreference," Technical Report NWR-2014-1, VU University Amsterdam.

annotation_task_name: "ECB+ Cross-Document Event Coreference"
task_dir: "."

data_files:
  - sample-data.json

item_properties:
  id_key: "id"
  text_key: "text"

output_annotation_dir: "annotation_output/"
output_annotation_format: "json"

port: 8000
server_name: localhost

annotation_instructions: |
  You are annotating events across a collection of news documents, following
  the ECB+ scheme. Events are modeled as a combination of four components:

  1. "an event action component describing what happens or holds true"
  2. "an event time slot anchoring an action in time describing when something
     happens or holds true"
  3. "an event location component specifying where something happens or holds
     true"
  4. "a participant component that gives the answer to the question: who or
     what is involved with, undergoes change as a result of or facilitates an
     event or a state; we divide event participants into human participants
     and non-human participants."

  Mentions of event components are annotated in text from the point of view of
  an event action, marking:

  1. "participants involved with an action as opposed to any participant
     mention occurring in a sentence"
  2. "time when an action happened as opposed to any time expression mentioned
     in text"
  3. "location in which the action was performed in contrast to a locational
     expression that does not refer to the place where an action happened."

  Coreference criteria: "Two or more time expressions, location or participant
  mentions corefer with each other if they refer respectively to the same
  time, place or participants. Two action mentions corefer if they refer to
  the same instance of an action that happens or holds true: (1) in the same
  time, (2) in the same place and (3) with the same participants involved."

  Workflow: create one event record per real-world event instance. Attach
  every document that describes that instance, and fill the action, time,
  location, and participant slots by highlighting evidence spans in the member
  documents. Different documents in this collection may describe similar but
  distinct instances of the same event type (for example, two different rehab
  check-ins or two different earthquakes) — these belong to separate event
  records, not the same one.

event_template:
  enabled: true
  name: ecb_plus_event
  allow_annotator_create: true
  slots:
    - name: action
      description: "The event action: what happens or holds true"
      type: text
    - name: time
      description: "When the action happened (the time anchoring the action, not just any time expression)"
      type: text
    - name: location
      description: "Where the action was performed (not just any locational expression)"
      type: text
    - name: human_participants
      description: "Human participants involved with, changed by, or facilitating the action"
      type: text
    - name: non_human_participants
      description: "Non-human participants involved with, changed by, or facilitating the action"
      type: text

annotation_schemes:
  - annotation_type: multi_document_event
    name: events
    description: "Group documents into events and fill each component slot with evidence cited from the texts"
    slots:
      - name: action
        description: "The event action: what happens or holds true"
        type: text
      - name: time
        description: "When the action happened (the time anchoring the action, not just any time expression)"
        type: text
      - name: location
        description: "Where the action was performed (not just any locational expression)"
        type: text
      - name: human_participants
        description: "Human participants involved with, changed by, or facilitating the action"
        type: text
      - name: non_human_participants
        description: "Non-human participants involved with, changed by, or facilitating the action"
        type: text
    allow_annotator_create: true

instance_display:
  fields:
    - key: text
      type: text
      span_target: true

html_layout: |
  <div style="padding: 15px; max-width: 860px; margin: auto;">
    <div style="background: #f1f5f9; border: 1px solid #cbd5e1; border-radius: 8px; padding: 6px 12px; margin-bottom: 8px; font-size: 13px; color: #475569;">
      <strong>{{title}}</strong> &nbsp;·&nbsp; {{source}} &nbsp;·&nbsp; {{date}}
    </div>
    <div style="font-size: 16px; line-height: 1.8;">{{text}}</div>
  </div>

allow_all_users: true
instances_per_annotator: 30
annotation_per_instance: 2
allow_skip: true

Sample Datasample-data.json

json
[
  {
    "id": "ecb_001",
    "title": "Actress Dana Whitfield checks into Malibu rehab facility",
    "source": "Daily Beacon",
    "date": "2024-01-12",
    "text": "Actress Dana Whitfield checked into a rehabilitation facility in Malibu on Thursday, her publicist confirmed. The 34-year-old star of the series Harbor Lights entered the Oceanview Recovery Center after what her representative described as an exhausting year of filming. Whitfield is expected to remain at the facility for at least 30 days."
  },
  {
    "id": "ecb_002",
    "title": "Harbor Lights star enters treatment center",
    "source": "Celebrity Wire",
    "date": "2024-01-13",
    "text": "Dana Whitfield, best known for her role on Harbor Lights, has entered a treatment center on the California coast, sources close to the actress said Friday. Whitfield was admitted to Oceanview Recovery Center in Malibu on Thursday evening. Her team asked for privacy while she focuses on her health."
  }
]

// ... and 8 more items

Get This Design

View on GitHub

Clone or download from the repository

Quick start:

git clone https://github.com/davidjurgens/potato-showcase.git
cd potato-showcase/text/coreference/ecb-plus-cross-document-events
potato start config.yaml

Dataset & paper

Agata Cybulska and Piek Vossen. 2014. Using a sledgehammer to crack a nut? Lexical diversity and event coreference resolution. In Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC'14), pages 4545-4552.

Citation (BibTeX)

bibtex
@inproceedings{cybulska-vossen-2014-using,
    title = "Using a sledgehammer to crack a nut? Lexical diversity and event coreference resolution",
    author = "Cybulska, Agata  and
      Vossen, Piek",
    editor = "Calzolari, Nicoletta  and
      Choukri, Khalid  and
      Declerck, Thierry  and
      Loftsson, Hrafn  and
      Maegaard, Bente  and
      Mariani, Joseph  and
      Moreno, Asuncion  and
      Odijk, Jan  and
      Piperidis, Stelios",
    booktitle = "Proceedings of the Ninth International Conference on Language Resources and Evaluation ({LREC}'14)",
    month = may,
    year = "2014",
    address = "Reykjavik, Iceland",
    publisher = "European Language Resources Association (ELRA)",
    url = "https://aclanthology.org/L14-1646/",
    pages = "4545--4552"
}

Details

Annotation Types

multi_document_event

Domain

NLPCoreferenceNews

Use Cases

Event Coreference ResolutionCross-Document Information ExtractionEvent Extraction

Tags

event-coreferencecross-documentevent-extractionECB-plusnewscoreference

Found an issue or want to improve this design?

Open an Issue

Related Designs

DWIE Document-Level Entity Annotation and Linking

Document-level entity mention annotation with knowledge-base linking based on DWIE (Zaporojets, Deleu, Develder, and Demeester, Information Processing & Management 2021). Annotators mark every entity mention in a full news article, assign an entity type, and link mentions to Wikipedia/Wikidata entries, leaving NIL entities unlinked. Complements the mention-level AIDA CoNLL-YAGO task (text/entity-linking/aida-conll-entity-disambiguation): AIDA presents one pre-marked mention at a time for disambiguation, whereas DWIE is entity-centric and document-level — annotators find all mentions themselves and keep links consistent across each entity's coreferent mentions.

span

Multilingual Coreference Resolution (CorefUD)

Multilingual coreference resolution across 17 languages using Universal Dependencies-style annotations. Annotators identify entity mentions (names, nominals, pronouns) and link them into coreference chains. Based on the CorefUD dataset and the CRAC 2023 Shared Task on Multilingual Coreference Resolution.

spanspan_link

RCV1: Hierarchical Topic Coding

Hierarchical multi-label Topic coding of newswire stories, based on Reuters Corpus Volume 1 (Lewis, Yang, Rose, and Li, JMLR 2004). Annotators read an English news story and assign all applicable Topic codes from a tree-structured taxonomy spanning CCAT (Corporate/Industrial), ECAT (Economics), GCAT (Government/Social), and MCAT (Markets), following the documented Reuters coding policies: at least one Topic code per story and the most specific appropriate codes. Note that the original RCV1 coding was production editorial work by Reuters editors, partly machine-assisted, under the Reuters Business Briefing coding policies; this task recreates the manual Topic-coding judgment in isolation.

hierarchical_multiselect