ECB+ Cross-Document Event Coreference
Cross-document event coreference annotation based on the ECB+ corpus (Cybulska and Vossen, LREC 2014). Annotators mark event mentions — actions with their times, locations, and human and non-human participants — across multiple news documents describing similar but distinct event instances, and build shared event records that link coreferent event descriptions across documents by citing evidence spans. Complements the within-document MAVEN-ERE task (text/coreference/maven-ere-event-coreference): MAVEN-ERE links event mentions inside a single document, whereas ECB+ requires deciding whether descriptions in different documents refer to the same real-world event instance or merely to the same event type.
About this dataset
ECB+ extends the EventCorefBank (ECB) of Bejan and Harabagiu (2010) to make cross-document event coreference genuinely hard. The authors observed that each ECB topic covered a single seminal event, so systems could solve coreference by little more than topic classification and lemma matching. They added 502 new texts describing different instances of the same event types — for example, a second celebrity's rehab check-in alongside the original one — so that annotators and systems must distinguish same-type-but-different-instance events.
The annotation is event-centric: events are modeled as a combination of four components — an action (what happens or holds true), a time slot anchoring the action, a location, and participants, divided into human and non-human. Component mentions are marked from the point of view of the event action: only participants, times, and locations of the action count, not every entity or time expression in the sentence. Two action mentions corefer if they refer to the same instance of an action happening at the same time, in the same place, with the same participants.
The corpus was built in three rounds by two trained student annotators: component mentions and intra-document coreference in the new ECB+ texts, re-annotation of the original ECB, and finally cross-document coreference links per topic (using the CROMER tool, in which annotators create named event instances and attach coreferent mentions). Inter-annotator agreement reached Cohen's kappa of 0.74 on the full 30-tag set and 0.76 on cross-document coreference assignment.
The Potato config reproduces the cross-document stage with the multi_document_event scheme: annotators create one shared event record per real-world event instance, attach the member documents, and fill the action, time, location, and participant slots by citing evidence spans from the texts. The sample data mirrors the ECB+ design, pairing documents about the same instance with documents about a distinct instance of the same event type. ECB+'s token-level 30-tag inventory and separate intra-document layer are deliberately not reproduced.
- Released
- LREC 2014
- Topics
- 43
- Texts
- 982 (502 newly added ECB+ texts)
- Action mentions
- 14,884
- Time / location mentions
- 2,392 / 2,255
- Participant mentions
- 9,577 human; 2,963 non-human
- Coreference chains
- 7,671 intra-document; 2,204 cross-document
- Agreement
- Cohen's kappa 0.74 (mentions), 0.76 (cross-document coreference)
Configuration Fileconfig.yaml
This Potato config reproduces the annotation task. Save it as config.yaml and run potato start config.yaml to try it.
# ECB+: Cross-Document Event Coreference Annotation
# Based on: Agata Cybulska and Piek Vossen (2014), "Using a Sledgehammer to
# Crack a Nut? Lexical Diversity and Event Coreference Resolution."
# Proceedings of the Ninth International Conference on Language Resources
# and Evaluation (LREC 2014), pp. 4545-4552.
# Paper: https://aclanthology.org/L14-1646/
# Dataset: https://github.com/cltl/ecbPlus
#
# Task: annotate event mentions — actions together with their times, locations,
# and (human and non-human) participants — across multiple news documents that
# describe similar but distinct event instances (e.g. two different celebrity
# rehab check-ins), and establish which event descriptions corefer across
# documents. Following the ECB+ event-centric design, annotators build one
# shared event record per real-world event instance, attach the documents that
# describe it, and fill the component slots by citing evidence spans from the
# member documents; documents describing a different instance of the same event
# type belong to a separate event record. Deliberate simplification: ECB+'s
# token-level tagset (30 tags, including seven action classes and entity
# subtypes) and its separate intra-document coreference layer are not
# reproduced; cross-document event identity is captured through the shared
# event records and their cited evidence.
#
# Annotation instructions reproduced verbatim (event-component definitions,
# event-centric marking rules, and coreference criteria) from Sections 1, 4.2,
# and 4.3 of the paper. The complete ECB+ guidelines are published separately
# as Cybulska & Vossen (2014), "Guidelines for ECB+ annotation of events and
# their coreference," Technical Report NWR-2014-1, VU University Amsterdam.
annotation_task_name: "ECB+ Cross-Document Event Coreference"
task_dir: "."
data_files:
- sample-data.json
item_properties:
id_key: "id"
text_key: "text"
output_annotation_dir: "annotation_output/"
output_annotation_format: "json"
port: 8000
server_name: localhost
annotation_instructions: |
You are annotating events across a collection of news documents, following
the ECB+ scheme. Events are modeled as a combination of four components:
1. "an event action component describing what happens or holds true"
2. "an event time slot anchoring an action in time describing when something
happens or holds true"
3. "an event location component specifying where something happens or holds
true"
4. "a participant component that gives the answer to the question: who or
what is involved with, undergoes change as a result of or facilitates an
event or a state; we divide event participants into human participants
and non-human participants."
Mentions of event components are annotated in text from the point of view of
an event action, marking:
1. "participants involved with an action as opposed to any participant
mention occurring in a sentence"
2. "time when an action happened as opposed to any time expression mentioned
in text"
3. "location in which the action was performed in contrast to a locational
expression that does not refer to the place where an action happened."
Coreference criteria: "Two or more time expressions, location or participant
mentions corefer with each other if they refer respectively to the same
time, place or participants. Two action mentions corefer if they refer to
the same instance of an action that happens or holds true: (1) in the same
time, (2) in the same place and (3) with the same participants involved."
Workflow: create one event record per real-world event instance. Attach
every document that describes that instance, and fill the action, time,
location, and participant slots by highlighting evidence spans in the member
documents. Different documents in this collection may describe similar but
distinct instances of the same event type (for example, two different rehab
check-ins or two different earthquakes) — these belong to separate event
records, not the same one.
event_template:
enabled: true
name: ecb_plus_event
allow_annotator_create: true
slots:
- name: action
description: "The event action: what happens or holds true"
type: text
- name: time
description: "When the action happened (the time anchoring the action, not just any time expression)"
type: text
- name: location
description: "Where the action was performed (not just any locational expression)"
type: text
- name: human_participants
description: "Human participants involved with, changed by, or facilitating the action"
type: text
- name: non_human_participants
description: "Non-human participants involved with, changed by, or facilitating the action"
type: text
annotation_schemes:
- annotation_type: multi_document_event
name: events
description: "Group documents into events and fill each component slot with evidence cited from the texts"
slots:
- name: action
description: "The event action: what happens or holds true"
type: text
- name: time
description: "When the action happened (the time anchoring the action, not just any time expression)"
type: text
- name: location
description: "Where the action was performed (not just any locational expression)"
type: text
- name: human_participants
description: "Human participants involved with, changed by, or facilitating the action"
type: text
- name: non_human_participants
description: "Non-human participants involved with, changed by, or facilitating the action"
type: text
allow_annotator_create: true
instance_display:
fields:
- key: text
type: text
span_target: true
html_layout: |
<div style="padding: 15px; max-width: 860px; margin: auto;">
<div style="background: #f1f5f9; border: 1px solid #cbd5e1; border-radius: 8px; padding: 6px 12px; margin-bottom: 8px; font-size: 13px; color: #475569;">
<strong>{{title}}</strong> · {{source}} · {{date}}
</div>
<div style="font-size: 16px; line-height: 1.8;">{{text}}</div>
</div>
allow_all_users: true
instances_per_annotator: 30
annotation_per_instance: 2
allow_skip: true
Sample Datasample-data.json
[
{
"id": "ecb_001",
"title": "Actress Dana Whitfield checks into Malibu rehab facility",
"source": "Daily Beacon",
"date": "2024-01-12",
"text": "Actress Dana Whitfield checked into a rehabilitation facility in Malibu on Thursday, her publicist confirmed. The 34-year-old star of the series Harbor Lights entered the Oceanview Recovery Center after what her representative described as an exhausting year of filming. Whitfield is expected to remain at the facility for at least 30 days."
},
{
"id": "ecb_002",
"title": "Harbor Lights star enters treatment center",
"source": "Celebrity Wire",
"date": "2024-01-13",
"text": "Dana Whitfield, best known for her role on Harbor Lights, has entered a treatment center on the California coast, sources close to the actress said Friday. Whitfield was admitted to Oceanview Recovery Center in Malibu on Thursday evening. Her team asked for privacy while she focuses on her health."
}
]
// ... and 8 more itemsGet This Design
Clone or download from the repository
Quick start:
git clone https://github.com/davidjurgens/potato-showcase.git cd potato-showcase/text/coreference/ecb-plus-cross-document-events potato start config.yaml
Dataset & paper
Agata Cybulska and Piek Vossen. 2014. Using a sledgehammer to crack a nut? Lexical diversity and event coreference resolution. In Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC'14), pages 4545-4552.
Citation (BibTeX)
@inproceedings{cybulska-vossen-2014-using,
title = "Using a sledgehammer to crack a nut? Lexical diversity and event coreference resolution",
author = "Cybulska, Agata and
Vossen, Piek",
editor = "Calzolari, Nicoletta and
Choukri, Khalid and
Declerck, Thierry and
Loftsson, Hrafn and
Maegaard, Bente and
Mariani, Joseph and
Moreno, Asuncion and
Odijk, Jan and
Piperidis, Stelios",
booktitle = "Proceedings of the Ninth International Conference on Language Resources and Evaluation ({LREC}'14)",
month = may,
year = "2014",
address = "Reykjavik, Iceland",
publisher = "European Language Resources Association (ELRA)",
url = "https://aclanthology.org/L14-1646/",
pages = "4545--4552"
}Details
Annotation Types
Domain
Use Cases
Tags
Found an issue or want to improve this design?
Open an IssueRelated Designs
DWIE Document-Level Entity Annotation and Linking
Document-level entity mention annotation with knowledge-base linking based on DWIE (Zaporojets, Deleu, Develder, and Demeester, Information Processing & Management 2021). Annotators mark every entity mention in a full news article, assign an entity type, and link mentions to Wikipedia/Wikidata entries, leaving NIL entities unlinked. Complements the mention-level AIDA CoNLL-YAGO task (text/entity-linking/aida-conll-entity-disambiguation): AIDA presents one pre-marked mention at a time for disambiguation, whereas DWIE is entity-centric and document-level — annotators find all mentions themselves and keep links consistent across each entity's coreferent mentions.
Multilingual Coreference Resolution (CorefUD)
Multilingual coreference resolution across 17 languages using Universal Dependencies-style annotations. Annotators identify entity mentions (names, nominals, pronouns) and link them into coreference chains. Based on the CorefUD dataset and the CRAC 2023 Shared Task on Multilingual Coreference Resolution.
RCV1: Hierarchical Topic Coding
Hierarchical multi-label Topic coding of newswire stories, based on Reuters Corpus Volume 1 (Lewis, Yang, Rose, and Li, JMLR 2004). Annotators read an English news story and assign all applicable Topic codes from a tree-structured taxonomy spanning CCAT (Corporate/Industrial), ECAT (Economics), GCAT (Government/Social), and MCAT (Markets), following the documented Reuters coding policies: at least one Topic code per story and the most specific appropriate codes. Note that the original RCV1 coding was production editorial work by Reuters editors, partly machine-assisted, under the Reuters Business Briefing coding policies; this task recreates the manual Topic-coding judgment in isolation.