Cross-Document Annotation for Coreference and Event Linking

Cross-document annotation links a mention, event, or claim in one document to the same entity, event, or evidence in other documents. In the method used to build the ECB+ corpus, annotators attach mentions to a shared record for each real-world event or entity instead of comparing every pair of documents. CROMER, CoRefi, and Potato's multi_document_event scheme keep shared records of that kind, and INCEpTION and brat connect documents through knowledge-base identifiers on each mention.

Two expressions are in a relation of coreference when both expressions refer to the same thing. Cross-document coreference extends the relation across texts. An early test set gathered 197 New York Times articles that mention 35 different people named John Smith (Bagga and Baldwin, 1998). Event coreference applies the same idea to actions. The EventCorefBank (ECB) grouped 482 news documents into 43 topics with one seminal event per topic, and annotators linked mentions of each event across the reports (Bejan and Harabagiu, 2010).

Claim-evidence linking pairs a claim with the sentences that support or refute the claim, as in automated fact-checking. In the FEVER dataset, 12.15% of claims needed evidence from more than one Wikipedia page (Thorne et al., 2018). Multi-document event extraction applies information extraction to many reports of the same event and fills one event template (who, what, where, and when).

The guide below covers links between documents. Coreference Resolution covers mention chains inside one text, Relation and Event Extraction covers relations and event arguments inside one text, and Entity Linking covers resolving a mention to a knowledge-base entry. Span Annotation covers marking the mentions themselves, and Text Annotation Tools Compared compares text tools on every task. Potato is free and open source (GPL-3.0-or-later) and runs on your own server.

Sources of difficulty in cross-document annotation

The number of document pairs grows with the square of the number of documents. A corpus of n documents has n(n − 1)/2 pairs. Twenty-five reports make 300 pairs, and the 982 texts of ECB+ make 481,671. ECB and ECB+ linked documents only within a topic, and the corpus builders assembled each topic in advance around one kind of event. With a shared record, an annotator checks each new report against the events recorded so far, and the length of that list depends on the number of real-world events in the corpus.

Annotators have to keep details from several documents in mind at once. ECB+ added 502 articles that report a second instance of each topic's event type (Cybulska and Vossen, 2014). For example, the first topic pairs reports of Tara Reid's 2008 check-in to rehab with reports of Lindsay Lohan's 2013 check-in. Under the ECB+ guidelines, two action mentions corefer only when the actions share a time, a place, and participants. An annotator therefore checks the time, the place, and the participants before linking two reports.

Agreement on clusters is harder to measure than agreement on labels. Two annotators can sort the same ten documents into clusters that differ by one document, and the score depends on whether a metric counts links, mentions, or whole clusters. The reference scorer from Pradhan et al. (2014) computes five coreference metrics (MUC, B3, CEAFe, CEAFm, and BLANC). B3 comes from the John Smith study.

The ECB+ annotation procedure

ECB+ added 502 news articles to the 43 ECB topics, for 982 texts in all (ECB+ repository). Annotators marked event actions, times, locations, and human and non-human participants with 30 tags. The annotators were two paid student assistants, both native speakers of English. The two assistants worked in three rounds:

1. In the CAT tool, the assistants marked event components in the 502 new articles and linked coreferent mentions inside each article.
2. The assistants revised the existing ECB 0.1 annotation of the original articles.
3. In CROMER, the assistants linked coreferent mentions across all the articles of each topic.

In CROMER, an annotator first creates a named instance such as "barack obama" and then attaches each coreferent mention to the instance. Everyone working on a task shares the instances. Both annotators labeled the 55 articles of the first four topics and reached Cohen's kappa of 0.74 on the 30 tags and 0.79 on the five main component tags. Most disagreements concerned mention extent because the guidelines specified the head word and the annotators sometimes marked the whole phrase. After retraining, the two annotators split the remaining 447 articles and annotated each article once.

For the cross-document layer, the second annotator reused the first annotator's instances without seeing the first annotator's links. With shared instances, the instance ID worked as a category label, and Cohen's kappa over 490 cross-document coreference IDs was 0.76. The ECB+ team made three choices that apply to any cross-document project. First, documents were grouped into topics before annotation. Second, annotators linked mentions to shared named records. Third, the team measured agreement on a double-annotated subset before splitting the rest of the work.

Tools for cross-document linking

Annotation tools handle cross-document links in two ways. First, some tools keep a shared list of records that a mention in any document can point to. Second, other tools annotate one document at a time and connect documents through knowledge-base identifiers. Each entry in the table below comes from the tool's own paper or documentation as of November 2, 2026:

Choose by the unit you link. Mention-level cross-document coreference needs a tool that links individual mentions across documents, such as CoRefi or CROMER. INCEpTION fits projects that resolve mentions to Wikidata or a domain ontology and want chain layers and recommenders inside each document. Potato links whole documents to shared events and records the text that fills each event slot. That design fits event-level coreference, multi-document event extraction, and claim-evidence linking.

Cross-document events in Potato

Potato models a cross-document event as a record in a shared registry, much as CROMER models an instance. Each article is one item. An annotator reading an article attaches the article to an existing event or creates a new event, then fills the event's slots by selecting text in the article. The config below sets up ECB+-style event coreference over news articles, with one slot per event component:

annotation_task_name: "Cross-document event coreference"
task_dir: "."
data_files:
  - "data/articles.jsonl"
item_properties:
  id_key: id
  text_key: text
output_annotation_dir: "annotation_output/"
export_annotation_format: jsonl
 
user_config:
  allow_all_users: true
 
instance_display:
  fields:
    - key: text
      type: text
      span_target: true
 
event_template:
  enabled: true
  name: news_event
  allow_annotator_create: true
  slots: &event_slots
    - {name: action, description: "What happened", type: text}
    - {name: time, description: "When the action happened", type: text}
    - {name: location, description: "Where the action happened", type: text}
    - {name: participants, description: "Who took part or was affected", type: text}
 
annotation_schemes:
  - annotation_type: multi_document_event
    name: events
    description: "Attach this article to the real-world event the article reports, or create a new event, and cite text for each slot."
    slots: *event_slots
    allow_annotator_create: true

The event_template block creates the registry. Potato stops at startup with a validation error when a config has a multi_document_event scheme and no event_template block with enabled: true. The YAML anchor &event_slots keeps the scheme's slots identical to the template's slots. span_target: true on the text field lets an annotator cite evidence by selecting text. Each line of articles.jsonl has an id and a text. Run potato start config.yaml to start the server.

Potato writes the registry to annotation_output/event_registry.json. Each event record contains a title, the slot values, the IDs of the member documents, and a list of evidence citations with the slot name, document ID, character offsets, and quoted text. Citing evidence from an article adds the article to the event. A seed_events file can preload events. For example, a seed file can hold one event per seminal event in each topic.

The registry is shared across annotators. A second annotator opening an article sees the events earlier annotators attached to the article. When two annotators edit the same event at once, Potato rejects the second save and reloads the current version of the event.

The corpus map shortens the list of documents an annotator compares. With corpus_map enabled, Potato embeds every document with a sentence-transformer model, clusters the embeddings, projects the clusters onto a 2D map at /corpus/map, and precomputes each document's nearest neighbors. An annotator then reviews one cluster or one document's neighbors instead of the whole corpus. Add this block to the config above:

corpus_map:
  enabled: true
  embedding_model: all-MiniLM-L6-v2
  clustering: {num_clusters: auto, items_per_cluster: 6}
  knn: {k: 5}

Install sentence-transformers separately for the corpus map. pip install potato-annotation[viz] adds umap-learn for the 2D layout. Without umap-learn, Potato falls back to a PCA projection. Clicking a document's point on the map assigns the document to the annotator on demand, outside the usual assignment queue.

The coreference and span_link schemes link spans within one item. For mention chains inside one article, add a span scheme and a coreference scheme as Coreference Chains describes. The ECB+ showcase design applies the event registry to sample articles.

Agreement and export in Potato

Potato 2.13.1 computes agreement on mention spans only. For a span scheme, Potato's agreement report contains token-level Cohen's kappa, span F1 at exact and partial match, Krippendorff's alpha over character positions, and a simplified gamma. Agreement for Spans explains each measure. Potato does not score coreference chains, span links, or multi_document_event memberships. Cluster agreement is computed outside Potato from the export.

With export_annotation_format: jsonl, Potato writes one record per annotator and article to annotation_output/exports/jsonl/, with the labels, spans, and links for that pair. The events label lists the events the article belonged to when the annotator last changed the article's memberships, and event_registry.json contains the events themselves. For an agreement study like the ECB+ study, run one Potato server per annotator from copies of the config with different output_annotation_dir values. Each server then keeps a separate registry, and you compare the two clusterings with a cluster metric such as B3 or CEAF. Adjudication and Disagreement covers resolving the differences afterward.

Frequently asked questions

Which tools support cross-document annotation and linking?

CROMER, CoRefi, and Potato keep a shared record for each event or entity that mentions in any document can point to, and INCEpTION and brat connect documents through knowledge-base identifiers on each mention. CROMER produced the cross-document layer of ECB+, and CoRefi is an MIT-licensed tool for crowdsourced cross-document coreference. Potato's multi_document_event scheme links whole documents to shared events and stores the cited text for each event slot.

How was the ECB+ corpus annotated?

Two paid student annotators marked event components with 30 tags and linked coreferent mentions inside each article in the CAT tool, then linked mentions across the articles of each topic to shared, named instances in CROMER. On a double-annotated subset of 55 articles, Cohen's kappa was 0.74 on the 30 tags and 0.76 on cross-document coreference. After retraining, the two annotators split the remaining 447 articles and annotated each article once.

Further reading

• Coreference Resolution
• Relation and Event Extraction
• Entity Linking
• Span Annotation
• Agreement for Spans
• Text Annotation Tools Compared
• Potato for text annotation
• Potato for digital humanities
• Multi-document event annotation source documentation

References

• Bagga, A., and Baldwin, B. (1998). Entity-Based Cross-Document Coreferencing Using the Vector Space Model. ACL-COLING 1998, 79–85. https://aclanthology.org/P98-1012/
• Bejan, C., and Harabagiu, S. (2010). Unsupervised Event Coreference Resolution with Rich Linguistic Features. ACL 2010, 1412–1422. https://aclanthology.org/P10-1143/
• Bornstein, A., Cattan, A., and Dagan, I. (2020). CoRefi: A Crowd Sourcing Suite for Coreference Annotation. EMNLP 2020 System Demonstrations, 205–215. https://aclanthology.org/2020.emnlp-demos.27/
• Cybulska, A., and Vossen, P. (2014). Using a Sledgehammer to Crack a Nut? Lexical Diversity and Event Coreference Resolution. LREC 2014, 4545–4552. https://aclanthology.org/L14-1646/
• Girardi, C., Speranza, M., Sprugnoli, R., and Tonelli, S. (2014). CROMER: a Tool for Cross-Document Event and Entity Coreference. LREC 2014, 3204–3208. https://aclanthology.org/L14-1568/
• Pradhan, S., Luo, X., Recasens, M., Hovy, E., Ng, V., and Strube, M. (2014). Scoring Coreference Partitions of Predicted Mentions: A Reference Implementation. ACL 2014, 30–35. https://aclanthology.org/P14-2006/
• Thorne, J., Vlachos, A., Christodoulopoulos, C., and Mittal, A. (2018). FEVER: a Large-scale Dataset for Fact Extraction and VERification. NAACL 2018, 809–819. https://aclanthology.org/N18-1074/
• Vogel, J. (2023). Code Book for the Annotation of Diverse Cross-Document Coreference of Entities in News Articles. arXiv:2310.12064. https://arxiv.org/abs/2310.12064