Counting Events and Participants in News
Counting the number of events or incidents described in news reports and providing event descriptions. Based on SemEval-2018 Task 5 (Counting Events and Participants).
Configuration Fileconfig.yaml
This Potato config reproduces the annotation task. Save it as config.yaml and run potato start config.yaml to try it.
# Counting Events and Participants in News
# Based on Postma et al., SemEval 2018
# Paper: https://aclanthology.org/S18-1009/
# Dataset: https://cltl.github.io/LongTailQATask/
#
# This task asks annotators to count the number of distinct events or
# incidents described in a news article and provide a brief description
# of each event.
#
# Annotators enter a number for the event count and provide a textual
# description summarizing the distinct events found.
annotation_task_name: "Counting Events and Participants in News"
task_dir: "."
data_files:
- sample-data.json
item_properties:
id_key: "id"
text_key: "text"
output_annotation_dir: "annotation_output/"
output_annotation_format: "json"
port: 8000
server_name: localhost
annotation_schemes:
- annotation_type: number
name: event_count
description: "How many distinct events or incidents are described in this text?"
- annotation_type: text
name: event_description
description: "Briefly describe each distinct event mentioned in the text."
annotation_instructions: |
You will be shown a news article along with its event type category.
Your task is to:
1. Count the number of distinct events or incidents described in the text.
2. Provide a brief description of each event identified.
Note: Related follow-up actions count as separate events if they are distinct incidents.
html_layout: |
<div style="padding: 15px; max-width: 800px; margin: auto;">
<div style="background: #fefce8; border: 1px solid #fde68a; border-radius: 8px; padding: 12px; margin-bottom: 12px;">
<strong style="color: #a16207;">Event Type:</strong>
<span style="font-size: 15px;">{{event_type}}</span>
</div>
<div style="background: #f0f9ff; border: 1px solid #bae6fd; border-radius: 8px; padding: 16px; margin-bottom: 16px;">
<strong style="color: #0369a1;">News Article:</strong>
<p style="font-size: 16px; line-height: 1.7; margin: 8px 0 0 0;">{{text}}</p>
</div>
</div>
allow_all_users: true
instances_per_annotator: 50
annotation_per_instance: 2
allow_skip: true
skip_reason_required: false
Sample Datasample-data.json
[
{
"id": "eventcount_001",
"text": "A magnitude 6.2 earthquake struck central Italy on Wednesday, causing widespread damage to buildings in Amatrice. Rescue teams recovered 12 survivors from the rubble. A 4.5 magnitude aftershock hit the same region the following day.",
"event_type": "Natural Disaster"
},
{
"id": "eventcount_002",
"text": "Police arrested three suspects in connection with the robbery of a downtown jewelry store. The heist, which occurred last Tuesday, resulted in the theft of over $500,000 in merchandise.",
"event_type": "Crime"
}
]
// ... and 8 more itemsGet This Design
Clone or download from the repository
Quick start:
git clone https://github.com/davidjurgens/potato-showcase.git cd potato-showcase/semeval/2018/task05-event-counting potato start config.yaml
Dataset & paper
Postma et al., SemEval 2018
Citation (BibTeX)
@inproceedings{postma-etal-2018-semeval,
title = "{S}em{E}val-2018 Task 5: Counting Events and Participants in the Long Tail",
author = "Postma, Marten and Ilievski, Filip and Vossen, Piek",
booktitle = "Proceedings of the 12th International Workshop on Semantic Evaluation",
year = "2018",
address = "New Orleans, Louisiana",
publisher = "Association for Computational Linguistics",
pages = "70--80",
doi = "10.18653/v1/S18-1009",
url = "https://aclanthology.org/S18-1009"
}Details
Annotation Types
Domain
Use Cases
Tags
Found an issue or want to improve this design?
Open an IssueRelated Designs
NumEval - Numeral-Aware Language Understanding and Generation
Numeral-aware language understanding and generation across the five NumEval datasets: Quantitative Prediction (QP), Quantitative Natural Language Inference (QNLI), Quantitative Question Answering (QQA), reading comprehension of numerals (NQuAD), and numeral-aware headline generation (Num-HG). Based on SemEval-2024 Task 7 (NumEval).
Parsing Time Normalizations
Parsing time expressions in text using the SCATE (Semantically Compositional Annotation of Time Expressions) schema, marking typed time entities and operators (e.g., Year, Month-Of-Year, Day-Of-Month, Last, Next, This, Between) that compose into interval semantics. Based on SemEval-2018 Task 6.
BIG-Bench Task Evaluation
Evaluate language model responses on diverse reasoning tasks from the BIG-Bench benchmark. Annotators assess correctness, provide reasoning explanations, and rate confidence for model outputs across multiple task categories.