Fine-Grained Sentiment Analysis on Financial Microblogs and News
Graded sentiment analysis of financial text with topic classification, rating market sentiment from very bearish to very bullish on a 7-point scale. Based on SemEval-2017 Task 5.
Configuration Fileconfig.yaml
This Potato config reproduces the annotation task. Save it as config.yaml and run potato start config.yaml to try it.
# Fine-Grained Sentiment Analysis on Financial Microblogs and News
# Based on Cortis et al., SemEval 2017
# Paper: https://aclanthology.org/S17-2089/
# Dataset: https://alt.qcri.org/semeval2017/task5/
#
# This task asks annotators to rate the sentiment of financial text on a
# 7-point scale from very bearish to very bullish, and to classify the
# topic type of the financial content.
#
# Sentiment Scale (Likert 1-7):
# 1 = Very Bearish (extremely negative market outlook)
# 4 = Neutral (no clear sentiment direction)
# 7 = Very Bullish (extremely positive market outlook)
#
# Topic Categories:
# - Stock-Related: About a specific stock or equity
# - Market-Related: About overall market conditions
# - Company-Related: About company operations, earnings, management
# - Economic: About macroeconomic factors (GDP, employment, etc.)
# - Other: Other financial topics
annotation_task_name: "Fine-Grained Financial Sentiment Analysis"
task_dir: "."
data_files:
- sample-data.json
item_properties:
id_key: "id"
text_key: "text"
output_annotation_dir: "annotation_output/"
output_annotation_format: "json"
port: 8000
server_name: localhost
annotation_schemes:
- annotation_type: likert
name: sentiment_rating
description: "Rate the financial sentiment from very bearish to very bullish."
min_label: "Very Bearish"
max_label: "Very Bullish"
size: 7
- annotation_type: radio
name: topic_type
description: "What type of financial topic is discussed?"
labels:
- "Stock-Related"
- "Market-Related"
- "Company-Related"
- "Economic"
- "Other"
keyboard_shortcuts:
"Stock-Related": "1"
"Market-Related": "2"
"Company-Related": "3"
"Economic": "4"
"Other": "5"
tooltips:
"Stock-Related": "Discussion about a specific stock, equity, or share price"
"Market-Related": "Discussion about overall market conditions or indices"
"Company-Related": "Discussion about company operations, earnings, or management"
"Economic": "Discussion about macroeconomic factors like GDP, employment, or interest rates"
"Other": "Other financial topics not covered above"
annotation_instructions: |
You will be shown a financial text (microblog post or news excerpt) along with
the associated cashtag. Your task is to:
1. Rate the sentiment on a 7-point scale from Very Bearish to Very Bullish.
2. Classify the financial topic type.
Consider the tone and implied market direction when rating sentiment.
html_layout: |
<div style="padding: 15px; max-width: 800px; margin: auto;">
<div style="background: #fefce8; border: 1px solid #fde68a; border-radius: 8px; padding: 12px; margin-bottom: 12px;">
<strong style="color: #a16207;">Cashtag:</strong>
<span style="font-size: 15px; font-weight: bold;">{{cashtag}}</span>
</div>
<div style="background: #f0f9ff; border: 1px solid #bae6fd; border-radius: 8px; padding: 16px; margin-bottom: 16px;">
<strong style="color: #0369a1;">Text:</strong>
<p style="font-size: 16px; line-height: 1.7; margin: 8px 0 0 0;">{{text}}</p>
</div>
</div>
allow_all_users: true
instances_per_annotator: 50
annotation_per_instance: 2
allow_skip: true
skip_reason_required: false
Sample Datasample-data.json
[
{
"id": "finsent_001",
"text": "$AAPL crushing earnings expectations yet again! Revenue up 12% YoY and guidance looks fantastic. This stock is going to the moon!",
"cashtag": "$AAPL"
},
{
"id": "finsent_002",
"text": "$TSLA shares plummeted 8% after production numbers came in well below estimates. The company faces significant headwinds in Q4.",
"cashtag": "$TSLA"
}
]
// ... and 8 more itemsGet This Design
Clone or download from the repository
Quick start:
git clone https://github.com/davidjurgens/potato-showcase.git cd potato-showcase/semeval/2017/task05-financial-sentiment potato start config.yaml
Dataset & paper
Cortis et al., SemEval 2017
Citation (BibTeX)
@inproceedings{cortis-etal-2017-semeval,
title = "{S}em{E}val-2017 Task 5: Fine-Grained Sentiment Analysis on Financial Microblogs and News",
author = "Cortis, Keith and Freitas, Andr{\'e} and Daudert, Tobias and Huerlimann, Manuela and Zarrouk, Manel and Handschuh, Siegfried and Davis, Brian",
booktitle = "Proceedings of the 11th International Workshop on Semantic Evaluation ({S}em{E}val-2017)",
year = "2017",
publisher = "Association for Computational Linguistics",
pages = "519--535",
doi = "10.18653/v1/S17-2089",
url = "https://aclanthology.org/S17-2089"
}Details
Annotation Types
Domain
Use Cases
Tags
Found an issue or want to improve this design?
Open an IssueRelated Designs
Assessing Humor in Edited News Headlines
Rate the funniness of edited news headlines on a 0-3 scale and judge which of two edited versions is funnier, based on SemEval-2020 Task 7 (Hossain et al.). Headlines are minimally edited by replacing a single word to create humorous effect.
MTRAGEval: Evaluating Multi-Turn RAG Conversations
SemEval-2026 Task 8 (MTRAGEval): evaluating retrieval-augmented generation (RAG) in multi-turn conversations, built on the MTRAG benchmark. Given a conversation, the retrieved reference passages, and a system-generated response, annotators judge how faithful the response is to the passages, how well it answers the user's latest question, and whether it correctly handles unanswerable, underspecified, non-standalone, or unclear questions. The shared task covers retrieval (A), generation (B), and full RAG (C).
PreTENS - Presuppositional Acceptability
Judge the acceptability of sentences whose two nominal arguments are presupposed to stand (or not) in an ordered taxonomic relation. Includes a binary acceptable/unacceptable judgment and a 1-7 acceptability rating. Based on SemEval-2022 Task 3 (Zamparelli et al.).