Skip to content
intermediatesurvey

Fine-Grained Sentiment Analysis on Financial Microblogs and News

Graded sentiment analysis of financial text with topic classification, rating market sentiment from very bearish to very bullish on a 7-point scale. Based on SemEval-2017 Task 5.

Get This Design

View on GitHub(opens in a new tab)

Clone or download from the repository

Quick start:

bash
git clone https://github.com/davidjurgens/potato-showcase.git
cd potato-showcase/semeval/2017/task05-financial-sentiment
potato start config.yaml

Configuration File

This Potato config reproduces the annotation task. Save it as config.yaml and run potato start config.yaml to try it.

yaml
# Fine-Grained Sentiment Analysis on Financial Microblogs and News
# Based on Cortis et al., SemEval 2017
# Paper: https://aclanthology.org/S17-2089/
# Dataset: https://alt.qcri.org/semeval2017/task5/
#
# This task asks annotators to rate the sentiment of financial text on a
# 7-point scale from very bearish to very bullish, and to classify the
# topic type of the financial content.
#
# Sentiment Scale (Likert 1-7):
# 1 = Very Bearish (extremely negative market outlook)
# 4 = Neutral (no clear sentiment direction)
# 7 = Very Bullish (extremely positive market outlook)
#
# Topic Categories:
# - Stock-Related: About a specific stock or equity
# - Market-Related: About overall market conditions
# - Company-Related: About company operations, earnings, management
# - Economic: About macroeconomic factors (GDP, employment, etc.)
# - Other: Other financial topics

annotation_task_name: "Fine-Grained Financial Sentiment Analysis"
task_dir: "."

data_files:
  - sample-data.json

item_properties:
  id_key: "id"
  text_key: "text"

output_annotation_dir: "annotation_output/"
output_annotation_format: "json"

port: 8000
server_name: localhost

annotation_schemes:
  - annotation_type: likert
    name: sentiment_rating
    description: "Rate the financial sentiment from very bearish to very bullish."
    min_label: "Very Bearish"
    max_label: "Very Bullish"
    size: 7

  - annotation_type: radio
    name: topic_type
    description: "What type of financial topic is discussed?"
    labels:
      - "Stock-Related"
      - "Market-Related"
      - "Company-Related"
      - "Economic"
      - "Other"
    keyboard_shortcuts:
      "Stock-Related": "1"
      "Market-Related": "2"
      "Company-Related": "3"
      "Economic": "4"
      "Other": "5"
    tooltips:
      "Stock-Related": "Discussion about a specific stock, equity, or share price"
      "Market-Related": "Discussion about overall market conditions or indices"
      "Company-Related": "Discussion about company operations, earnings, or management"
      "Economic": "Discussion about macroeconomic factors like GDP, employment, or interest rates"
      "Other": "Other financial topics not covered above"

annotation_instructions: |
  You will be shown a financial text (microblog post or news excerpt) along with
  the associated cashtag. Your task is to:
  1. Rate the sentiment on a 7-point scale from Very Bearish to Very Bullish.
  2. Classify the financial topic type.
  Consider the tone and implied market direction when rating sentiment.

html_layout: |
  <div style="padding: 15px; max-width: 800px; margin: auto;">
    <div style="background: #fefce8; border: 1px solid #fde68a; border-radius: 8px; padding: 12px; margin-bottom: 12px;">
      <strong style="color: #a16207;">Cashtag:</strong>
      <span style="font-size: 15px; font-weight: bold;">{{cashtag}}</span>
    </div>
    <div style="background: #f0f9ff; border: 1px solid #bae6fd; border-radius: 8px; padding: 16px; margin-bottom: 16px;">
      <strong style="color: #0369a1;">Text:</strong>
      <p style="font-size: 16px; line-height: 1.7; margin: 8px 0 0 0;">{{text}}</p>
    </div>
  </div>

allow_all_users: true
instances_per_annotator: 50
annotation_per_instance: 2
allow_skip: true
skip_reason_required: false

Sample Data

json
[
  {
    "id": "finsent_001",
    "text": "$AAPL crushing earnings expectations yet again! Revenue up 12% YoY and guidance looks fantastic. This stock is going to the moon!",
    "cashtag": "$AAPL"
  },
  {
    "id": "finsent_002",
    "text": "$TSLA shares plummeted 8% after production numbers came in well below estimates. The company faces significant headwinds in Q4.",
    "cashtag": "$TSLA"
  }
]

// ... and 8 more items

Dataset & paper

Cortis et al., SemEval 2017

Official dataset(opens in a new tab)Read the paper(opens in a new tab)

Citation (BibTeX)

bibtex
@inproceedings{cortis-etal-2017-semeval,
    title = "{S}em{E}val-2017 Task 5: Fine-Grained Sentiment Analysis on Financial Microblogs and News",
    author = "Cortis, Keith and Freitas, Andr{\'e} and Daudert, Tobias and Huerlimann, Manuela and Zarrouk, Manel and Handschuh, Siegfried and Davis, Brian",
    booktitle = "Proceedings of the 11th International Workshop on Semantic Evaluation ({S}em{E}val-2017)",
    year = "2017",
    publisher = "Association for Computational Linguistics",
    pages = "519--535",
    doi = "10.18653/v1/S17-2089",
    url = "https://aclanthology.org/S17-2089"
}

Details

Annotation Types

likertradio

Domain

SemEvalNLPFinancial AnalysisSentiment Analysis

Use Cases

Financial SentimentMarket AnalysisSocial Media Analysis

Tags

semevalsemeval-2017shared-taskfinancial-sentimentstock-marketmicroblogsfine-grained

Found an issue or want to improve this design?

Open an Issue(opens in a new tab)

Related Designs

Assessing Humor in Edited News Headlines

Rate the funniness of edited news headlines on a 0-3 scale and judge which of two edited versions is funnier, based on SemEval-2020 Task 7 (Hossain et al.). Headlines are minimally edited by replacing a single word to create humorous effect.

likertradio

MTRAGEval: Evaluating Multi-Turn RAG Conversations

SemEval-2026 Task 8 (MTRAGEval): evaluating retrieval-augmented generation (RAG) in multi-turn conversations, built on the MTRAG benchmark. Given a conversation, the retrieved reference passages, and a system-generated response, annotators judge how faithful the response is to the passages, how well it answers the user's latest question, and whether it correctly handles unanswerable, underspecified, non-standalone, or unclear questions. The shared task covers retrieval (A), generation (B), and full RAG (C).

radiolikert

PreTENS - Presuppositional Acceptability

Judge the acceptability of sentences whose two nominal arguments are presupposed to stand (or not) in an ordered taxonomic relation. Includes a binary acceptable/unacceptable judgment and a 1-7 acceptability rating. Based on SemEval-2022 Task 3 (Zamparelli et al.).

radiolikert