Skip to content
beginnertext

Suggestion Mining from Online Reviews and Forums

Binary classification of sentences as suggestions or non-suggestions in online reviews and forum posts. Based on SemEval-2019 Task 9 (Suggestion Mining).

Submit

Configuration Fileconfig.yaml

This Potato config reproduces the annotation task. Save it as config.yaml and run potato start config.yaml to try it.

yaml
# Suggestion Mining from Online Reviews and Forums
# Based on Negi et al., SemEval 2019
# Paper: https://aclanthology.org/S19-2151/
# Dataset: https://github.com/Semeval2019Task9/Subtask-A
#
# This task asks annotators to classify sentences from online reviews
# and forums as either containing a suggestion or not.
#
# Classification Labels:
# - Suggestion: The sentence contains an explicit or implicit suggestion
# - Non-Suggestion: The sentence does not contain a suggestion

annotation_task_name: "Suggestion Mining from Online Reviews"
task_dir: "."

data_files:
  - sample-data.json

item_properties:
  id_key: "id"
  text_key: "text"

output_annotation_dir: "annotation_output/"
output_annotation_format: "json"

port: 8000
server_name: localhost

annotation_schemes:
  - annotation_type: radio
    name: suggestion_classification
    description: "Does this sentence contain a suggestion?"
    labels:
      - "Suggestion"
      - "Non-Suggestion"
    keyboard_shortcuts:
      "Suggestion": "1"
      "Non-Suggestion": "2"
    tooltips:
      "Suggestion": "The sentence contains an explicit or implicit suggestion for improvement or action"
      "Non-Suggestion": "The sentence is a statement, opinion, or complaint without a suggestion"

annotation_instructions: |
  You will be shown a sentence from an online review or forum along with its domain.
  Your task is to determine whether the sentence contains a suggestion -- an explicit
  or implicit recommendation for improvement or action. Suggestions may use language
  like "should", "would be nice if", "I recommend", or imply a desired change.

html_layout: |
  <div style="padding: 15px; max-width: 800px; margin: auto;">
    <div style="background: #fefce8; border: 1px solid #fde68a; border-radius: 8px; padding: 12px; margin-bottom: 12px;">
      <strong style="color: #a16207;">Domain:</strong>
      <span style="font-size: 15px;">{{domain}}</span>
    </div>
    <div style="background: #f0f9ff; border: 1px solid #bae6fd; border-radius: 8px; padding: 16px; margin-bottom: 16px;">
      <strong style="color: #0369a1;">Text:</strong>
      <p style="font-size: 16px; line-height: 1.7; margin: 8px 0 0 0;">{{text}}</p>
    </div>
  </div>

allow_all_users: true
instances_per_annotator: 50
annotation_per_instance: 2
allow_skip: true
skip_reason_required: false

Sample Datasample-data.json

json
[
  {
    "id": "suggestion_001",
    "text": "It would be great if the app had a dark mode option for nighttime use.",
    "domain": "Software Reviews"
  },
  {
    "id": "suggestion_002",
    "text": "The battery life on this phone is absolutely terrible. It barely lasts half a day.",
    "domain": "Electronics Reviews"
  }
]

// ... and 8 more items

Get This Design

View on GitHub

Clone or download from the repository

Quick start:

git clone https://github.com/davidjurgens/potato-showcase.git
cd potato-showcase/semeval/2019/task09-suggestion-mining
potato start config.yaml

Dataset & paper

Negi et al., SemEval 2019

Official dataset ↗Read the paper ↗

Citation (BibTeX)

bibtex
@inproceedings{negi-etal-2019-semeval,
    title = "{S}em{E}val-2019 Task 9: Suggestion Mining from Online Reviews and Forums",
    author = "Negi, Sapna and Daudert, Tobias and Buitelaar, Paul",
    booktitle = "Proceedings of the 13th International Workshop on Semantic Evaluation",
    month = "June",
    year = "2019",
    address = "Minneapolis, Minnesota, USA",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/S19-2151/",
    doi = "10.18653/v1/S19-2151",
    pages = "877--887"
}

Details

Annotation Types

radio

Domain

SemEvalNLPOpinion MiningText Classification

Use Cases

Suggestion MiningReview AnalysisCustomer Feedback

Tags

semevalsemeval-2019shared-tasksuggestion-miningreviewsopinion-mining

Found an issue or want to improve this design?

Open an Issue

Related Designs

Machine-Generated Text Detection

Machine-generated text detection task requiring annotators to classify whether a given text was written by a human, generated by a machine, contains mixed authorship, or is uncertain. Based on SemEval-2024 Task 8.

radio

AdMIRe - Advancing Multimodal Idiomaticity Representation

Multimodal idiomaticity task (SemEval-2025 Task 1, Subtask A): given a potentially idiomatic nominal compound in a context sentence, annotators rank five candidate images by how well each represents the sense in which the compound is used.

radio

AfriSenti - African Language Sentiment

Sentiment analysis for tweets in African languages, classifying text as positive, negative, or neutral. Covers 14 African languages including Amharic, Hausa, Igbo, Yoruba, and Swahili. Based on SemEval-2023 Task 12 (Muhammad et al.).

radio