Skip to content
intermediatesurvey

Text Summarization Evaluation

Rate the quality of AI-generated summaries on fluency, coherence, and faithfulness.

Try it live — no install

Boot the real Potato server in your browser (WebAssembly) and annotate with this exact config. Nothing leaves your machine.

Run live in your browser

Get This Design

View on GitHub(opens in a new tab)

Clone or download from the repository

Quick start:

bash
git clone https://github.com/davidjurgens/potato-showcase.git
cd potato-showcase/evaluation/text-summarization-eval
potato start config.yaml

Configuration File

yaml
annotation_task_name: "Text Summarization Evaluation"

task_description: "Rate the quality of the summary compared to the source document."
task_dir: "."
port: 8000

data_files:
  - "sample-data.json"

item_properties:
  id_key: id
  text_key: source
  context_key: summary

annotation_schemes:
  - annotation_type: likert
    name: fluency
    description: "How fluent and grammatical is the summary?"
    size: 5
    min_label: "Not fluent"
    max_label: "Very fluent"
    required: true

  - annotation_type: likert
    name: coherence
    description: "How well-organized and coherent is the summary?"
    size: 5
    min_label: "Incoherent"
    max_label: "Very coherent"
    required: true

  - annotation_type: likert
    name: faithfulness
    description: "Does the summary accurately reflect the source without hallucinations?"
    size: 5
    min_label: "Unfaithful"
    max_label: "Faithful"
    required: true

  - annotation_type: text
    name: comments
    description: "Optional comments on the summary quality"
    required: false

output_annotation_dir: "output/"
output_annotation_format: "json"

Sample Data

json
[
  {
    "id": "1",
    "source": "The International Space Station (ISS) has been continuously occupied since November 2000. It serves as a microgravity and space environment research laboratory where crew members conduct experiments in biology, physics, astronomy, and other fields. The ISS is a joint project among five space agencies: NASA, Roscosmos, JAXA, ESA, and CSA.",
    "summary": "The ISS has been occupied since 2000 and serves as a research lab for experiments. It's run by five space agencies including NASA."
  },
  {
    "id": "2",
    "source": "Machine learning models require large amounts of training data to achieve good performance. Data annotation is the process of labeling data to provide ground truth for model training. High-quality annotations are essential for building reliable AI systems.",
    "summary": "ML models need lots of labeled training data. Good annotations are crucial for building reliable AI."
  }
]

Details

Annotation Types

likerttext

Domain

NLPEvaluation

Use Cases

SummarizationNLG Evaluation

Tags

summarizationevaluationnlgquality

Found an issue or want to improve this design?

Open an Issue(opens in a new tab)

Related Designs

MTRAGEval: Evaluating Multi-Turn RAG Conversations

SemEval-2026 Task 8 (MTRAGEval): evaluating retrieval-augmented generation (RAG) in multi-turn conversations, built on the MTRAG benchmark. Given a conversation, the retrieved reference passages, and a system-generated response, annotators judge how faithful the response is to the passages, how well it answers the user's latest question, and whether it correctly handles unanswerable, underspecified, non-standalone, or unclear questions. The shared task covers retrieval (A), generation (B), and full RAG (C).

radiolikert

Automated Essay Scoring

Holistic and analytic scoring of student essays using a deep-neural approach to automated essay scoring (Uto, arXiv 2022). Annotators provide overall quality ratings, holistic scores on a 1-6 scale, and detailed feedback comments for educational assessment.

likertslider

Coreference Resolution (OntoNotes)

Link pronouns and noun phrases to the entities they refer to in text. Based on the OntoNotes coreference annotation guidelines and CoNLL shared tasks. Identify mention spans and cluster coreferent mentions together.

likertradio