Skip to content
intermediatesurvey

Machine Translation Evaluation

Evaluate machine translation quality with adequacy and fluency ratings.

Try it live — no install

Boot the real Potato server in your browser (WebAssembly) and annotate with this exact config. Nothing leaves your machine.

Run live in your browser

Get This Design

View on GitHub(opens in a new tab)

Clone or download from the repository

Quick start:

bash
git clone https://github.com/davidjurgens/potato-showcase.git
cd potato-showcase/evaluation/machine-translation-eval
potato start config.yaml

Configuration File

yaml
annotation_task_name: "Machine Translation Evaluation"

task_description: "Evaluate the quality of the machine translation."
task_dir: "."
port: 8000

data_files:
  - "sample-data.json"

item_properties:
  id_key: id
  text_key: source
  context_key: translation

annotation_schemes:
  - annotation_type: likert
    name: adequacy
    description: "How much of the source meaning is preserved in the translation?"
    size: 5
    min_label: "None"
    max_label: "All"
    required: true

  - annotation_type: likert
    name: fluency
    description: "How fluent is the translation in the target language?"
    size: 5
    min_label: "Incomprehensible"
    max_label: "Flawless"
    required: true

  - annotation_type: multiselect
    name: errors
    description: "Select any errors present in the translation"
    labels:
      - "Mistranslation"
      - "Omission"
      - "Addition"
      - "Grammar error"
      - "Word order"
      - "Terminology"
    required: false

output_annotation_dir: "output/"
output_annotation_format: "json"

Sample Data

json
[
  {
    "id": "1",
    "source": "El gato negro duerme en el sofá.",
    "source_lang": "Spanish",
    "target_lang": "English",
    "translation": "The black cat sleeps on the couch."
  },
  {
    "id": "2",
    "source": "Je voudrais réserver une table pour deux personnes.",
    "source_lang": "French",
    "target_lang": "English",
    "translation": "I would like to book a table for two people."
  }
]

Details

Annotation Types

likertmultiselect

Domain

NLPTranslation

Use Cases

Machine TranslationMT Evaluation

Tags

translationmtevaluationmultilingual

Found an issue or want to improve this design?

Open an Issue(opens in a new tab)

Related Designs

AnnoMI Counselling Dialogue Annotation

Annotation of motivational interviewing counselling dialogues based on the AnnoMI dataset. Annotators label therapist and client utterances for MI techniques (open questions, reflections, affirmations) and client change talk (sustain talk, change talk), with quality ratings for therapeutic interactions.

radiomultiselect

Clickbait Detection (Webis Clickbait Corpus)

Classify headlines and social media posts as clickbait or non-clickbait based on the Webis Clickbait Corpus. Identify manipulative content designed to attract clicks through sensationalism, curiosity gaps, or misleading framing.

likertmultiselect

Conversation Quality Attributes

A generic dialogue-quality annotation template. Annotators classify each conversation turn's engagement level, rate its overall quality on a 5-point scale, and select applicable attributes (interesting, repetitive, coherent, informative, empathetic, generic). It is a reusable rubric for chatbot and conversational-AI evaluation and does not reproduce a specific published dataset's label set.

radiolikert