intermediatesurvey

Machine Translation Evaluation

Name: Potato
Author: Potato

Evaluate machine translation quality with adequacy and fluency ratings.

Configuration Fileconfig.yaml

yaml

annotation_task_name: "Machine Translation Evaluation"

task_description: "Evaluate the quality of the machine translation."
task_dir: "."
port: 8000

data_files:
  - "sample-data.json"

item_properties:
  id_key: id
  text_key: source
  context_key: translation

annotation_schemes:
  - annotation_type: likert
    name: adequacy
    description: "How much of the source meaning is preserved in the translation?"
    size: 5
    min_label: "None"
    max_label: "All"
    required: true

  - annotation_type: likert
    name: fluency
    description: "How fluent is the translation in the target language?"
    size: 5
    min_label: "Incomprehensible"
    max_label: "Flawless"
    required: true

  - annotation_type: multiselect
    name: errors
    description: "Select any errors present in the translation"
    labels:
      - "Mistranslation"
      - "Omission"
      - "Addition"
      - "Grammar error"
      - "Word order"
      - "Terminology"
    required: false

output_annotation_dir: "output/"
output_annotation_format: "json"

Sample Datasample-data.json

json

[
  {
    "id": "1",
    "source": "El gato negro duerme en el sofá.",
    "source_lang": "Spanish",
    "target_lang": "English",
    "translation": "The black cat sleeps on the couch."
  },
  {
    "id": "2",
    "source": "Je voudrais réserver une table pour deux personnes.",
    "source_lang": "French",
    "target_lang": "English",
    "translation": "I would like to book a table for two people."
  }
]

Try it live — no install

Boot the real Potato server in your browser (WebAssembly) and annotate with this exact config. Nothing leaves your machine.

▶ Run live in your browser

Get This Design

View on GitHub

Clone or download from the repository

Quick start:

git clone https://github.com/davidjurgens/potato-showcase.git
cd potato-showcase/evaluation/machine-translation-eval
potato start config.yaml

Details

Annotation Types

likertmultiselect

Domain

NLPTranslation

Use Cases

Machine TranslationMT Evaluation

Related Designs

AnnoMI Counselling Dialogue Annotation

Annotation of motivational interviewing counselling dialogues based on the AnnoMI dataset. Annotators label therapist and client utterances for MI techniques (open questions, reflections, affirmations) and client change talk (sustain talk, change talk), with quality ratings for therapeutic interactions.

radiomultiselect

Clickbait Detection (Webis Clickbait Corpus)

Classify headlines and social media posts as clickbait or non-clickbait based on the Webis Clickbait Corpus. Identify manipulative content designed to attract clicks through sensationalism, curiosity gaps, or misleading framing.