Exporting Annotations to Hugging Face Datasets

One command pushes Potato annotations to the Hugging Face Hub as a dataset with an annotations split, optional spans and items splits, and a generated dataset card.

Training code reads Hugging Face datasets, and Potato does not store annotations in that shape while a study is running. During annotation it persists per-user state, so the files on disk are organized around annotators rather than around training examples. The export step is where the two shapes meet, and Potato ships a Hub exporter so you do not have to write the conversion yourself.

What Potato writes during annotation

Potato saves each annotator's work as a user_state.json file under output_annotation_dir, one directory per user:

annotation_output/
├── user1/
│   └── user_state.json
├── user2/
│   └── user_state.json
└── ...

Those files are the live record, not an export format. Anything downstream reads them through the export CLI, which is also what keeps the schema names, the span offsets, and the original item fields lined up in one place.

Pushing to the Hub with the built-in exporter

The Hub exporter needs the optional dependencies, which install with the huggingface extra:

pip install potato-annotation[huggingface]

One command builds the dataset and pushes it. The --output value is the repository id on the Hub, and the token needs write access:

python -m potato.export \
    --config config.yaml \
    --format huggingface \
    --output your-org/my-annotations \
    --option token=hf_xxx

Set HF_TOKEN in the environment or run huggingface-cli login if you would rather not pass the token on the command line. Four other options control what goes up: private creates a private dataset, commit_message replaces the default commit text, and include_items and include_spans (both true by default) decide whether the source items and the span annotations travel with the labels.

The dataset that lands on the Hub carries up to three splits. The annotations split has one row per (instance_id, user_id) pair, with one column per annotation schema. The spans split has one row per span annotation, carrying instance_id, user_id, schema_name, start, end, label, and text. The items split has one row per original data item, keyed by item_id, with the rest of your source fields alongside it. Potato also generates a dataset card describing the schemas and their labels, counts the annotation records, and includes a usage snippet and a link back to the project.

Loading it back is the ordinary load_dataset call:

from datasets import load_dataset
 
ds = load_dataset("your-org/my-annotations")
for row in ds["annotations"]:
    print(row["instance_id"], row["sentiment"])

Two exported examples are public if you want to see the shape before running your own study: Blablablab/potato-demo-single-choice-annotations and Blablablab/potato-demo-agent-trace-annotations.

Reshaping the export for a training task

The Hub export keeps one row per annotator per item, which is the right shape for studying disagreement and the wrong shape for Trainer. When you need training examples, export to JSONL instead and reshape locally. The JSONL exporter writes annotations.jsonl into the output directory, with one record per (instance_id, user_id) pair:

python -m potato.export --config config.yaml --format jsonl --output ./export/

Each record carries instance_id, user_id, labels, spans, and links. Spans arrive with the text they cover already sliced out, so you do not have to re-resolve offsets against the source file.

import json
 
records = []
with open("export/annotations.jsonl") as f:
    for line in f:
        records.append(json.loads(line))
 
print(records[0]["instance_id"], records[0]["labels"])

One training example per item

With three annotators per item, a classifier needs one label per item, so the per-annotator rows have to be collapsed first. Majority vote is the usual choice, and keeping the annotator count alongside it lets you drop items where the vote was thin:

from collections import defaultdict
from statistics import mode
from datasets import Dataset
 
by_item = defaultdict(list)
for rec in records:
    by_item[rec["instance_id"]].append(rec["labels"]["sentiment"])
 
rows = [
    {"id": item_id, "label": mode(labels), "num_annotators": len(labels)}
    for item_id, labels in by_item.items()
]
dataset = Dataset.from_list([r for r in rows if r["num_annotators"] >= 2])

Keep the per-annotator rows as well. Items where annotators split are the ones worth reading, and a model trained on the soft label often beats one trained on the majority vote when the task is genuinely subjective.

Attaching the item text

The label rows reference items by id, so the text has to come from the items split or from your own data file:

items = {row["item_id"]: row for row in load_dataset("your-org/my-annotations")["items"]}
 
dataset = Dataset.from_list([
    {"text": items[r["id"]]["text"], "label": r["label"]}
    for r in rows
    if r["id"] in items
])

Fixing the label type

A string label column trains, but it loses the class order and the names in the model config. Casting to ClassLabel fixes both, and the names have to match the labels your Potato schema emitted:

from datasets import ClassLabel
 
dataset = dataset.cast_column(
    "label",
    ClassLabel(names=["Positive", "Negative", "Neutral"])
)

Train, validation, and test splits

Split on items rather than on annotation rows. Splitting on rows puts two annotators' judgments of the same item in different splits, which leaks the answer:

from datasets import DatasetDict
 
split = dataset.train_test_split(test_size=0.2, seed=42)
held_out = split["test"].train_test_split(test_size=0.5, seed=42)
 
dataset = DatasetDict({
    "train": split["train"],
    "validation": held_out["train"],
    "test": held_out["test"],
})

Span annotations as token labels

Sequence labeling models read a tag per token, and the spans split stores character offsets. The conversion is the same whether the spans came from a span schema or from an imported corpus, and it needs a tokenizer whose offsets you can map back to characters:

def to_iob(text, spans, tokenizer):
    encoding = tokenizer(text, return_offsets_mapping=True)
    tags = ["O"] * len(encoding["input_ids"])
 
    for span in spans:
        for i, (start, end) in enumerate(encoding["offset_mapping"]):
            if start == end:
                continue
            if start >= span["start"] and end <= span["end"]:
                prefix = "B" if start == span["start"] else "I"
                tags[i] = f"{prefix}-{span['label']}"
 
    return encoding["input_ids"], tags

Group the spans by instance_id first, since the export has one row per span rather than one row per item.

Audio and image columns

Audio and image tasks store a path or a URL in the item fields, and datasets loads the media when the column is cast to the matching feature type:

from datasets import Audio, Image
 
audio_ds = audio_ds.cast_column("audio", Audio(sampling_rate=16000))
image_ds = image_ds.cast_column("image", Image())

The sampling rate has to match what the model expects, so check the model card before you pick one. Casting resamples on read rather than rewriting the files.

What to record in the dataset card

The generated card covers the schemas, the label sets, and the record count. The rest is what the people who reuse the dataset will ask about, and it is much easier to write while the study is still fresh:

---
license: cc-by-4.0
task_categories:
  - text-classification
language:
  - en
---
 
## Annotation process
 
Who annotated, how many annotators per item, how they were recruited and paid,
and what the guidelines said. Link the guidelines if they are public.
 
## Agreement
 
Which coefficient you computed, over which items, and what it came out at.
 
## Label definitions
 
One sentence per label, in the words the annotators saw.

State the license in the frontmatter rather than leaving it implicit, since a dataset with no license is one nobody else can safely use. Version the repository when the labels change, so a paper can cite the state of the data it used.

Loading the result into training code

The dataset is now the shape Trainer expects, and nothing between the export and the training loop needs to touch it:

from transformers import Trainer
 
trainer = Trainer(
    model=model,
    train_dataset=dataset["train"],
    eval_dataset=dataset["validation"],
)

For the exporter's full option list and the round trip back from the Hub into a new Potato task, see the Hugging Face export documentation. For the other formats Potato writes, see Data Formats.