Parquet Export

New in v2.3.0

Apache Parquet is a columnar storage format built for analytical workloads. For large annotation datasets it compresses better than JSON or CSV, reads a subset of columns without touching the rest of the file, and loads natively in pandas, DuckDB, PyArrow, Spark, Polars and Hugging Face Datasets.

Parquet export needs PyArrow:

pip install "pyarrow>=12.0.0"

Running the Export

Parquet is one of Potato's registered export formats, so the export CLI writes it directly:

python -m potato.export -c config.yaml -f parquet -o ./parquet_export/

Potato reads the annotations recorded under the config's output_annotation_dir and writes the Parquet tables into the output directory you name.

Exporting on a Schedule

A long-running study usually wants a Parquet snapshot without someone remembering to run the CLI. Setting export_annotation_format makes Potato write the export periodically while annotation is underway:

output_annotation_dir: "output/"
 
# A list is accepted, so Parquet can sit alongside another format
export_annotation_format:
  - parquet
  - jsonl
 
# Seconds between automatic exports (default: 60)
auto_export_interval: 300

Auto-exports land in <output_annotation_dir>/exports/<format>/, so the Parquet tables appear under output/exports/parquet/. Annotations themselves are always stored as <output_annotation_dir>/<user>/user_state.json regardless of this setting.

The older output_annotation_format key is deprecated. Potato reads it as export_annotation_format and logs a warning, so rename it.

Output Files

An export produces up to four tables, each covering a different level of the data.

annotations.parquet

One row per (instance, annotator) pair, with the schemas flattened into columns.

Column types follow the annotation type: radio and select become strings holding the selected label, likert, slider and number become float64 (or string for a scale with named labels), multiselect becomes a list of strings, and text becomes a string.

A schema that stores several answers at once gets one column per sub-answer, named <schema>.<key>. A multirate schema named handling produces handling.Urgency and handling.Customer tone; constant_sum and soft_label spread the same way. A single-select stays in one column, because severity holding "Serious" is what is usable in a dataframe.

Every column appears on every row, filled with null where an annotator did not answer, so a schema only some annotators reached is still present in the file.

spans.parquet

One row per span annotation, written when the study has span schemas.

Read target_field whenever a study has more than one span target. A single span schema records in every span-target field on the page, so the field varies within a scheme, and span_id cannot recover it: the same words marked in two fields produce otherwise identical rows.

text is derived from the offsets at export time rather than stored, so it can never disagree with them.

items.parquet

One row per source item, carrying the original data.

phase_responses.parquet

Written when the study collected phase data, such as a consent form or a post-study survey, and export_include_phase_data is true. The key defaults to false, so a study that gathered consent and a post-study questionnaire gets no phase_responses.parquet until you turn it on, and the export lists the left-out responses as a warning.

Export Options

Pass format options through --option key=value, repeating the flag for each one:

python -m potato.export -c config.yaml -f parquet -o ./parquet_export/ \
    --option compression=zstd --option include_items=false

snappy is the right default for most projects because it decompresses fast enough that reads stay interactive. Choose zstd when the dataset is large enough that file size matters, since it compresses considerably better at similar speed.

Loading Parquet Data

pandas

import pandas as pd
 
annotations = pd.read_parquet("parquet_export/annotations.parquet")
spans = pd.read_parquet("parquet_export/spans.parquet")
items = pd.read_parquet("parquet_export/items.parquet")
 
# One column per schema, so a schema is selected by name
print(annotations["sentiment"].value_counts())
 
# Compute inter-annotator agreement on the sentiment schema
from sklearn.metrics import cohen_kappa_score
pivot = annotations.pivot(index="instance_id", columns="user_id", values="sentiment")
kappa = cohen_kappa_score(pivot.iloc[:, 0], pivot.iloc[:, 1])

DuckDB

-- Direct query without loading into memory
SELECT instance_id, sentiment, COUNT(*) AS annotator_count
FROM 'parquet_export/annotations.parquet'
GROUP BY instance_id, sentiment
ORDER BY annotator_count DESC;
 
-- Join annotations with the source items
SELECT a.instance_id, i.text, a.sentiment, a.user_id
FROM 'parquet_export/annotations.parquet' a
JOIN 'parquet_export/items.parquet' i
  ON a.instance_id = i.item_id;

PyArrow

import pyarrow.parquet as pq
 
# Read specific columns only (fast for wide tables)
table = pq.read_table(
    "parquet_export/annotations.parquet",
    columns=["instance_id", "user_id", "sentiment"]
)
 
df = table.to_pandas()
 
parquet_file = pq.ParquetFile("parquet_export/annotations.parquet")
print(f"Row groups: {parquet_file.metadata.num_row_groups}")
print(f"Total rows: {parquet_file.metadata.num_rows}")

Hugging Face Datasets

from datasets import load_dataset
 
dataset = load_dataset("parquet", data_files={
    "annotations": "parquet_export/annotations.parquet",
    "spans": "parquet_export/spans.parquet",
    "items": "parquet_export/items.parquet",
})
 
print(dataset["annotations"][0])
 
dataset["annotations"].push_to_hub("my-org/my-annotations", split="train")

Potato also ships a huggingface exporter that builds the same splits directly, which avoids the intermediate files when the Hub is the destination.

Polars

import polars as pl
 
annotations = pl.read_parquet("parquet_export/annotations.parquet")
 
label_counts = (
    annotations
    .group_by("sentiment")
    .agg(pl.len().alias("count"))
    .sort("count", descending=True)
)
print(label_counts)

Full Example

A span-annotation project that keeps JSONL as the working format and writes Parquet alongside it for analysis:

annotation_task_name: "NER Annotation Project"
task_dir: "."
 
data_files:
  - "data/documents.jsonl"
 
item_properties:
  id_key: doc_id
  text_key: text
 
annotation_schemes:
  - annotation_type: span
    name: entities
    description: "Highlight named entities"
    labels:
      - name: PERSON
      - name: ORGANIZATION
      - name: LOCATION
 
output_annotation_dir: "output/"
 
# Both formats are written periodically to output/exports/<format>/
export_annotation_format:
  - jsonl
  - parquet
auto_export_interval: 300
 
user_config:
  allow_all_users: true

After annotation, the span table is what carries the entity offsets:

import pandas as pd
 
spans = pd.read_parquet("output/exports/parquet/spans.parquet")
 
# Entity type distribution
print(spans["label"].value_counts())
 
# Average span length by type
spans["length"] = spans["end"] - spans["start"]
print(spans.groupby("label")["length"].mean())

Further Reading

• Export Formats -- COCO, YOLO, CoNLL, and the other registered exporters
• Remote Data Sources -- loading data from cloud storage
• Admin Dashboard -- monitoring annotation progress

For implementation details, see the source documentation.