Skip to content

Paper Mode

Turn an annotation project into a compilable LaTeX dataset report with one command. Label distributions, agreement statistics with correct citations, annotator tables, timing, and limitations.

New in v2.7.0

Every dataset paper ends the same way: reconstructing annotator counts, agreement statistics, label distributions, and timing from scattered output files, then hand-formatting them into LaTeX. Paper Mode does it in one command.

bash
python -m potato.paper config.yaml -o paper_export

One command turns the project into a compilable LaTeX dataset report — methods section and all.

The output is fully offline. No LLM, no server, no network.

text
paper_export/
├── paper.tex        # compilable standalone; every section cut-paste-able
├── paper.bib        # citations for every metric used (+ Potato)
├── tables/*.csv     # each table as CSV, for your own styling
└── summary.json     # every computed number, machine-readable

paper.tex compiles as-is, and every paragraph and table sits between markers so you can lift sections straight into a manuscript:

latex
%% === BLOCK: paragraph-annotation-methods ===
Annotators spent a median of 42 seconds per item (12.3 person-hours in total).
Inter-annotator agreement for \emph{politeness}, measured by Krippendorff's
$\alpha$ (nominal) \citep{krippendorff2004content} over the 214
multiply-annotated units, was $\alpha = 0.712$ (tentative agreement). ...
%% === END BLOCK: paragraph-annotation-methods ===

What it generates

BlockContent
paragraph-dataset-descriptionItem counts, schemes, labels, annotator and label totals — ready for a Data section
paragraph-annotation-methodsTiming, Krippendorff's α per scheme, mean pairwise Cohen's κ, all cited
table-distribution-<scheme>A booktabs label-distribution table per scheme, with totals
table-annotatorsItems, labels, and median seconds per item for each annotator
table-agreementα, mean κ, and interpretation per scheme against Krippendorff's 0.667 and 0.8 thresholds
paragraph-limitationsCaveats with real numbers: single-annotated units, low-α schemes, small annotator pools

CLI options

FlagDefaultDescription
-o, --output-dirpaper_exportWhere to write the report
--no-anonymizeoffKeep real usernames; by default annotators become A1..An

Method notes

  • Paper Mode reads {output_annotation_dir}/<user>/user_state.json directly, resolved relative to the config file, so it works on archived projects without starting a server.
  • Krippendorff's α (nominal) is computed with simpledorff — the same implementation as the admin dashboard — over units with two or more annotations. Pairwise Cohen's κ is computed per annotator pair with two or more shared items.
  • Categorical schemes (radio, likert, multiselect, select, bws) get distributions and agreement. Multiselect agreement treats each (item, label) checkbox as a binary-coded unit. Span and textbox schemes are listed as not covered.
  • Timing comes from behavioral logs, using total_time_ms and falling back to interaction timestamp spans, ignoring gaps over an hour.

Further reading