Project Snapshots for Citable Datasets and Audit Trails

potato snapshot freezes a project's config, data files, and annotations under an id that a paper or a dataset card can cite. Each snapshot records a SHA-256 content hash over those files and answers, and anyone holding the snapshot's files can recompute the hash to confirm that their copy is the one cited. A snapshot also stores Krippendorff's alpha for each annotation scheme as agreement stood on that day. Snapshots shipped in Potato 2.12.0. Every command reads the project from disk, whether or not the server is running.

A study keeps changing after the paper is written. Annotators join, and codes are corrected after adjudication. An audit trail in qualitative research is the record of data, coding decisions, and analysis steps that lets a reader judge whether the analysis was credible (Nowell et al., 2017). A series of snapshots gives a coding project a dated, hashed entry for each round, with the intercoder agreement at each round. A machine learning team can use the same commands to version a labeled dataset. A model card can then name the snapshot id of the labels a model was trained on.

• Commands: The subcommands are create, list, show, diff, tag, verify, and deposit, all under potato snapshot.
• Storage: Each snapshot is one folder under snapshots/ in the task directory, with tags in snapshots/tags.json and Zenodo deposits in snapshots/deposits.json.
• Integrity: Potato writes the files read-only and renames the folder into place only after every file is complete.
• Sharing: The admin dashboard's Publish Dataset page has a Snapshots card, two admin API routes list and create snapshots, and an optional deposit puts a snapshot on Zenodo with a DOI.

Commands

Every command takes the project's config file:

potato snapshot create config.yaml --note "pilot round finished" --tag pilot
potato snapshot list config.yaml
potato snapshot show config.yaml pilot
potato snapshot diff config.yaml pilot latest
potato snapshot tag config.yaml 20261102-175533 paper-v1
potato snapshot verify config.yaml paper-v1
potato snapshot verify --dir ./downloaded-snapshot
potato snapshot deposit config.yaml paper-v1 --check

From a Potato checkout, python potato/flask_server.py snapshot takes the same arguments. The table lists each command's arguments and flags:

A ref is latest, a full snapshot id, a tag, or the start of an id or of a content hash. A prefix has to match exactly one snapshot, and an ambiguous prefix stops the command with a list of the matches. A tag is 1 to 64 letters, digits, ., _, or -, starts with a letter or digit, and cannot be latest. Tags work like git tags. Potato changes only two files in the store after create, tags.json for tags and deposits.json for Zenodo deposits. A snapshot folder never changes after create writes the folder.

show prints a snapshot's summary line, hashes, Potato version, and alpha for each scheme:

20261102-175504-45d41bfa [pilot]  6 items, 2 annotators, 20 answers  pilot round finished
  created       2026-11-02T17:55:04+00:00
  content hash  45d41bfad181190c161eff442c2bd3e65bd0546f3a55c197f356db4447e72c93
  bundle sha256 addd652f15ee682301c6cab6968d8aa2f0b6d05e1e5fc6748d242d07f9f07af4
  potato        2.13.1
  alpha sentiment: 0.511
  alpha aspects: 0.718

An answer in the counts above is one annotator's labels on one item for one scheme. create prints the new id and content hash, the number of study files besides the config, each file left out as a credential, each folder the config names but the snapshot does not copy, and a warning for any file outside the task directory.

Files in a snapshot

Each snapshot is a folder, snapshots/<id>/ in the task directory. The id is the UTC time of creation followed by the first eight characters of the content hash, such as 20261102-175533-68e05868. The folder holds four files:

Each row of annotations.jsonl keeps label answers under "labels" and span answers under "spans" as [label, start, end] triples. A row with a label answer looks like the line below:

{"annotator": "bob", "item": "r2", "schema": "sentiment", "value": {"labels": [["neutral", "true"]]}}

The study's files are the data files plus every file a string in the config resolves to. Phase and survey files, attention-check and gold-item files, consent and instruction pages, and custom layouts all count. Potato follows references inside a referenced YAML file as well, up to three levels deep. manifest.json lists the study files under project_files. Potato lists each folder the config names under referenced_directories and does not copy the folder because a folder of audio or images can be any size. Files in such a folder are outside the content hash. Archive the media separately and cite the media archive beside the snapshot.

A study file outside the task directory goes into the bundle as external/<name>, and create prints a warning. Potato refuses to serve a config that points outside its project, and a study frozen that way cannot be started as written. Move the file into the task directory and take a new snapshot.

Credentials left out

Because a snapshot is made to be cited and handed on, the bundle leaves out three kinds of credential:

• The account roster and its password hashes stay out. The roster is authentication.user_config_path when that key is set, and otherwise user_config.json beside the output directory, or inside the output directory when output_annotation_dir ends in /.
• A file named under a config key that mentions a password, secret, token, credential, API key, or private key stays out. The rule covers ssl_key and every key ending in key_file, such as sp_key_file.
• admin_api_key.txt stays out.

manifest.json lists each omitted file with the reason under omitted, and create prints the list. None of the omitted files is in the content hash. A project unpacked from a snapshot has no accounts, and annotators register again unless you restore the roster from your own copy.

The content hash

content_hash is a SHA-256 digest over the config file, each study file under its name in the bundle, the list of item ids, and the annotation rows, in that order. The hash changes when the config, a study file, or any answer changes. The SQLite databases are in the bundle and outside the hash.

The databases stay out of the hash because the server rewrites the databases while the server runs. A hash that changed with every rewrite could not tell a reader whether the annotations had changed. QDA Mode's memos, codebook, and cases live in project.sqlite. An edit to a memo or a code definition alone leaves the content hash unchanged, and create reports that nothing has changed. Pass --force to freeze a codebook revision that changed no answer.

verify runs three checks:

1. bundle.tar.gz must match bundle_sha256 in the manifest.
2. annotations.jsonl must match annotations_sha256.
3. The config and study files unpacked from the bundle, hashed with the item ids and the annotation rows, must reproduce content_hash.

A reader holding only the four files can run the same checks with --dir:

$ potato snapshot verify --dir ./downloaded-snapshot
20261102-175533-68e05868: OK
  content hash 68e058683e7088e3a344d56d7e39172b3d7d3b9b469ba9e1a4a1876b4296b4d2
  Compare it with the hash the paper or dataset card cites.

A passing check shows only that the files agree with the manifest in the same folder. Any other snapshot passes its own check too. Compare the printed hash with the hash the paper cites. For a changed or damaged file, verify prints one line per problem, such as annotations.jsonl does not match annotations_sha256, and exits with status 1.

Comparing two snapshots

diff compares two snapshots answer by answer and counts the answers added, removed, and changed. Answers are matched on annotator, item, and scheme. The report also lists items and annotators added or removed, says whether the config changed, and gives alpha for each scheme before and after:

$ potato snapshot diff config.yaml pilot paper-v1
20261102-175504-45d41bfa -> 20261102-175533-68e05868
Items: +0 -0
Annotations: +0 -0 changed 1
  changed  bob  r2  sentiment
Alpha aspects: 0.718 -> 0.718 (+0.000)
Alpha sentiment: 0.511 -> 0.298 (-0.213)

Between the two snapshots in the example, one annotator changed one answer, and alpha on the six-item sentiment scheme fell from 0.511 to 0.298. The text report names up to 20 rows of each kind. --json gives the full counts with the same rows in machine-readable form. For two snapshots with equal content hashes, diff prints Same content.

Alpha in a snapshot is Krippendorff's alpha (nominal) as Paper Mode computes the coefficient. A multiselect scheme is scored per item and label. Each scheme also gets one of Krippendorff's interpretations, "acceptable agreement" from 0.800, "tentative agreement" from 0.667, or "low agreement" below 0.667. A scheme with no computable alpha shows n/a.

Restoring a snapshot

Potato has no restore command. A snapshot's bundle.tar.gz unpacks to a project/ folder with the config, the study files, the annotation output, and the databases. Potato starts on the unpacked config like any other project:

mkdir restored
tar -xzf snapshots/20261102-175533-68e05868/bundle.tar.gz -C restored
potato start restored/project/config.yaml -p 8000

The restored project has no accounts and no copied media folders. Copy the folders listed under referenced_directories back from your own archive before annotators return.

Snapshots in the admin dashboard

The admin dashboard's Publish Dataset page has a Snapshots card. The card lists the project's snapshots with item, annotator, and answer counts, and creates a snapshot from a note and an optional tag. Two admin routes do the same over HTTP:

The POST body takes note, tag, and force, all optional:

curl -H "X-API-Key: $KEY" http://localhost:8000/admin/api/snapshots
curl -X POST -H "X-API-Key: $KEY" -H "Content-Type: application/json" \
     -d '{"note": "second coding round", "tag": "round-2"}' \
     http://localhost:8000/admin/api/snapshots

Before each release from the Publish Dataset page, Potato freezes a snapshot or reuses the latest snapshot when nothing changed. A release to a local archive, to the Hugging Face Hub, or to Zenodo gets a snapshot the same way. The generated dataset card names the snapshot in its citation section:

This release is Potato snapshot 20261102-175533-68e05868 (content hash 68e058683e70…). Anyone holding the project folder can check that their copy is this one with potato snapshot verify config.yaml 20261102-175533-68e05868.

If the freeze fails, the release goes ahead and Potato shows the failure as a warning. python -m potato.publish freezes no snapshot, and the dataset card written by that command names no snapshot. Run potato snapshot create before a command-line release and cite the id yourself.

Depositing a snapshot on Zenodo

potato snapshot deposit uploads a snapshot's four files and a short README to a new Zenodo record. A reader can download the files and run potato snapshot verify --dir on the downloaded folder. The record's title, authors, license, keywords, and description come from the config's dataset_metadata block. Potato reads the block from the config as the config stands at deposit time. The record's version is the snapshot id, and the description adds the snapshot id, counts, and content hash. Add a dataset_metadata block to the config:

dataset_metadata:
  pretty_name: Clinic Access Interviews (coded excerpts)
  description: Interview excerpts coded for barriers to clinic access by two coders.
  license: cc-by-4.0
  keywords: [qualitative-coding, health-services]
  authors:
    - name: A. Researcher
      affiliation: University of Example
      orcid: 0000-0002-1825-0097

Deposits go to the Zenodo sandbox as unpublished drafts unless you ask otherwise:

export ZENODO_TOKEN=...
potato snapshot deposit config.yaml paper-v1 --check
potato snapshot deposit config.yaml paper-v1 --annotator-ids-public
potato snapshot deposit config.yaml paper-v1 --annotator-ids-public --production --publish

The first command lists the files, runs the checks below, and sends nothing. The second creates a sandbox draft. The third publishes a record on zenodo.org and mints the DOI.

The files go out exactly as frozen because the hash check depends on the bytes. Potato runs two checks before sending anything, and no flag overrides the credentials check:

• Credentials. Potato parses the config and every other YAML, JSON, .env, or TOML file in the bundle except the data files. The deposit stops when one of those files has a key such as api_key, token, password, or client_secret, or a key ending in _api_key, _secret, _token, or _password, whose value is a literal of 12 characters or more. A value shaped like a known provider's key stops the deposit in any file, including a custom layout's HTML or JavaScript. A PEM private key in any file, a config file that cannot be parsed, and a config file over 8 MB all stop the deposit as well. Replace each value with ${ENV_VAR} or move the file out of the project, take a new snapshot, and deposit the new snapshot.
• Annotator ids. Potato counts the annotator ids in the answers and in every saved state in the bundle, names the ids that look like email addresses, and stops the deposit unless you pass --annotator-ids-public. An annotator with a saved state and no answers is counted because the saved state holds consent and survey answers.

A deposited snapshot keeps annotator ids as stored, while python -m potato.publish maps annotator ids to A1 through An by default. Use the publisher when the ids must stay private. Each deposit is recorded in snapshots/deposits.json, and show prints each deposit with its Zenodo URL and DOI. Once a snapshot has a published DOI on zenodo.org, the dataset card of any release frozen as that snapshot links to the DOI.

Removing a snapshot

A snapshot's files are read-only. Make the files writable, delete the folder, and delete any tag that pointed at the snapshot from snapshots/tags.json:

chmod -R u+w snapshots/20261102-175504-45d41bfa
rm -rf snapshots/20261102-175504-45d41bfa

A create interrupted partway leaves a <id>.partial folder that no command reads. You can delete that folder the same way. The message Another snapshot create is writing … right now means two creates of the same content started in the same second. One of the two creates writes the snapshot, and potato snapshot list shows the result.

Further reading

• Paper Mode - The LaTeX dataset report with the same agreement statistics
• QDA Mode - Codes, memos, and cases for qualitative coding
• Annotation History - Timestamps for every annotation action
• Inter-Annotator Agreement - Choosing and reading an agreement coefficient
• Documenting Datasets and Models - Dataset cards and datasheets for a release

For implementation details, see the source documentation.

References

1. Nowell, L. S., Norris, J. M., White, D. E., & Moules, N. J. (2017). Thematic Analysis: Striving to Meet the Trustworthiness Criteria. International Journal of Qualitative Methods, 16(1). doi:10.1177/1609406917733847