# Project Snapshots for Citable Datasets and Audit Trails

Source: https://www.potatoannotator.com/docs/tools/project-snapshots

`potato snapshot` freezes a project's config, data files, and annotations under an id that a paper or a dataset card can cite. Each snapshot records a SHA-256 content hash over those files and answers, and anyone holding the snapshot's files can recompute the hash to confirm that their copy is the one cited. A snapshot also stores [Krippendorff's alpha](https://en.wikipedia.org/wiki/Krippendorff%27s_alpha) for each annotation scheme as agreement stood on that day. Snapshots shipped in Potato 2.12.0. Every command reads the project from disk, whether or not the server is running.

A study keeps changing after the paper is written. Annotators join, and codes are corrected after adjudication. An audit trail in qualitative research is the record of data, coding decisions, and analysis steps that lets a reader judge whether the analysis was credible ([Nowell et al., 2017](https://doi.org/10.1177/1609406917733847)). A series of snapshots gives a coding project a dated, hashed entry for each round, with the intercoder agreement at each round. A machine learning team can use the same commands to version a labeled dataset. A model card can then name the snapshot id of the labels a model was trained on.

- **Commands:** The subcommands are `create`, `list`, `show`, `diff`, `tag`, `verify`, and `deposit`, all under `potato snapshot`.
- **Storage:** Each snapshot is one folder under `snapshots/` in the task directory, with tags in `snapshots/tags.json` and Zenodo deposits in `snapshots/deposits.json`.
- **Integrity:** Potato writes the files read-only and renames the folder into place only after every file is complete.
- **Sharing:** The admin dashboard's Publish Dataset page has a Snapshots card, two admin API routes list and create snapshots, and an optional deposit puts a snapshot on Zenodo with a DOI.

## Commands

Every command takes the project's config file:

```bash
potato snapshot create config.yaml --note "pilot round finished" --tag pilot
potato snapshot list config.yaml
potato snapshot show config.yaml pilot
potato snapshot diff config.yaml pilot latest
potato snapshot tag config.yaml 20261102-175533 paper-v1
potato snapshot verify config.yaml paper-v1
potato snapshot verify --dir ./downloaded-snapshot
potato snapshot deposit config.yaml paper-v1 --check
```

From a Potato checkout, `python potato/flask_server.py snapshot` takes the same arguments. The table lists each command's arguments and flags:

| Command | Arguments and flags | What the command does |
|---|---|---|
| `create` | `config`, `--note TEXT` (default empty), `--tag NAME`, `--force` | Freezes the project. When the content hash equals the latest snapshot's hash, `create` writes nothing and says so. `--force` writes a new snapshot anyway. A `--tag` given with unchanged content is pointed at the existing snapshot |
| `list` | `config`, `--json` | Lists snapshots newest first, with tags, counts of items, annotators, and answers, and the note |
| `show` | `config`, `ref` (default `latest`), `--json` | Prints one snapshot's hashes, Potato version, previous snapshot, deposits, and alpha per scheme |
| `diff` | `config`, `old`, `new` (default `latest`), `--json` | Reports what changed between two snapshots |
| `tag` | `config`, `ref`, `name` | Points a name at a snapshot. A name that already pointed at another snapshot moves |
| `verify` | `config` and `ref` (default `latest`), or `--dir FOLDER` | Re-hashes a snapshot against its manifest. With `--dir`, checks a folder of snapshot files and needs no config |
| `deposit` | `config`, `ref` (default `latest`), `--token`, `--production`, `--publish`, `--annotator-ids-public`, `--check` | Uploads the snapshot's files to Zenodo |

A `ref` is `latest`, a full snapshot id, a tag, or the start of an id or of a content hash. A prefix has to match exactly one snapshot, and an ambiguous prefix stops the command with a list of the matches. A tag is 1 to 64 letters, digits, `.`, `_`, or `-`, starts with a letter or digit, and cannot be `latest`. Tags work like git tags. Potato changes only two files in the store after `create`, `tags.json` for tags and `deposits.json` for Zenodo deposits. A snapshot folder never changes after `create` writes the folder.

`show` prints a snapshot's summary line, hashes, Potato version, and alpha for each scheme:

```text
20261102-175504-45d41bfa [pilot]  6 items, 2 annotators, 20 answers  pilot round finished
  created       2026-11-02T17:55:04+00:00
  content hash  45d41bfad181190c161eff442c2bd3e65bd0546f3a55c197f356db4447e72c93
  bundle sha256 addd652f15ee682301c6cab6968d8aa2f0b6d05e1e5fc6748d242d07f9f07af4
  potato        2.13.1
  alpha sentiment: 0.511
  alpha aspects: 0.718
```

An answer in the counts above is one annotator's labels on one item for one scheme. `create` prints the new id and content hash, the number of study files besides the config, each file left out as a credential, each folder the config names but the snapshot does not copy, and a warning for any file outside the task directory.

## Files in a snapshot

Each snapshot is a folder, `snapshots/<id>/` in the task directory. The id is the UTC time of creation followed by the first eight characters of the content hash, such as `20261102-175533-68e05868`. The folder holds four files:

| File | Contents |
|---|---|
| `bundle.tar.gz` | The config, the study's files, the annotation output directory, and the project's SQLite databases, all under `project/`. The databases are `project.sqlite` (memos, the codebook, cases, typing sessions, and the review workflow), `datasets.sqlite`, and `engine.sqlite` (the labeling engine) |
| `annotations.jsonl` | One row per annotator, item, and scheme, with every label and span the annotator saved, in a fixed sort order. `diff` reads `annotations.jsonl` |
| `items.json` | The item ids from the data files, in file order |
| `manifest.json` | The id, creation time, note, task name, Potato version, hashes, counts, agreement summary, study files, omitted files, the previous snapshot's id, and a sequence number |

Each row of `annotations.jsonl` keeps label answers under `"labels"` and span answers under `"spans"` as `[label, start, end]` triples. A row with a label answer looks like the line below:

```json
{"annotator": "bob", "item": "r2", "schema": "sentiment", "value": {"labels": [["neutral", "true"]]}}
```

The study's files are the data files plus every file a string in the config resolves to. Phase and survey files, attention-check and gold-item files, consent and instruction pages, and custom layouts all count. Potato follows references inside a referenced YAML file as well, up to three levels deep. `manifest.json` lists the study files under `project_files`. Potato lists each folder the config names under `referenced_directories` and does not copy the folder because a folder of audio or images can be any size. Files in such a folder are outside the content hash. Archive the media separately and cite the media archive beside the snapshot.

A study file outside the task directory goes into the bundle as `external/<name>`, and `create` prints a warning. Potato refuses to serve a config that points outside its project, and a study frozen that way cannot be started as written. Move the file into the task directory and take a new snapshot.

### Credentials left out

Because a snapshot is made to be cited and handed on, the bundle leaves out three kinds of credential:

- The account roster and its password hashes stay out. The roster is `authentication.user_config_path` when that key is set, and otherwise `user_config.json` beside the output directory, or inside the output directory when `output_annotation_dir` ends in `/`.
- A file named under a config key that mentions a password, secret, token, credential, API key, or private key stays out. The rule covers `ssl_key` and every key ending in `key_file`, such as `sp_key_file`.
- `admin_api_key.txt` stays out.

`manifest.json` lists each omitted file with the reason under `omitted`, and `create` prints the list. None of the omitted files is in the content hash. A project unpacked from a snapshot has no accounts, and annotators register again unless you restore the roster from your own copy.

## The content hash

`content_hash` is a [SHA-256](https://en.wikipedia.org/wiki/SHA-2) digest over the config file, each study file under its name in the bundle, the list of item ids, and the annotation rows, in that order. The hash changes when the config, a study file, or any answer changes. The SQLite databases are in the bundle and outside the hash.

The databases stay out of the hash because the server rewrites the databases while the server runs. A hash that changed with every rewrite could not tell a reader whether the annotations had changed. QDA Mode's memos, codebook, and cases live in `project.sqlite`. An edit to a memo or a code definition alone leaves the content hash unchanged, and `create` reports that nothing has changed. Pass `--force` to freeze a codebook revision that changed no answer.

`verify` runs three checks:

1. `bundle.tar.gz` must match `bundle_sha256` in the manifest.
2. `annotations.jsonl` must match `annotations_sha256`.
3. The config and study files unpacked from the bundle, hashed with the item ids and the annotation rows, must reproduce `content_hash`.

A reader holding only the four files can run the same checks with `--dir`:

```text
$ potato snapshot verify --dir ./downloaded-snapshot
20261102-175533-68e05868: OK
  content hash 68e058683e7088e3a344d56d7e39172b3d7d3b9b469ba9e1a4a1876b4296b4d2
  Compare it with the hash the paper or dataset card cites.
```

A passing check shows only that the files agree with the manifest in the same folder. Any other snapshot passes its own check too. Compare the printed hash with the hash the paper cites. For a changed or damaged file, `verify` prints one line per problem, such as `annotations.jsonl does not match annotations_sha256`, and exits with status 1.

## Comparing two snapshots

`diff` compares two snapshots answer by answer and counts the answers added, removed, and changed. Answers are matched on annotator, item, and scheme. The report also lists items and annotators added or removed, says whether the config changed, and gives alpha for each scheme before and after:

```text
$ potato snapshot diff config.yaml pilot paper-v1
20261102-175504-45d41bfa -> 20261102-175533-68e05868
Items: +0 -0
Annotations: +0 -0 changed 1
  changed  bob  r2  sentiment
Alpha aspects: 0.718 -> 0.718 (+0.000)
Alpha sentiment: 0.511 -> 0.298 (-0.213)
```

Between the two snapshots in the example, one annotator changed one answer, and alpha on the six-item `sentiment` scheme fell from 0.511 to 0.298. The text report names up to 20 rows of each kind. `--json` gives the full counts with the same rows in machine-readable form. For two snapshots with equal content hashes, `diff` prints `Same content.`

Alpha in a snapshot is Krippendorff's alpha (nominal) as [Paper Mode](/docs/features/paper-mode) computes the coefficient. A multiselect scheme is scored per item and label. Each scheme also gets one of Krippendorff's interpretations, "acceptable agreement" from 0.800, "tentative agreement" from 0.667, or "low agreement" below 0.667. A scheme with no computable alpha shows `n/a`.

## Restoring a snapshot

Potato has no restore command. A snapshot's `bundle.tar.gz` unpacks to a `project/` folder with the config, the study files, the annotation output, and the databases. Potato starts on the unpacked config like any other project:

```bash
mkdir restored
tar -xzf snapshots/20261102-175533-68e05868/bundle.tar.gz -C restored
potato start restored/project/config.yaml -p 8000
```

The restored project has no accounts and no copied media folders. Copy the folders listed under `referenced_directories` back from your own archive before annotators return.

## Snapshots in the admin dashboard

The admin dashboard's **Publish Dataset** page has a Snapshots card. The card lists the project's snapshots with item, annotator, and answer counts, and creates a snapshot from a note and an optional tag. Two admin routes do the same over HTTP:

| Route | Who may call the route | Result |
|---|---|---|
| `GET /admin/api/snapshots` | the admin API key, or a session that may view the admin dashboard | `{"snapshots": [...]}`, newest first, each with its manifest, tags, and deposits |
| `POST /admin/api/snapshots` | the admin API key, or a session with the `export_data` permission | 201 with the new manifest. 200 with `"unchanged": true` and the existing snapshot when nothing changed. 400 for an invalid tag, with nothing written. 409 while another create of the same snapshot is running |

The `POST` body takes `note`, `tag`, and `force`, all optional:

```bash
curl -H "X-API-Key: $KEY" http://localhost:8000/admin/api/snapshots
curl -X POST -H "X-API-Key: $KEY" -H "Content-Type: application/json" \
     -d '{"note": "second coding round", "tag": "round-2"}' \
     http://localhost:8000/admin/api/snapshots
```

Before each release from the Publish Dataset page, Potato freezes a snapshot or reuses the latest snapshot when nothing changed. A release to a local archive, to the Hugging Face Hub, or to Zenodo gets a snapshot the same way. The generated dataset card names the snapshot in its citation section:

> This release is Potato snapshot `20261102-175533-68e05868` (content hash `68e058683e70…`). Anyone holding the project folder can check that their copy is this one with `potato snapshot verify config.yaml 20261102-175533-68e05868`.

If the freeze fails, the release goes ahead and Potato shows the failure as a warning. `python -m potato.publish` freezes no snapshot, and the dataset card written by that command names no snapshot. Run `potato snapshot create` before a command-line release and cite the id yourself.

## Depositing a snapshot on Zenodo

`potato snapshot deposit` uploads a snapshot's four files and a short README to a new [Zenodo](https://zenodo.org/) record. A reader can download the files and run `potato snapshot verify --dir` on the downloaded folder. The record's title, authors, license, keywords, and description come from the config's `dataset_metadata` block. Potato reads the block from the config as the config stands at deposit time. The record's version is the snapshot id, and the description adds the snapshot id, counts, and content hash. Add a `dataset_metadata` block to the config:

```yaml
dataset_metadata:
  pretty_name: Clinic Access Interviews (coded excerpts)
  description: Interview excerpts coded for barriers to clinic access by two coders.
  license: cc-by-4.0
  keywords: [qualitative-coding, health-services]
  authors:
    - name: A. Researcher
      affiliation: University of Example
      orcid: 0000-0002-1825-0097
```

Deposits go to the Zenodo sandbox as unpublished drafts unless you ask otherwise:

```bash
export ZENODO_TOKEN=...
potato snapshot deposit config.yaml paper-v1 --check
potato snapshot deposit config.yaml paper-v1 --annotator-ids-public
potato snapshot deposit config.yaml paper-v1 --annotator-ids-public --production --publish
```

The first command lists the files, runs the checks below, and sends nothing. The second creates a sandbox draft. The third publishes a record on zenodo.org and mints the DOI.

| Flag | Default | Effect |
|---|---|---|
| `--token` | `ZENODO_TOKEN` | The Zenodo access token |
| `--production` | off (sandbox) | Deposit on zenodo.org instead of sandbox.zenodo.org |
| `--publish` | off (draft) | Publish the record and mint a DOI. A draft holds a reserved DOI until the draft is published |
| `--annotator-ids-public` | off | Confirm that the annotator ids may be published as stored |
| `--check` | off | List the files and run the checks without sending anything |

The files go out exactly as frozen because the hash check depends on the bytes. Potato runs two checks before sending anything, and no flag overrides the credentials check:

- **Credentials.** Potato parses the config and every other YAML, JSON, `.env`, or TOML file in the bundle except the data files. The deposit stops when one of those files has a key such as `api_key`, `token`, `password`, or `client_secret`, or a key ending in `_api_key`, `_secret`, `_token`, or `_password`, whose value is a literal of 12 characters or more. A value shaped like a known provider's key stops the deposit in any file, including a custom layout's HTML or JavaScript. A PEM private key in any file, a config file that cannot be parsed, and a config file over 8 MB all stop the deposit as well. Replace each value with `${ENV_VAR}` or move the file out of the project, take a new snapshot, and deposit the new snapshot.
- **Annotator ids.** Potato counts the annotator ids in the answers and in every saved state in the bundle, names the ids that look like email addresses, and stops the deposit unless you pass `--annotator-ids-public`. An annotator with a saved state and no answers is counted because the saved state holds consent and survey answers.

A deposited snapshot keeps annotator ids as stored, while `python -m potato.publish` maps annotator ids to `A1` through `An` by default. Use the publisher when the ids must stay private. Each deposit is recorded in `snapshots/deposits.json`, and `show` prints each deposit with its Zenodo URL and DOI. Once a snapshot has a published DOI on zenodo.org, the dataset card of any release frozen as that snapshot links to the DOI.

## Removing a snapshot

A snapshot's files are read-only. Make the files writable, delete the folder, and delete any tag that pointed at the snapshot from `snapshots/tags.json`:

```bash
chmod -R u+w snapshots/20261102-175504-45d41bfa
rm -rf snapshots/20261102-175504-45d41bfa
```

A `create` interrupted partway leaves a `<id>.partial` folder that no command reads. You can delete that folder the same way. The message `Another snapshot create is writing … right now` means two creates of the same content started in the same second. One of the two creates writes the snapshot, and `potato snapshot list` shows the result.

## Further reading

- [Paper Mode](/docs/features/paper-mode) - The LaTeX dataset report with the same agreement statistics
- [QDA Mode](/docs/qda/qda-mode) - Codes, memos, and cases for qualitative coding
- [Annotation History](/docs/features/annotation-history) - Timestamps for every annotation action
- [Inter-Annotator Agreement](/docs/guides/inter-annotator-agreement) - Choosing and reading an agreement coefficient
- [Documenting Datasets and Models](/docs/guides/documenting-datasets-and-models) - Dataset cards and datasheets for a release

For implementation details, see the [source documentation](https://github.com/davidjurgens/potato/blob/master/docs/tools/snapshots.md).

## References

1. Nowell, L. S., Norris, J. M., White, D. E., & Moules, N. J. (2017). Thematic Analysis: Striving to Meet the Trustworthiness Criteria. *International Journal of Qualitative Methods*, 16(1). [doi:10.1177/1609406917733847](https://doi.org/10.1177/1609406917733847)