How to Let a Coding Agent Write Your Annotation Config
The common failure when an LLM writes a config is inventing an option that does not exist. A JSON Schema plus a validator turns that from a runtime surprise into an editor error.
The most common failure when an LLM writes an annotation config is inventing an option that does not exist. The model produces something plausible, the server ignores the unknown key or rejects the file, and the error surfaces minutes later as behaviour nobody configured. A machine-readable schema moves that failure to the moment of writing.
Add the modeline
One line at the top of a config:
# yaml-language-server: $schema=https://potatoannotator.readthedocs.io/en/latest/schemas/potato-config.schema.jsonThat turns on live validation in VS Code, JetBrains IDEs, Zed and Helix. An invented annotation_type is underlined as you type, and completion offers the real ones.
Potato ships this line on every example config, and its playground emits it on every download.
Validate in CI
check-jsonschema --schemafile \
https://potatoannotator.readthedocs.io/en/latest/schemas/potato-config.schema.json \
config.yamlThe JSON Schema is deliberately permissive about unknown keys, because the server only warns on them, and a stricter schema would reject working configs. So pair it with the project's own validator, which reports them:
python -m potato.validate_cli --strict config.yaml--strict treats an unknown key as fatal. That is what catches a typo like annotaton_type, which a permissive schema accepts silently.
Give the agent the spec
If you are pointing Claude Code, Codex or Cursor at a tool, generated specs are worth more than prose documentation, because they describe what the code does today rather than what someone remembered:
| Artifact | Covers |
|---|---|
| Config JSON Schema | 159 config keys, 61 annotation types, 24 display types, plus per-type conditionals |
| OpenAPI 3.1 | 419 paths, 456 operations |
llms.txt | Curated documentation index |
The per-type conditionals matter more than the key list: they are what tells an agent that a constant_sum scheme needs labels, before the server does.
Generate the spec from the code
If you maintain the tool rather than use it, the property that makes this work is that the schema is generated from the same registries the server validates against, and CI fails when it drifts.
A hand-maintained schema is worse than none. It will be trusted, and it will be wrong the first time someone adds a type without updating it.
What we do on this site
This site's own config builder is built the same way. It reads Potato's registries into a committed spec file, maps every field against it, and drops anything the registry does not contain. A harness then puts every generated config — one per annotation type, plus every starter template — through validate_cli --strict.
That harness caught real bugs: an annotation type that did not exist, three schemes missing required fields, and a duplicated YAML key that silently discarded the authorized-user list. None of them was visible by reading the output.