Potato lets clinicians annotate entities and relations in clinical notes, segment CT and MR volumes from DICOM or NIfTI, and draw on whole-slide pathology images, with several readers per case and their agreement measured. Potato is free and open source and runs on your own servers, so clinical notes and images stay inside your institution.
Clinical annotation projects tend to share two requirements. The data cannot go to a cloud service, so the tool has to run on the institution's own infrastructure, and the annotators are scarce experts whose labels need checking against each other. Potato runs wherever you install the package, including on a network with no route out once setup is done. Potato assigns cases to annotators, keeps each annotator's labels, and reports agreement for every kind of label.
Clinical annotation tasks in Potato
Each row is an annotation type or display that ships with Potato 2.13, and the linked page has its configuration.
| Task | How Potato does it | Docs |
|---|---|---|
| Clinical NER and relations | Span labels for problems, treatments, tests, or social determinants, with links between them | Span annotation |
| Concept normalization | Mentions linked to UMLS concepts, Wikidata or a terminology service behind a REST API | Entity linking |
| Medical coding taxonomies | Hierarchical choices loaded from a file, such as a code system subset | Hierarchical multiselect |
| CT and MR segmentation | volume_annotation shows a DICOM series or NIfTI file in axial, coronal and sagittal views, with masks painted by brush or polygon, slice interpolation, and boxes saved as separate shapes outside the mask | Export formats |
| Pathology slides | Whole-slide images (.svs, .ndpi, .mrxs and others) read through openslide, with annotations at any zoom and regions imported from or exported to QuPath GeoJSON and ASAP XML | Deep zoom |
| Sleep studies and ECGs | A time_series display draws EDF, EDF+, and WFDB recordings, and an interval_label scheme scores 30-second epochs or marks intervals and beats | Time series |
| Chest X-rays and other images | Labels, boxes, and polygons on images, with agreement on the boxes | Computer vision |
| Questionnaires | Validated instruments such as PHQ-9 and GAD-7 before or after a task | Survey instruments |
Patient information in DICOM headers
DICOM headers usually contain the patient's name, ID, birth date, and institution, so the volume viewer shows annotators only the header fields you list in display_tags. A field that the DICOM de-identification basic profile removes, such as PatientName or PatientID, is shown only if you also list the field in allow_identifying_tags. The server enforces the list, so a browser cannot request other fields. The raw DICOM and NIfTI files are not served for download unless you turn that on.
annotation_schemes:
- annotation_type: volume_annotation
name: ct_segmentation
description: Outline the lesion and the bone ring
source_field: volume
labels:
- {name: Lesion, color: "#e6553a", key_value: "1"}
- {name: Bone, color: "#3a8ee6", key_value: "2"}
default_window: Soft tissue
display_tags: [Modality, BodyPartExamined, SliceThickness]The filter cannot cover everything. File and directory names reach the browser, and PACS exports often name folders after the patient, so use pseudonymous names such as case_001/. A series flagged with burned-in annotation is refused, but pixels in other series are not inspected, and the files on disk stay as uploaded. De-identify the files before upload when the people running the server must not see identifying data either. Volume support needs pip install 'potato-annotation[volumes]', and whole-slide images need the wsi extra.
Agreement and export
Masks from volume_annotation are scored by 3D Dice per label between each pair of annotators, and a pair where neither painted a label is left out for that label rather than counted as agreeing. Masks export as NIfTI label volumes in the source geometry (volume_nifti) or as DICOM SEG objects that reference the source series (dicom_seg). Text annotations export as JSONL, CSV, CoNLL, or Parquet, and every annotator's labels are kept for adjudication.
Working alongside 3D Slicer, MedCAT, and REDCap
3D Slicer is a desktop platform for medical image analysis and detailed segmentation by one person, and MedCATtrainer is an interface for training a biomedical concept recognition and linking model. Potato fits when the same cases go to several readers through a browser and you need their agreement. Potato is not a clinical data capture system and has no audit trail of the kind REDCap provides for regulated studies, so keep identifiable health records in a system built for that purpose, and use Potato for the annotation.
Setting up the study with a coding agent
potato-skill gives Claude Code, Codex, or Cursor the instructions for designing and building a Potato study. A plain description of the work is enough to start. A request can be this short:
Set up span annotation of medication names and doses in 500 de-identified discharge summaries on our own server, with two annotators per note and an adjudication step for disagreements.
The agent lists each assumption made along the way, builds the Potato task, and renders the task to check the result. Open the task and label a few items yourself before inviting annotators.
Frequently asked questions
Can I annotate clinical notes containing PHI on our own servers?
Potato runs on servers you control and writes annotations to those servers' disks. Potato sends nothing to an outside service unless you configure one, such as a hosted AI endpoint. Whether a given deployment may store protected health information is a decision for your institution's privacy and security office, which will look at access controls, sign-in, encryption, and logging on the server as well as the application.
Does a self-hosted annotation tool make a project HIPAA compliant?
No tool makes a project compliant by itself. Compliance depends on how the system is deployed and operated, including access control, agreements, and safeguards on the server. Potato helps in two ways. First, Potato can run inside your institution's network with no outside connection once installed. Second, the volume viewer limits which DICOM header fields annotators see.
Can Potato open DICOM or NIfTI images, and what do the masks export to?
Yes, from version 2.12. A volume_annotation scheme opens a DICOM series or a .nii or .nii.gz file in three linked views, and an RTSTRUCT on the item can provide starting masks. Masks export as NIfTI label volumes or as DICOM SEG objects that reference the source series. 4D volumes, fused PET/CT, and oblique reformats are not supported.
How do I split imaging cases among readers and measure their agreement?
Set how many readers see each case with num_annotators_per_item, and Potato assigns the cases and keeps each reader's masks. The agreement report gives 3D Dice per label for each pair of readers on a case.
Further reading
- Medical image and clinical text annotation tools compared, for 3D Slicer, QuPath, MONAI Label, V7, and Encord beside Potato
- Air-gapped deployment, for what Potato serves locally
- SSO and OAuth, for signing annotators in through your institution's identity provider
- Life sciences and NLP and text, for neighboring fields on this site
- Ontology Term Labels from OBO Files and OLS4