You can use Potato's image annotation for medical images such as X-rays, photographs of skin lesions, or other clinical images, as long as they are in a standard format like PNG or JPG. This guide covers how to set up that kind of project well.

For the underlying image annotation mechanics, see the source documentation.

Before you start

A medical annotation project comes with obligations that have nothing to do with Potato. Strip all PHI from the images first. Keep the data on your institution's secure infrastructure for storage and processing. Gate annotator access through whatever authentication you already run. And get your IRB and compliance teams involved early so you meet the institutional requirements rather than discovering them later.

One thing to be clear about: Potato is an annotation tool, not a medical imaging platform. It has no built-in HIPAA compliance, no DICOM viewer, and no specialized imaging features. Handle those at the infrastructure level.

Basic medical image annotation setup

Here is how to configure Potato for standard medical images:

yaml

annotation_task_name: "Medical Image Annotation"
 
data_files:
  - path: data/medical_images.json
    type: json
 
annotation_schemes:
  - annotation_type: bounding_box
    name: lesions
    allow_multiple: true
    labels:
      - name: primary
        color: "#EF4444"
        description: "Primary finding"
 
      - name: secondary
        color: "#F97316"
        description: "Secondary finding"
 
      - name: artifact
        color: "#6B7280"
        description: "Artifact or noise"
 
  - annotation_type: radio
    name: image_quality
    question: "Overall image quality?"
    options:
      - name: diagnostic
        label: "Diagnostic Quality"
      - name: limited
        label: "Limited Quality"
      - name: non_diagnostic
        label: "Non-Diagnostic"

X-ray and clinical photo annotation

For X-rays or clinical photographs that have been exported to standard image formats:

yaml

annotation_task_name: "X-Ray Findings Annotation"
 
data_files:
  - path: data/xray_images.json
    type: json
 
annotation_schemes:
  - annotation_type: polygon
    name: abnormalities
    allow_multiple: true
    labels:
      - name: opacity
        color: "#EF4444"
        description: "Pulmonary opacity"
 
      - name: consolidation
        color: "#F97316"
        description: "Consolidation"
 
      - name: nodule
        color: "#EAB308"
        description: "Pulmonary nodule"
 
      - name: effusion
        color: "#3B82F6"
        description: "Pleural effusion"
 
  - annotation_type: multiselect
    name: findings
    question: "Select all findings present"
    options:
      - name: normal
        label: "No acute findings"
      - name: pneumonia
        label: "Pneumonia"
      - name: atelectasis
        label: "Atelectasis"
      - name: pneumothorax
        label: "Pneumothorax"
      - name: fracture
        label: "Fracture"

Skin lesion annotation

For dermatology images:

yaml

annotation_task_name: "Dermoscopy Annotation"
 
data_files:
  - path: data/skin_lesions.json
    type: json
 
annotation_schemes:
  - annotation_type: polygon
    name: lesion_boundary
    labels:
      - name: lesion
        color: "#EF4444"
        description: "Lesion boundary"
 
  - annotation_type: multiselect
    name: dermoscopic_features
    question: "Select all features present"
    options:
      - name: pigment_network
        label: "Pigment Network"
      - name: dots_globules
        label: "Dots/Globules"
      - name: streaks
        label: "Streaks"
      - name: blue_white_veil
        label: "Blue-White Veil"
      - name: regression
        label: "Regression Structures"
      - name: vascular
        label: "Vascular Structures"
 
  - annotation_type: radio
    name: diagnosis
    question: "Most likely diagnosis?"
    options:
      - name: benign_nevus
        label: "Benign Nevus"
      - name: seborrheic_keratosis
        label: "Seborrheic Keratosis"
      - name: basal_cell
        label: "Basal Cell Carcinoma"
      - name: melanoma
        label: "Melanoma"
      - name: other
        label: "Other"
 
  - annotation_type: likert
    name: confidence
    question: "Diagnostic confidence"
    size: 5
    min_label: "Low"
    max_label: "High"

Retinal image annotation

For fundus photographs and other retinal images:

yaml

annotation_task_name: "Fundus Image Annotation"
 
data_files:
  - path: data/fundus_images.json
    type: json
 
annotation_schemes:
  - annotation_type: polygon
    name: anatomical_structures
    labels:
      - name: optic_disc
        color: "#FDE68A"
        description: "Optic disc boundary"
 
      - name: fovea
        color: "#A78BFA"
        description: "Fovea region"
 
      - name: macula
        color: "#93C5FD"
        description: "Macular region"
 
  - annotation_type: polygon
    name: pathology
    allow_multiple: true
    labels:
      - name: hemorrhage
        color: "#EF4444"
        description: "Retinal hemorrhage"
 
      - name: exudate
        color: "#FCD34D"
        description: "Hard/soft exudate"
 
      - name: microaneurysm
        color: "#F97316"
        description: "Microaneurysm"
 
  - annotation_type: radio
    name: dr_grade
    question: "Diabetic retinopathy grade"
    options:
      - name: none
        label: "No DR"
      - name: mild
        label: "Mild NPDR"
      - name: moderate
        label: "Moderate NPDR"
      - name: severe
        label: "Severe NPDR"
      - name: proliferative
        label: "PDR"

Data format

Prepare your image data in JSON:

json

[
  {
    "id": "case_001",
    "image": "images/case_001.png",
    "metadata": {
      "body_part": "chest",
      "modality": "xray"
    }
  },
  {
    "id": "case_002",
    "image": "images/case_002.png",
    "metadata": {
      "body_part": "chest",
      "modality": "xray"
    }
  }
]

Note: Images must be in standard web-compatible formats (PNG, JPG, etc.). DICOM files need to be converted to standard image formats before use with Potato.

Best practices

Use annotators with real clinical training, since this is not work you can hand to anyone. Preprocess the images first: convert DICOM and other specialized formats to standard ones, and confirm the PHI is gone. Have more than one annotator label each image so you can measure agreement. Write guidelines that are specific to your clinical domain rather than generic. Salt in gold-standard cases to keep an eye on accuracy. And keep your compliance team in the loop on how the data is handled.

Limitations

Potato does general-purpose image annotation. It does not give you native DICOM support or a DICOM viewer, whole slide image viewers with multi-resolution zoom, built-in HIPAA compliance or audit logging, credential verification, or radiology windowing tools. If you need those, preprocess your data or put Potato alongside dedicated medical imaging infrastructure.

For more on image annotation, see /docs/features/image-annotation.