Fluorescence Microscopy Annotation for OME-Zarr and OME-TIFF

New in v2.13.0

The microscopy annotation scheme shows a multichannel microscopy image stored as OME-Zarr or OME-TIFF, and records the regions of interest and the cell classes annotators assign on the image. Annotators switch channels on and off, set each channel's color and contrast limits, step through z-slices and time points, and zoom through the file's own resolution pyramid. Cell biologists and imaging cores use the scheme for cell classification on an existing segmentation. In an immunofluorescence panel, for example, annotators call each nucleus marker-positive or marker-negative. Screening groups use the same scheme to score phenotypes in high-content screening fields, and tissue projects mark damaged areas and debris as regions. Several annotators can label the same image, and the export keeps each annotator's regions and classes apart.

OME-Zarr (also called OME-NGFF) is the chunked, multiscale image format of the Open Microscopy Environment. OME-TIFF is a TIFF file whose OME-XML header names the channels and dimensions. A label image is a segmentation stored as integers, with one id per cell and 0 for background. Segmentation tools such as Cellpose and StarDist write label images. Potato reads a label image produced elsewhere and does not run any segmentation model.

• Formats: Potato reads OME-Zarr 0.4 (Zarr v2) and 0.5 (Zarr v3) through zarr-python, including multiscales with axes listed under coordinateSystems as in the current NGFF development specification. Potato reads OME-TIFF through tifffile, and tifffile also reads pyramids stored as SubIFDs.
• Axes: an image may have any of the t, c, and z axes, plus y and x. A file with any other axis is refused with a message. An RGB TIFF with no C axis is read with the RGB samples as channels.
• Channels: names, colors, and display windows come from the OME-Zarr omero block or the OME-XML Channel elements. A file without a display window starts at the 0.5th and 99.8th percentiles of the coarsest pyramid level, and the slider runs to the smallest bit depth that fits the data (4095 for 12-bit data stored as uint16). Channels without a color take blue, green, red, magenta, cyan, yellow, white, and orange in order.
• Label images: an OME-Zarr image's labels/ group is found without configuration. OME-TIFF has no standard place for a label image. An OME-TIFF item therefore names a separate integer label TIFF.
• Offline: the viewer loads nothing from outside the Potato server. Remote OME-Zarr over HTTP or S3 is not supported.

Potato composites each tile on the server from the visible channels and the channels' contrast limits. The browser shows the composited JPEG tiles in OpenSeadragon, needs no WebGL, and receives no raw channel data. Whenever an annotator changes a channel or contrast setting, the browser fetches the visible tiles again in one round trip.

The readers need the microscopy extra:

pip install 'potato-annotation[microscopy]'

The extra installs zarr-python 3 on Python 3.11 and later, zarr-python 2 on older Pythons, and tifffile. Because zarr-python 2 cannot read OME-Zarr 0.5, OME-Zarr 0.5 stores need Python 3.11 or later.

Configuration

The config below asks three annotators to classify the cells in each image and to outline damaged tissue or debris. The cells come from the image's label image, and annotators draw the regions by hand:

annotation_task_name: "Cell classification"
task_dir: .
media_directory: media
output_annotation_dir: annotation_output
data_files: [data/images.json]
item_properties: {id_key: id, text_key: note}
num_annotators_per_item: 3
 
user_config:
  allow_all_users: true
 
annotation_schemes:
  - annotation_type: microscopy
    name: cells
    description: "Classify the cells, and outline any region that looks damaged."
    image_key: image              # item field naming the OME-Zarr or OME-TIFF
    labels_key: cells             # item field naming a label TIFF (OME-TIFF items)
    tools: [bbox, polygon, landmark]
    labels:                       # classes for regions
      - {name: damaged, color: "#e8590c"}
      - {name: debris, color: "#868e96"}
    cell_labels:                  # classes for cells
      - {name: "marker positive", color: "#f03e3e"}
      - {name: "marker negative", color: "#4dabf7"}
      - {name: dividing, color: "#fab005"}
    channels:                     # overrides of the file's own settings
      - {name: DAPI, label: "DAPI (nuclei)"}
      - {index: 2, contrast: [100, 3000]}
    min_cells: 1
    label_requirement: {required: true}

labels lists the region classes, and cell_labels lists the cell classes. Without cell_labels, cells take the region classes. The first channels entry renames the channel the file calls DAPI, and the second entry sets a starting contrast window for the third channel (indexes count from 0). With min_cells and label_requirement together, Next stays blocked until the annotator has classified at least one cell. Potato checks every scheme value when the server starts and stops the server with a message naming the key of any bad value.

Cells-only tasks

Phenotype scoring in a screen usually needs cell classes and no regions. The partial block below replaces annotation_schemes in the config above:

annotation_schemes:
  - annotation_type: microscopy
    name: phenotype
    description: "Give every cell in the field a phenotype."
    tools: []                     # no region tools
    label_image: cells            # entry of the OME-Zarr labels/ group
    labels: [normal, enlarged, multinucleated, apoptotic]
    channels:
      - {index: 0, color: "#0000ff"}
      - {index: 1, visible: false}
    min_cells: 20
    label_requirement: {required: true}

With tools: [], the region tools are hidden and the class list applies to cells. The label_image key picks one entry when an OME-Zarr labels/ group has several segmentations (e.g., nuclei and whole cells). The second channel starts hidden, and annotators can still switch the channel on.

Data format

Each item names an image path relative to media_directory:

{"id": "well_A01", "image": "screen/A01.ome.zarr"}
{"id": "tissue_b", "image": "tissue_b.ome.tif", "cells": "tissue_b_cells.tif"}

Potato looks for the path inside media_directory, then inside task_dir, and refuses absolute paths, URLs, and any path that resolves outside both directories. For an HCS plate or a bioformats2raw container, name one image inside the store, such as plate.ome.zarr/A/1/0. Potato shows an error for the plate root that suggests the inner path.

A label TIFF contains integers, has the image's full-resolution width and height, and uses only the T, Z, Y, and X axes the image has. Potato refuses a label TIFF of a different size because a cell outline drawn from a smaller raster would sit in the wrong place on the image. TIFF planes are decoded whole, and a plane above max_pixels is refused. Store large images as tiled, pyramidal OME-TIFF or as OME-Zarr. For those formats, the reader fetches only the chunks a tile covers.

Options

The browser tests clicks against the label raster, and the export reads cell ids from the same label image. A click therefore picks the id the export names. A label image larger than label_max_side is shown from the label image's finest pyramid level that fits, or sampled every few pixels when no level fits. A cell narrower than the sampling step can then be missing from the raster. A label image with more than 50,000 cells is refused for the cell list.

Classifying cells and drawing regions

Annotators pan by dragging and zoom by scrolling or pinching. Each channel row has a checkbox that shows or hides the channel, a color swatch, Min and Max contrast sliders, and a Reset button that restores the file's or the config's settings. Potato keeps channel settings from item to item within a browser session. Z-slice and time-point sliders appear under the image when the image has more than one of either.

To classify cells, the annotator picks Pan / select cell (key V) and clicks a cell. The selected cell is outlined in yellow, and a key from 1 to 9 or a class button gives the cell a class. Classified cells fill with the class color. A cell can also be selected by typing the cell's id. A class applies to the cell's id on every z-slice and time point, and the stored z and t record the plane where the annotator classified the cell.

To draw a region, the annotator chooses Rectangle (R), Polygon (P), or Point (O) and a class under "Region class". A rectangle is a drag. A polygon takes one click per vertex and ends with a click on the first vertex, a double-click, Enter, or the "Finish polygon" button. Escape cancels a polygon in progress and Backspace removes the last vertex. A region belongs to the z-slice and time point the region was drawn on and shows only there. Selecting a region in the Regions list moves the viewer to the region's plane, and each list entry has a class menu and a delete button.

Stored answer

Each scheme stores one JSON object per item:

{"image": {"width": 256, "height": 256},
 "rois": [{"type": "polygon", "label": "damaged", "color": "#e8590c",
           "coordinates": [{"x": 0.58, "y": 0.08}, {"x": 0.86, "y": 0.12}, {"x": 0.78, "y": 0.35}],
           "z": 1, "t": 0}],
 "cells": [{"label_id": 1, "class": "marker positive", "z": 1, "t": 0}],
 "label_image": "cells"}

Region coordinates are fractions of the full-resolution image, in the same form as the image_annotation scheme. A bbox is {x, y, width, height}, a polygon is a list of points, and a landmark is one point.

Export

The microscopy export writes each annotator's answers separately, into the files listed after the command:

python -m potato.export --config config.yaml --format microscopy --output out/

To open the regions in QuPath, import microscopy_rois.geojson with the image open. The GeoJSON contains every item and annotator. Filter the features on item_id and user_id before the import when several images or annotators share the file.

The label arrays are written as OME-Zarr 0.5 when the source image is OME-Zarr 0.5, and as 0.4 otherwise. --option zarr_format=2 or --option zarr_format=3 overrides the version, and --option label_arrays=false skips the label arrays. Each label-array store has only a labels/ group, and most viewers will not open the store on its own. Copy labels/<schema>__<user> into the source image's labels/ directory and add the name to the labels list of the source image's labels/ group. The label arrays need zarr-python and the source images under media_directory. Without zarr-python or the source images, the export writes the GeoJSON and the CSV and lists a warning.

Potato's agreement metrics do not cover microscopy schemes. To compute per-cell agreement after export, match rows of microscopy_cells.csv on instance_id, label_image, and label_id. Then compute Fleiss' kappa or Cohen's kappa over the matched classes.

Whole-slide pathology images

For brightfield whole-slide scans in SVS, NDPI, MRXS, and the other slide formats openslide reads, use the image_annotation scheme in the deep zoom viewer with the wsi extra. The microscopy scheme does not open SVS, NDPI, or MRXS files. From Potato 2.13, regions on whole-slide images import from QuPath GeoJSON or ASAP XML with potato import -i case_01.geojson --image-dir /data/slides -o case-project. The importer detects the format, and -f qupath_geojson or -f asap_xml names the format explicitly. The imported regions become pre-annotations. Annotators' regions export back to QuPath GeoJSON in level-0 slide pixels with python -m potato.export -c config.yaml -f qupath_geojson -o out/qupath, and -f asap_xml writes ASAP XML instead.

Example

python potato/flask_server.py start examples/image/multichannel-microscopy/config.yaml -p 8000

The example has two synthetic images. One image is an OME-Zarr 0.5 store with three channels, three z-slices, and a labels/cells group, and the other is an OME-TIFF with three channels, three time points, and a separate label TIFF. generate_data.py in the same directory writes both images.

Setting up a study with a coding agent

potato-skill 1.2.0 targets Potato 2.13.1 and knows microscopy by name. The skill guides the study-design side, including what to ask annotators, how many annotators classify each image, training and attention checks, and export. The skill does not yet document the OME-Zarr and OME-TIFF data formats, the scheme options, or the microscopy extra. Install potato-skill, describe the study to your agent, and point the agent at the Markdown copy of this page for the formats and options. The Markdown copy is at https://www.potatoannotator.com/docs/annotation-types/microscopy.md.

Further reading

• Image Annotation - Boxes, polygons, and masks on ordinary images
• Gigapixel Image Annotation with Deep Zoom - Tile pyramids and whole-slide images
• Export Formats - Potato's export formats and the export command
• How Many Annotators Do You Need? - Choosing annotators per item
• Life Sciences - Potato for life sciences and biocuration

For implementation details, see the source documentation.