# Fluorescence Microscopy Annotation for OME-Zarr and OME-TIFF Source: https://www.potatoannotator.com/docs/annotation-types/microscopy *New in v2.13.0* The `microscopy` annotation scheme shows a multichannel microscopy image stored as OME-Zarr or OME-TIFF, and records the regions of interest and the cell classes annotators assign on the image. Annotators switch channels on and off, set each channel's color and contrast limits, step through z-slices and time points, and zoom through the file's own resolution pyramid. Cell biologists and imaging cores use the scheme for cell classification on an existing segmentation, such as calling each nucleus marker-positive or marker-negative in an immunofluorescence panel. Screening groups use the same scheme to score phenotypes in [high-content screening](https://en.wikipedia.org/wiki/High-content_screening) fields, and tissue projects mark damaged areas and debris as regions. Several annotators can label the same image, and the export keeps each annotator's regions and classes apart. [OME-Zarr](https://ngff.openmicroscopy.org/) is the chunked, multiscale image format of the Open Microscopy Environment, also known as OME-NGFF, and [OME-TIFF](https://docs.openmicroscopy.org/ome-model/latest/ome-tiff/) is a TIFF file whose OME-XML header names the channels and dimensions. A label image is a segmentation stored as integers, with one id per cell and 0 for background. Segmentation tools such as Cellpose and StarDist write label images. Potato reads a label image produced elsewhere and does not run any segmentation model. - **Formats:** OME-Zarr 0.4 (Zarr v2) and 0.5 (Zarr v3) through zarr-python, including multiscales with axes listed under `coordinateSystems` as in the current NGFF development specification. OME-TIFF through tifffile, also when the pyramid is stored as SubIFDs. - **Axes:** any of `t`, `c`, and `z`, plus `y` and `x`. A file with any other axis is refused with a message. An RGB TIFF with no `C` axis is read with the RGB samples as channels. - **Channels:** names, colors, and display windows come from the OME-Zarr `omero` block or the OME-XML `Channel` elements. A file without a display window starts at the 0.5th and 99.8th percentiles of the coarsest pyramid level, and the slider runs to the smallest bit depth that fits the data (4095 for 12-bit data stored as uint16). Channels without a color take blue, green, red, magenta, cyan, yellow, white, and orange in order. - **Label images:** an OME-Zarr image's `labels/` group is found without configuration. OME-TIFF has no standard place for a label image, and an OME-TIFF item therefore names a separate integer label TIFF. - **Offline:** the viewer loads nothing from outside the Potato server. Remote OME-Zarr over HTTP or S3 is not supported. Potato composites each tile on the server from the visible channels and the channels' contrast limits, and the browser shows the composited JPEG tiles in OpenSeadragon. Server compositing needs no WebGL and sends no raw channel data to the browser, and the cost is one round trip for the visible tiles whenever a channel or contrast setting changes. The readers need the `microscopy` extra: ```bash pip install 'potato-annotation[microscopy]' ``` The extra installs zarr-python 3 on Python 3.11 and later, zarr-python 2 on older Pythons, and tifffile. zarr-python 2 cannot read OME-Zarr 0.5, and OME-Zarr 0.5 stores therefore need Python 3.11 or later. ## Configuration The config below asks three annotators to classify the cells in each image and to outline damaged tissue or debris. The cells come from the image's label image, and annotators draw the regions by hand: ```yaml annotation_task_name: "Cell classification" task_dir: . media_directory: media output_annotation_dir: annotation_output data_files: [data/images.json] item_properties: {id_key: id, text_key: note} num_annotators_per_item: 3 user_config: allow_all_users: true annotation_schemes: - annotation_type: microscopy name: cells description: "Classify the cells, and outline any region that looks damaged." image_key: image # item field naming the OME-Zarr or OME-TIFF labels_key: cells # item field naming a label TIFF (OME-TIFF items) tools: [bbox, polygon, landmark] labels: # classes for regions - {name: damaged, color: "#e8590c"} - {name: debris, color: "#868e96"} cell_labels: # classes for cells - {name: "marker positive", color: "#f03e3e"} - {name: "marker negative", color: "#4dabf7"} - {name: dividing, color: "#fab005"} channels: # overrides of the file's own settings - {name: DAPI, label: "DAPI (nuclei)"} - {index: 2, contrast: [100, 3000]} min_cells: 1 label_requirement: {required: true} ``` `labels` lists the region classes, and `cell_labels` lists the cell classes. Without `cell_labels`, cells take the region classes. The first `channels` entry renames the channel the file calls `DAPI`, and the second entry sets a starting contrast window for the third channel, counted from 0. With `min_cells` and `label_requirement` together, Next stays blocked until the annotator has classified at least one cell. Potato checks every scheme value when the server starts, and a bad value stops the server with a message naming the key. ### Cells-only tasks Phenotype scoring in a screen usually needs cell classes and no regions. The partial block below replaces `annotation_schemes` in the config above: ```yaml annotation_schemes: - annotation_type: microscopy name: phenotype description: "Give every cell in the field a phenotype." tools: [] # no region tools label_image: cells # entry of the OME-Zarr labels/ group labels: [normal, enlarged, multinucleated, apoptotic] channels: - {index: 0, color: "#0000ff"} - {index: 1, visible: false} min_cells: 20 label_requirement: {required: true} ``` With `tools: []`, the region tools are hidden and the class list applies to cells. The `label_image` key picks one entry when an OME-Zarr `labels/` group has several segmentations, for example nuclei and whole cells. The second channel starts hidden, and annotators can still switch the channel on. ## Data format Each item names an image path relative to `media_directory`: ```json {"id": "well_A01", "image": "screen/A01.ome.zarr"} {"id": "tissue_b", "image": "tissue_b.ome.tif", "cells": "tissue_b_cells.tif"} ``` Potato looks for the path inside `media_directory`, then inside `task_dir`, and refuses absolute paths, URLs, and any path that resolves outside both. For an HCS plate or a `bioformats2raw` container, name one image inside the store, such as `plate.ome.zarr/A/1/0`. Potato shows an error for the plate root that suggests the inner path. A label TIFF contains integers, has the image's full-resolution width and height, and uses only the `T`, `Z`, `Y`, and `X` axes the image has. Potato refuses a label TIFF of a different size, because a cell outline drawn from a smaller raster would sit in the wrong place on the image. TIFF planes are decoded whole, and a plane above `max_pixels` is refused. Large images belong in tiled, pyramidal OME-TIFF or in OME-Zarr, where the reader fetches only the chunks a tile covers. ## Options | `microscopy` option | Default | Meaning | |---|---|---| | `labels` | required | Region classes, as names or `{name, color}` with `#rrggbb` colors. Classes without a color take Potato's eight-color palette in order. The list also gives the cell classes when `cell_labels` is absent | | `cell_labels` | `labels` | Cell classes, in the same form. A class's number in the label-array export is the class's position in the list, from 1 | | `image_key` | `image` | The item field holding the image path. A `/media/` prefix on the path is accepted | | `labels_key` | none | The item field holding a separate integer label TIFF, for OME-TIFF items. The label image is then named `cells` | | `label_image` | first entry | The entry of an OME-Zarr `labels/` group to use | | `tools` | `[bbox, polygon, landmark]` | Region tools (rectangle, polygon, and point). `[]` gives a cells-only task | | `channels` | from the file | Per-channel overrides, matched by `index` (from 0) or `name`, with `label` (the display name), `color`, `contrast: [low, high]`, and `visible` | | `min_rois` | `0` | Next stays blocked until the item has this many regions, from 0 to 10,000. Enforced when the scheme is required | | `min_cells` | `0` | Next stays blocked until this many cells have a class, from 0 to 100,000. Enforced when the scheme is required | | `label_requirement` | none | `{required: true}` makes the scheme required | | `height` | `520` | Viewer height in pixels, from 240 to 2000, capped at 70% of the window height and lower on phones | | `tile_size` | `254` | Tile size in pixels, from 64 to 1024 | | `max_pixels` | `64000000` | The largest TIFF plane decoded whole, and the largest label plane read for the export, from 1,000,000 to 4,000,000,000 | | `label_max_side` | `2048` | The label raster sent to the browser is at most this many pixels a side, from 256 to 8192 | | `instructions` | none | Text shown under the description | The browser tests clicks against the label raster, and the export reads cell ids from the same label image. A click therefore picks the id the export names. A label image larger than `label_max_side` is shown from the label image's finest pyramid level that fits, or sampled every few pixels when no level fits, and a cell narrower than the sampling step can be missing from the raster. A label image with more than 50,000 cells is refused for the cell list. ## Classifying cells and drawing regions Annotators pan by dragging and zoom by scrolling or pinching. Each channel row has a checkbox that shows or hides the channel, a color swatch, Min and Max contrast sliders, and a Reset button that restores the file's or the config's settings. Potato keeps channel settings from item to item within a browser session. Z-slice and time-point sliders appear under the image when the image has more than one of either. To classify cells, the annotator picks Pan / select cell (key V) and clicks a cell. The selected cell is outlined in yellow, and a key from 1 to 9 or a class button gives the cell a class. Classified cells fill with the class color. A cell can also be selected by typing the cell's id. A class applies to the cell's id on every z-slice and time point, and the stored `z` and `t` record the plane where the annotator classified the cell. To draw a region, the annotator chooses Rectangle (R), Polygon (P), or Point (O) and a class under "Region class". A rectangle is a drag. A polygon takes one click per vertex and ends with a click on the first vertex, a double-click, Enter, or the "Finish polygon" button. Escape cancels a polygon in progress and Backspace removes the last vertex. A region belongs to the z-slice and time point the region was drawn on and shows only there. Selecting a region in the Regions list moves the viewer to the region's plane, and each list entry has a class menu and a delete button. ## Stored answer Each scheme stores one JSON object per item: ```json {"image": {"width": 256, "height": 256}, "rois": [{"type": "polygon", "label": "damaged", "color": "#e8590c", "coordinates": [{"x": 0.58, "y": 0.08}, {"x": 0.86, "y": 0.12}, {"x": 0.78, "y": 0.35}], "z": 1, "t": 0}], "cells": [{"label_id": 1, "class": "marker positive", "z": 1, "t": 0}], "label_image": "cells"} ``` Region coordinates are fractions of the full-resolution image, in the same form as the `image_annotation` scheme. A `bbox` is `{x, y, width, height}`, a `polygon` is a list of points, and a `landmark` is one point. ## Export The `microscopy` export writes each annotator's answers separately, into the files listed after the command: ```bash python -m potato.export --config config.yaml --format microscopy --output out/ ``` | File | Contents | |---|---| | `microscopy_rois.geojson` | A FeatureCollection of every region in full-resolution pixel coordinates (x right, y down, not georeferenced). Rectangles and polygons are closed Polygons, and points are Points. Each feature has `objectType: "annotation"` and `classification: {name, color: [r, g, b]}` in the form QuPath reads, plus `item_id`, `user_id`, `schema`, `roi_type`, `z`, and `t` | | `microscopy_cells.csv` | One row per classified cell, with `user_id, instance_id, schema, label_image, label_id, class, z, t` | | `ome_zarr/.ome.zarr/labels/__/` | The item's label image with each cell id replaced by the cell's class number (0 for unclassified cells and background), as an OME-Zarr label image with `image-label` colors and a `properties` list naming each class | To open the regions in [QuPath](https://qupath.github.io/), import `microscopy_rois.geojson` with the image open. The GeoJSON contains every item and annotator. Filter the features on `item_id` and `user_id` before the import when several images or annotators share the file. The label arrays are written as OME-Zarr 0.5 when the source image is OME-Zarr 0.5, and as 0.4 otherwise. `--option zarr_format=2` or `--option zarr_format=3` overrides the version, and `--option label_arrays=false` skips the label arrays. Each label-array store has only a `labels/` group, and most viewers will not open the store on its own. Copy `labels/__` into the source image's `labels/` directory and add the name to the `labels` list of the source image's `labels/` group. The label arrays need zarr-python and the source images under `media_directory`. Without zarr-python or the source images, the export writes the GeoJSON and the CSV and lists a warning. Potato's agreement metrics do not cover `microscopy` schemes. Per-cell agreement can be computed from `microscopy_cells.csv` after export, by matching rows on `instance_id`, `label_image`, and `label_id` and computing [Fleiss' kappa](https://en.wikipedia.org/wiki/Fleiss%27_kappa) or [Cohen's kappa](https://en.wikipedia.org/wiki/Cohen%27s_kappa) over the matched classes. ## Whole-slide pathology images Brightfield whole-slide scans in SVS, NDPI, MRXS, and the other slide formats openslide reads use the `image_annotation` scheme in the [deep zoom viewer](/docs/vision-spatial/deep-zoom), with the `wsi` extra. The `microscopy` scheme does not open SVS, NDPI, or MRXS files. From Potato 2.13, regions on whole-slide images import from QuPath GeoJSON or ASAP XML with `potato import -i case_01.geojson --image-dir /data/slides -o case-project`. The importer detects the format, and `-f qupath_geojson` or `-f asap_xml` names the format explicitly. The imported regions become pre-annotations. Annotators' regions export back to QuPath GeoJSON in level-0 slide pixels with `python -m potato.export -c config.yaml -f qupath_geojson -o out/qupath`, and `-f asap_xml` writes ASAP XML instead. ## Example ```bash python potato/flask_server.py start examples/image/multichannel-microscopy/config.yaml -p 8000 ``` The example has two synthetic images. One image is an OME-Zarr 0.5 store with three channels, three z-slices, and a `labels/cells` group, and the other is an OME-TIFF with three channels, three time points, and a separate label TIFF. `generate_data.py` in the same directory writes both images. ## Setting up a study with a coding agent [potato-skill](/docs/getting-started/quick-start-with-a-coding-agent) 1.2.0 targets Potato 2.13.1 and knows `microscopy` by name, and the skill guides the study-design side, such as what to ask annotators, how many annotators classify each image, training and attention checks, and export. The skill does not yet document the OME-Zarr and OME-TIFF data formats, the scheme options, or the `microscopy` extra. Install potato-skill, describe the study to your agent, and point the agent at the Markdown copy of this page, `https://www.potatoannotator.com/docs/annotation-types/microscopy.md`, for the formats and options. ## Further Reading - [Image Annotation](/docs/annotation-types/image-annotation) - Boxes, polygons, and masks on ordinary images - [Gigapixel Image Annotation with Deep Zoom](/docs/vision-spatial/deep-zoom) - Tile pyramids and whole-slide images - [Export Formats](/docs/features/export-formats) - Potato's export formats and the export command - [How Many Annotators Do You Need?](/docs/guides/how-many-annotators) - Choosing annotators per item - [Life Sciences](/use-cases/life-sciences) - Potato for life sciences and biocuration For implementation details, see the [source documentation](https://github.com/davidjurgens/potato/blob/master/docs/annotation-types/science/microscopy.md).