# Train Models on Molecules, Spectra, Signals, and Robot Episodes

Source: https://www.potatoannotator.com/docs/model-training/science-models

*New in v2.14.0*

This tutorial trains four models on scientific annotations in a Potato study. `rdkit-property` answers a question about a molecule, `peak-cnn` picks peaks on NMR, IR, Raman, UV-Vis, and mass spectra, `signal-cnn` stages sleep epochs or marks labeled intervals on EEG, ECG, and sensor recordings, and `episode-model` marks the phases, the outcome, and the reward curve of a robot episode. The page is for teams who already annotate molecules, spectra, signals, or robot episodes in Potato and want a model to pre-label the remaining items. For each trainer the page gives a config that passes `potato validate config.yaml --strict`, what a run learns, the held-out score to read, and how the predictions reach annotators. [Train your first model](/docs/model-training/first-model) walks through the Training page with a text classifier, and this tutorial assumes that walkthrough.

## The four trainers

Each trainer learns from the answers of one annotation scheme and reads the model's input from the display or the file the scheme points at:

| Trainer | Scheme | Input | Install | Held-out headline |
|---|---|---|---|---|
| `rdkit-property` | `radio`, `select`, `likert`, or `slider` about a molecule | a `molecule` display field, or a `chemical_structure` answer | `potato-annotation[chemistry]` | macro-F1, or exact-position accuracy for a slider |
| `peak-cnn` | `peak_picking` | spectra from JCAMP-DX, mzML, CSV, or inline data | `potato-annotation[train-vision]` | peak F1 within a tolerance |
| `signal-cnn` | `interval_label`, in epoch or free mode | signals from EDF, BDF, WFDB, CSV, Parquet, or inline data | `potato-annotation[train-vision]`, plus `[signals]` for WFDB and Parquet | macro-F1 over epochs, or interval F1 at IoU 0.5 |
| `episode-model` | `episode_annotation` | robot episodes from LeRobot, HDF5, or Potato's episode manifest | `potato-annotation[train-vision]` | phase macro-F1 |

`rdkit-property` fits a scikit-learn model, and the other three trainers fit small PyTorch networks on the device that `model_training.device` selects. Every run trains on one scheme, scores the model on the run's held-out `val` and `test` items, and writes predictions back only for the schemes listed under `writeback.schemas`. Items in a run's train split never get a pre-label. `peak-cnn`, `signal-cnn`, and `episode-model` keep the checkpoint from the epoch that scored best on `val`, so Potato takes the held-out score of the three networks from `test`. A small study has few test items, and the run's `test_n` metric gives the count behind the score.

The three networks give an item's prediction the confidence of the prediction's least certain part, i.e., the weakest peak, interval, phase, or outcome. A `writeback.min_confidence` floor therefore withholds the whole item when any one part falls below the floor. A slider prediction from `rdkit-property` has no confidence value and always passes the floor.

## Molecular properties with `rdkit-property`

`rdkit-property` learns the answers to a question about the molecule an item shows, and the molecule's structure is the model's input rather than the item's text. Each structure becomes a Morgan fingerprint of radius 2 and 2,048 bits followed by [RDKit](https://www.rdkit.org/)'s 2D descriptors, e.g., molecular weight and logP. At radius 2 the Morgan fingerprint matches ECFP4, the [extended-connectivity fingerprint](https://doi.org/10.1021/ci100050t) with a diameter of 4 bonds. Logistic regression fits a `radio`, `select`, or `likert` answer by default, and ridge regression fits a `slider`. A random forest is the alternative for both. Install RDKit on the server that trains:

```bash
pip install "potato-annotation[chemistry]"
```

The config below shows each compound as a structure drawing and asks two questions, a 1-to-10 slider for how hard the compound would be to make and a yes-or-no choice about making the compound next:

```yaml
annotation_task_name: "Compound triage"
task_dir: .
output_annotation_dir: annotation_output
data_files: [data/molecules.jsonl]
item_properties: {id_key: id, text_key: note}
user_config:
  allow_all_users: true

instance_display:
  fields:
    - key: smiles                 # SMILES, a molfile, or a .mol, .sdf or .smi file
      type: molecule
      label: "Compound"

annotation_schemes:
  - annotation_type: slider
    name: synthesis
    description: "How hard would this compound be to make? 1 is one step from a catalog reagent, 10 is a research project."
    min_value: 1
    max_value: 10
    starting_value: 5
    step: 1
  - annotation_type: radio
    name: advance
    description: "Would you make this compound next?"
    labels: ["yes", "no"]

model_training:
  enabled: true
  writeback:
    schemas: [synthesis, advance]
    include_unlabeled: true
    min_confidence: 0.6
```

Each line of the data file has one compound:

```json
{"id": "cpd_023", "smiles": "CC(=O)Nc1ccc(O)cc1", "note": "Compound 23"}
```

The trainer takes the structure from the first `molecule` field in `instance_display`. A study without a `molecule` field can ask the question beside a `chemical_structure` scheme instead, and the trainer then learns from the structure annotators drew. When annotators drew different structures, the trainer takes the most common drawing, and when nobody drew, the trainer takes the structure the editor started from (`prefill_field`). The Training page offers `rdkit-property` for a `molecule` field once at least 80% of the items fill the field. A structure RDKit cannot parse is left out of training, gets no prediction, and is counted in the run's `n_unparseable` metric. For a choice or Likert question, the Training page also offers the text trainers, because the items still have text.

A slider's prediction is rounded to the slider's `step` and kept between `min_value` and `max_value`, so the model always proposes a position the slider has. The headline score for a slider is exact-position accuracy, which counts a prediction of 6 against an answer of 7 as wrong. Read `val_mae` or `test_mae`, the mean absolute error, beside the accuracy before deciding whether the predictions are close enough to show. The run needs at least 10 labeled molecules in the train split. Parameters such as `model`, `class_weight`, `radius`, and `bits` go in the Parameters field of the Training page or in the API request:

```bash
curl -X POST -H "X-API-Key: $KEY" -H "Content-Type: application/json" \
  http://localhost:8000/admin/training/api/runs \
  -d '{"trainer": "rdkit-property", "schemas": ["advance"],
       "params": {"model": "random_forest", "class_weight": "balanced"}}'
```

The run's log names the molecule count, the feature count, and the model. A balanced class weight counts rare answers more heavily. Use `class_weight: balanced` when most compounds get the same answer. Once the run finishes, an annotator who opens an unlabeled compound finds the slider set to the model's value and the model's choice selected, both editable.

## Peaks on spectra with `peak-cnn`

`peak-cnn` learns to pick peaks from the `peak_picking` answers annotators gave. The trainer reads the spectrum the scheme's `viewer_field` shows, with the same readers the [spectrum display](/docs/annotation-types/spectra) uses. The training target is every peak the annotator kept, accepted suggestions and added peaks alike. A suggestion the annotator rejected marks a place with no peak. The model predicts peak positions only, and annotators still give each peak an assignment, a label, or a multiplicity. Install PyTorch on the machine that trains:

```bash
pip install "potato-annotation[train-vision]"
```

The config below asks annotators to review suggested peaks on 1H NMR spectra:

```yaml
annotation_task_name: "1H NMR peak picking"
task_dir: .
media_directory: media
output_annotation_dir: annotation_output
data_files: [data/spectra.jsonl]
item_properties: {id_key: id, text_key: note}
user_config:
  allow_all_users: true

instance_display:
  fields:
    - key: spectrum
      type: spectrum
      label: "1H NMR"
      display_options:
        prepick:
          method: auto            # item peaks, then model, then file peak table, then detector
          prominence: 0.05
          max_peaks: 30

annotation_schemes:
  - annotation_type: peak_picking
    name: peaks
    description: "Accept or reject each suggested peak, and add any the suggestions missed"
    viewer_field: spectrum
    peak_fields:
      - name: assignment
        placeholder: "e.g. OCH2"
    require_review: true
    label_requirement: {required: true}

model_training:
  enabled: true
  trainer_params:
    peak-cnn:
      tolerance: 0.02             # ppm
  writeback:
    schemas: [peaks]
    include_unlabeled: true
```

The model's peaks reach annotators only through the display's `prepick` setting, and without `prepick` the display shows no suggestions at all. Under `method: auto`, the display offers the item's own `peaks` first, then the model's peaks, then the file's peak table, and then the peaks the detector finds. Before the first run, annotators review the detector's suggestions, and after a run the same items show the model's peaks without a restart. With `method: model`, the display shows only the model's peaks. Use `method: model` to measure how often annotators accept the model's peaks without the detector's suggestions mixed in. Each model peak is listed with "· model", the display keeps the `max_peaks` most confident, and an accepted model peak is stored with `source: model`.

The `tolerance` parameter sets how far, in x units, a predicted peak may sit from an annotated peak and still count as found. Without `tolerance`, the run uses 0.5% of the spectrum's x range, and 0.5% of a 12 ppm 1H window is 0.06 ppm. At 400 MHz, a 7 Hz coupling separates the lines of a multiplet by 0.0175 ppm. The default tolerance is wider than that spacing, and the config sets 0.02 ppm instead. The run scores precision, recall, and F1 over peaks matched one to one, nearest first, and reports the mean offset of the matched peaks. A peak predicted on a spectrum the annotator said has no peaks counts as a false alarm. The run needs at least 5 annotated spectra in the train split.

Each spectrum is resampled to `length` points, 4,096 by default, before the network sees the spectrum. Raise `length` when peaks are narrower than 1/`length` of the x range, e.g., a high-resolution mass spectrum. Reading mzML files also needs the `spectra` extra, as the [spectrum display page](/docs/annotation-types/spectra) describes.

## Sleep stages and signal intervals with `signal-cnn`

`signal-cnn` learns from `interval_label` answers on physiological and sensor recordings such as EEG, ECG, EMG, and accelerometer data. The network is a small [InceptionTime](https://doi.org/10.1007/s10618-020-00710-y) model, a stack of parallel convolutions of three widths. The trainer reads each recording with the readers of the [time-series display](/docs/annotation-types/time-series), takes the channels the display names, resamples each channel to `rate` (100 Hz by default), and standardizes each channel. Install PyTorch, and the `signals` extra for WFDB and Parquet recordings:

```bash
pip install "potato-annotation[train-vision,signals]"
```

The config below scores sleep in 30-second epochs on overnight polysomnography, the way a [human scorer](https://en.wikipedia.org/wiki/Polysomnography) labels each epoch as wake, N1, N2, N3, or REM:

```yaml
annotation_task_name: "Sleep staging"
task_dir: .
media_directory: media
output_annotation_dir: annotation_output
data_files: [data/psg.jsonl]
item_properties: {id_key: id, text_key: note}
user_config:
  allow_all_users: true

instance_display:
  fields:
    - key: signal
      type: time_series
      label: "PSG"
      display_options:
        window: 30
        channels: ["EEG Fpz-Cz", "EOG", "EMG chin"]   # also the channels signal-cnn reads

annotation_schemes:
  - annotation_type: interval_label
    name: stage
    description: "Sleep stage of each 30-second epoch"
    signal_field: signal
    mode: epoch
    epoch_length: 30
    require_all_epochs: true
    label_requirement: {required: true}
    labels:
      - {name: W, key_value: "1"}
      - {name: N1, key_value: "2"}
      - {name: N2, key_value: "3"}
      - {name: N3, key_value: "4"}
      - {name: REM, key_value: "5"}

model_training:
  enabled: true
  writeback:
    schemas: [stage]
    include_unlabeled: true
```

The scheme's `mode` sets the task. In epoch mode, each annotated epoch is one training example, and the model labels each epoch of a new recording. A last epoch with less than half of the epoch's length recorded gets no label. The held-out headline is macro-F1 over the annotated epochs. Macro-F1 gives a rare stage such as N1 the same weight as N2. By default the training loss weights each stage by inverse label frequency (`balanced: true`), and an N1 epoch then counts for more than an N2 epoch. Every recording is read with the channels of the first training recording, in that order, and a recording that lacks one of the training channels gets a flat line in place of the missing channel. A display that names no channels gives the trainer every channel, up to `max_channels` (16).

Free mode trains on marked stretches instead. Add a second `interval_label` scheme with `mode: free` to the same `signal_field` to mark artifacts or arousals:

```yaml
  - annotation_type: interval_label
    name: artifacts
    description: "Mark artifacts"
    signal_field: signal
    mode: free
    labels:
      - {name: artifact, key_value: "a"}
      - {name: spike, key_value: "s", kind: event}
```

In free mode the network trains on random 30-second crops (`window_s`), labels each 1-second frame (`frame_s`), merges consecutive frames with one label into an interval, and drops intervals shorter than `min_duration`. The headline is interval F1, which matches a predicted interval to an annotated one with the same label at IoU 0.5. Point events such as the `spike` label above are neither trained nor predicted. The run needs at least 3 annotated recordings in the train split, and epoch mode also needs at least two different labels in the data.

The model's intervals appear on the recording as dashed pre-annotations, listed with "· from the model", and are saved when the annotator makes a first change or clicks "Keep these". A recording's own annotations take precedence. An EDF+ file that contains a hypnogram shows the hypnogram and not the model's intervals, and the same goes for an item with `preannotation_field` intervals. Setting `pre_annotations: false` on the scheme turns off both kinds.

## Robot episodes with `episode-model`

`episode-model` learns the three layers of an `episode_annotation` answer, i.e., the phases, the outcome, and the progress reward curve. The model's input is every numeric series the episode records, such as joint positions and gripper state, with one row per frame. The input includes series that `series_shown` leaves off the screen and excludes the video. The model cannot find a phase that shows only on camera and leaves the robot's measurements unchanged. Install PyTorch, plus pyarrow for LeRobot datasets or h5py for HDF5 demos:

```bash
pip install "potato-annotation[train-vision]"
```

The config below collects all three layers on pick-and-place demonstrations, as the [robot episode page](/docs/vision-spatial/robot-episodes) describes:

```yaml
annotation_task_name: "Robot episode review"
task_dir: .
media_directory: media
output_annotation_dir: annotation_output
data_files: [data/episodes.jsonl]
item_properties: {id_key: id, text_key: note}
user_config:
  allow_all_users: true

annotation_schemes:
  - annotation_type: episode_annotation
    name: episode_review
    description: "Mark the phases, judge the outcome, and draw the progress."
    source_field: episode
    layers: [phases, outcome, reward]
    phases:
      - {name: reach, key_value: "1"}
      - {name: grasp, key_value: "2"}
      - {name: transport, key_value: "3"}
      - {name: place, key_value: "4"}
      - {name: retract, key_value: "5"}
    outcomes: [success, partial, failure]
    failure_causes: [missed grasp, object slipped, collision]
    reward_range: [0.0, 1.0]
    series_shown: [gripper, wrist_force, shoulder_lift, elbow]

model_training:
  enabled: true
  writeback:
    schemas: [episode_review]
    include_unlabeled: true
```

The network runs a convolution over the series into a bidirectional GRU with three heads, one for the phase of each frame, one for the outcome of the episode, and one for the reward at each frame. A layer an answer leaves empty is left out of that episode's loss, so an episode marked only with an outcome still trains the outcome head. The outcome head trains once the training episodes include at least two different outcomes. The reward head trains only on the stretch where the annotator drew the curve, because outside that stretch the annotator gave no answer. Episodes longer than `max_frames` (2,000) are trained on random crops.

The prediction has the shape of a stored answer, with phases from the scheme's list, one of the scheme's outcomes, and a reward curve of 50 points inside `reward_range`, for the layers the scheme shows. The prediction pre-fills the timeline and is stored once the annotator edits the timeline. The failure cause and the instruction are left to the annotator. The held-out headline is phase macro-F1 on a 0.1-second grid. The run also reports phase segment F1 at IoU 0.5, outcome accuracy, and Pearson's r of the reward curve where a curve was drawn. A scheme without phases takes outcome accuracy as the headline, and then reward correlation. The run needs at least 5 annotated episodes in the train split. Potato has no trainer for `rollout_evaluation`, because a rollout preference compares videos and `episode-model` reads no video.

## Start a run and check the result

Restart the server after adding `model_training`:

```bash
potato start config.yaml -p 8000
```

Open `/admin/training`, pick the scheme and the trainer, and click **Train now**. The page offers only the trainers that fit each scheme, and a trainer that is not installed is listed as "not installed", with the install command. When the run finishes, Potato lists the held-out score in the Runs table, with the split and the metric in a tooltip. The run page lists every metric and the split of each item. The write-back report in `GET /admin/training/api/status` counts the predictions published and the ones each check held back, e.g., `n_skipped_low_confidence`.

Pre-labels anchor annotators to the model's answer, so turn on `include_unlabeled` in a new study only once the held-out score supports showing the model's answers. [Review model pre-labels in the loop](/docs/model-training/model-in-the-loop) covers serving the least-confident pre-labels first and retraining on corrections, and [Configuration](/docs/model-training/configuration) lists every key, the backends, and the write-back checks.

## Further reading

- [Model training overview](/docs/model-training/overview)
- [All trainers](/docs/model-training/trainers)
- [Chemical structures](/docs/annotation-types/chemical-structures), [spectra](/docs/annotation-types/spectra), and [time series](/docs/annotation-types/time-series)
- [Robot episode annotation](/docs/vision-spatial/robot-episodes) and the [robot episode guide](/docs/guides/robot-episode-annotation)
- [Potato for chemistry](/use-cases/chemistry), [physiology](/use-cases/physiology), and [robotics](/use-cases/robotics)
- [Model-assisted labeling](/docs/guides/model-assisted-labeling)
- Upstream reference for [`rdkit-property`](https://github.com/davidjurgens/potato/blob/master/docs/ai-intelligence/model_training.md#rdkit-property), [`peak-cnn`](https://github.com/davidjurgens/potato/blob/master/docs/ai-intelligence/model_training.md#peak-cnn), [`signal-cnn`](https://github.com/davidjurgens/potato/blob/master/docs/ai-intelligence/model_training.md#signal-cnn), and [`episode-model`](https://github.com/davidjurgens/potato/blob/master/docs/ai-intelligence/model_training.md#episode-model)

## References

- Ismail Fawaz, H., Lucas, B., Forestier, G., Pelletier, C., Schmidt, D. F., Weber, J., Webb, G. I., Idoumghar, L., Muller, P.-A., and Petitjean, F. (2020). InceptionTime: Finding AlexNet for Time Series Classification. *Data Mining and Knowledge Discovery*, 34(6), 1936-1962. https://doi.org/10.1007/s10618-020-00710-y
- Rogers, D., and Hahn, M. (2010). Extended-Connectivity Fingerprints. *Journal of Chemical Information and Modeling*, 50(5), 742-754. https://doi.org/10.1021/ci100050t