Train Models on Molecules, Spectra, Signals, and Robot Episodes

New in v2.14.0

This tutorial trains four models on scientific annotations in a Potato study. rdkit-property answers a question about a molecule, peak-cnn picks peaks on NMR, IR, Raman, UV-Vis, and mass spectra, signal-cnn stages sleep epochs or marks labeled intervals on EEG, ECG, and sensor recordings, and episode-model marks the phases, the outcome, and the reward curve of a robot episode. The page is for teams who already annotate molecules, spectra, signals, or robot episodes in Potato and want a model to pre-label the remaining items. For each trainer the page gives a config that passes potato validate config.yaml --strict, what a run learns, the held-out score to read, and how the predictions reach annotators. Train your first model walks through the Training page with a text classifier, and this tutorial assumes that walkthrough.

The four trainers

Each trainer learns from the answers of one annotation scheme and reads the model's input from the display or the file the scheme points at:

rdkit-property fits a scikit-learn model, and the other three trainers fit small PyTorch networks on the device that model_training.device selects. Every run trains on one scheme, scores the model on the run's held-out val and test items, and writes predictions back only for the schemes listed under writeback.schemas. Items in a run's train split never get a pre-label. peak-cnn, signal-cnn, and episode-model keep the checkpoint from the epoch that scored best on val, so Potato takes the held-out score of the three networks from test. A small study has few test items, and the run's test_n metric gives the count behind the score.

The three networks give an item's prediction the confidence of the prediction's least certain part, i.e., the weakest peak, interval, phase, or outcome. A writeback.min_confidence floor therefore withholds the whole item when any one part falls below the floor. A slider prediction from rdkit-property has no confidence value and always passes the floor.

Molecular properties with rdkit-property

rdkit-property learns the answers to a question about the molecule an item shows, and the molecule's structure is the model's input rather than the item's text. Each structure becomes a Morgan fingerprint of radius 2 and 2,048 bits followed by RDKit's 2D descriptors, e.g., molecular weight and logP. At radius 2 the Morgan fingerprint matches ECFP4, the extended-connectivity fingerprint with a diameter of 4 bonds. Logistic regression fits a radio, select, or likert answer by default, and ridge regression fits a slider. A random forest is the alternative for both. Install RDKit on the server that trains:

pip install "potato-annotation[chemistry]"

The config below shows each compound as a structure drawing and asks two questions, a 1-to-10 slider for how hard the compound would be to make and a yes-or-no choice about making the compound next:

annotation_task_name: "Compound triage"
task_dir: .
output_annotation_dir: annotation_output
data_files: [data/molecules.jsonl]
item_properties: {id_key: id, text_key: note}
user_config:
  allow_all_users: true
 
instance_display:
  fields:
    - key: smiles                 # SMILES, a molfile, or a .mol, .sdf or .smi file
      type: molecule
      label: "Compound"
 
annotation_schemes:
  - annotation_type: slider
    name: synthesis
    description: "How hard would this compound be to make? 1 is one step from a catalog reagent, 10 is a research project."
    min_value: 1
    max_value: 10
    starting_value: 5
    step: 1
  - annotation_type: radio
    name: advance
    description: "Would you make this compound next?"
    labels: ["yes", "no"]
 
model_training:
  enabled: true
  writeback:
    schemas: [synthesis, advance]
    include_unlabeled: true
    min_confidence: 0.6

Each line of the data file has one compound:

{"id": "cpd_023", "smiles": "CC(=O)Nc1ccc(O)cc1", "note": "Compound 23"}

The trainer takes the structure from the first molecule field in instance_display. A study without a molecule field can ask the question beside a chemical_structure scheme instead, and the trainer then learns from the structure annotators drew. When annotators drew different structures, the trainer takes the most common drawing, and when nobody drew, the trainer takes the structure the editor started from (prefill_field). The Training page offers rdkit-property for a molecule field once at least 80% of the items fill the field. A structure RDKit cannot parse is left out of training, gets no prediction, and is counted in the run's n_unparseable metric. For a choice or Likert question, the Training page also offers the text trainers, because the items still have text.

A slider's prediction is rounded to the slider's step and kept between min_value and max_value, so the model always proposes a position the slider has. The headline score for a slider is exact-position accuracy, which counts a prediction of 6 against an answer of 7 as wrong. Read val_mae or test_mae, the mean absolute error, beside the accuracy before deciding whether the predictions are close enough to show. The run needs at least 10 labeled molecules in the train split. Parameters such as model, class_weight, radius, and bits go in the Parameters field of the Training page or in the API request:

curl -X POST -H "X-API-Key: $KEY" -H "Content-Type: application/json" \
  http://localhost:8000/admin/training/api/runs \
  -d '{"trainer": "rdkit-property", "schemas": ["advance"],
       "params": {"model": "random_forest", "class_weight": "balanced"}}'

The run's log names the molecule count, the feature count, and the model. A balanced class weight counts rare answers more heavily. Use class_weight: balanced when most compounds get the same answer. Once the run finishes, an annotator who opens an unlabeled compound finds the slider set to the model's value and the model's choice selected, both editable.

Peaks on spectra with peak-cnn

peak-cnn learns to pick peaks from the peak_picking answers annotators gave. The trainer reads the spectrum the scheme's viewer_field shows, with the same readers the spectrum display uses. The training target is every peak the annotator kept, accepted suggestions and added peaks alike. A suggestion the annotator rejected marks a place with no peak. The model predicts peak positions only, and annotators still give each peak an assignment, a label, or a multiplicity. Install PyTorch on the machine that trains:

pip install "potato-annotation[train-vision]"

The config below asks annotators to review suggested peaks on 1H NMR spectra:

annotation_task_name: "1H NMR peak picking"
task_dir: .
media_directory: media
output_annotation_dir: annotation_output
data_files: [data/spectra.jsonl]
item_properties: {id_key: id, text_key: note}
user_config:
  allow_all_users: true
 
instance_display:
  fields:
    - key: spectrum
      type: spectrum
      label: "1H NMR"
      display_options:
        prepick:
          method: auto            # item peaks, then model, then file peak table, then detector
          prominence: 0.05
          max_peaks: 30
 
annotation_schemes:
  - annotation_type: peak_picking
    name: peaks
    description: "Accept or reject each suggested peak, and add any the suggestions missed"
    viewer_field: spectrum
    peak_fields:
      - name: assignment
        placeholder: "e.g. OCH2"
    require_review: true
    label_requirement: {required: true}
 
model_training:
  enabled: true
  trainer_params:
    peak-cnn:
      tolerance: 0.02             # ppm
  writeback:
    schemas: [peaks]
    include_unlabeled: true

The model's peaks reach annotators only through the display's prepick setting, and without prepick the display shows no suggestions at all. Under method: auto, the display offers the item's own peaks first, then the model's peaks, then the file's peak table, and then the peaks the detector finds. Before the first run, annotators review the detector's suggestions, and after a run the same items show the model's peaks without a restart. With method: model, the display shows only the model's peaks. Use method: model to measure how often annotators accept the model's peaks without the detector's suggestions mixed in. Each model peak is listed with "· model", the display keeps the max_peaks most confident, and an accepted model peak is stored with source: model.

The tolerance parameter sets how far, in x units, a predicted peak may sit from an annotated peak and still count as found. Without tolerance, the run uses 0.5% of the spectrum's x range, and 0.5% of a 12 ppm 1H window is 0.06 ppm. At 400 MHz, a 7 Hz coupling separates the lines of a multiplet by 0.0175 ppm. The default tolerance is wider than that spacing, and the config sets 0.02 ppm instead. The run scores precision, recall, and F1 over peaks matched one to one, nearest first, and reports the mean offset of the matched peaks. A peak predicted on a spectrum the annotator said has no peaks counts as a false alarm. The run needs at least 5 annotated spectra in the train split.

Each spectrum is resampled to length points, 4,096 by default, before the network sees the spectrum. Raise length when peaks are narrower than 1/length of the x range, e.g., a high-resolution mass spectrum. Reading mzML files also needs the spectra extra, as the spectrum display page describes.

Sleep stages and signal intervals with signal-cnn

signal-cnn learns from interval_label answers on physiological and sensor recordings such as EEG, ECG, EMG, and accelerometer data. The network is a small InceptionTime model, a stack of parallel convolutions of three widths. The trainer reads each recording with the readers of the time-series display, takes the channels the display names, resamples each channel to rate (100 Hz by default), and standardizes each channel. Install PyTorch, and the signals extra for WFDB and Parquet recordings:

pip install "potato-annotation[train-vision,signals]"

The config below scores sleep in 30-second epochs on overnight polysomnography, the way a human scorer labels each epoch as wake, N1, N2, N3, or REM:

annotation_task_name: "Sleep staging"
task_dir: .
media_directory: media
output_annotation_dir: annotation_output
data_files: [data/psg.jsonl]
item_properties: {id_key: id, text_key: note}
user_config:
  allow_all_users: true
 
instance_display:
  fields:
    - key: signal
      type: time_series
      label: "PSG"
      display_options:
        window: 30
        channels: ["EEG Fpz-Cz", "EOG", "EMG chin"]   # also the channels signal-cnn reads
 
annotation_schemes:
  - annotation_type: interval_label
    name: stage
    description: "Sleep stage of each 30-second epoch"
    signal_field: signal
    mode: epoch
    epoch_length: 30
    require_all_epochs: true
    label_requirement: {required: true}
    labels:
      - {name: W, key_value: "1"}
      - {name: N1, key_value: "2"}
      - {name: N2, key_value: "3"}
      - {name: N3, key_value: "4"}
      - {name: REM, key_value: "5"}
 
model_training:
  enabled: true
  writeback:
    schemas: [stage]
    include_unlabeled: true

The scheme's mode sets the task. In epoch mode, each annotated epoch is one training example, and the model labels each epoch of a new recording. A last epoch with less than half of the epoch's length recorded gets no label. The held-out headline is macro-F1 over the annotated epochs. Macro-F1 gives a rare stage such as N1 the same weight as N2. By default the training loss weights each stage by inverse label frequency (balanced: true), and an N1 epoch then counts for more than an N2 epoch. Every recording is read with the channels of the first training recording, in that order, and a recording that lacks one of the training channels gets a flat line in place of the missing channel. A display that names no channels gives the trainer every channel, up to max_channels (16).

Free mode trains on marked stretches instead. Add a second interval_label scheme with mode: free to the same signal_field to mark artifacts or arousals:

  - annotation_type: interval_label
    name: artifacts
    description: "Mark artifacts"
    signal_field: signal
    mode: free
    labels:
      - {name: artifact, key_value: "a"}
      - {name: spike, key_value: "s", kind: event}

In free mode the network trains on random 30-second crops (window_s), labels each 1-second frame (frame_s), merges consecutive frames with one label into an interval, and drops intervals shorter than min_duration. The headline is interval F1, which matches a predicted interval to an annotated one with the same label at IoU 0.5. Point events such as the spike label above are neither trained nor predicted. The run needs at least 3 annotated recordings in the train split, and epoch mode also needs at least two different labels in the data.

The model's intervals appear on the recording as dashed pre-annotations, listed with "· from the model", and are saved when the annotator makes a first change or clicks "Keep these". A recording's own annotations take precedence. An EDF+ file that contains a hypnogram shows the hypnogram and not the model's intervals, and the same goes for an item with preannotation_field intervals. Setting pre_annotations: false on the scheme turns off both kinds.

Robot episodes with episode-model

episode-model learns the three layers of an episode_annotation answer, i.e., the phases, the outcome, and the progress reward curve. The model's input is every numeric series the episode records, such as joint positions and gripper state, with one row per frame. The input includes series that series_shown leaves off the screen and excludes the video. The model cannot find a phase that shows only on camera and leaves the robot's measurements unchanged. Install PyTorch, plus pyarrow for LeRobot datasets or h5py for HDF5 demos:

pip install "potato-annotation[train-vision]"

The config below collects all three layers on pick-and-place demonstrations, as the robot episode page describes:

annotation_task_name: "Robot episode review"
task_dir: .
media_directory: media
output_annotation_dir: annotation_output
data_files: [data/episodes.jsonl]
item_properties: {id_key: id, text_key: note}
user_config:
  allow_all_users: true
 
annotation_schemes:
  - annotation_type: episode_annotation
    name: episode_review
    description: "Mark the phases, judge the outcome, and draw the progress."
    source_field: episode
    layers: [phases, outcome, reward]
    phases:
      - {name: reach, key_value: "1"}
      - {name: grasp, key_value: "2"}
      - {name: transport, key_value: "3"}
      - {name: place, key_value: "4"}
      - {name: retract, key_value: "5"}
    outcomes: [success, partial, failure]
    failure_causes: [missed grasp, object slipped, collision]
    reward_range: [0.0, 1.0]
    series_shown: [gripper, wrist_force, shoulder_lift, elbow]
 
model_training:
  enabled: true
  writeback:
    schemas: [episode_review]
    include_unlabeled: true

The network runs a convolution over the series into a bidirectional GRU with three heads, one for the phase of each frame, one for the outcome of the episode, and one for the reward at each frame. A layer an answer leaves empty is left out of that episode's loss, so an episode marked only with an outcome still trains the outcome head. The outcome head trains once the training episodes include at least two different outcomes. The reward head trains only on the stretch where the annotator drew the curve, because outside that stretch the annotator gave no answer. Episodes longer than max_frames (2,000) are trained on random crops.

The prediction has the shape of a stored answer, with phases from the scheme's list, one of the scheme's outcomes, and a reward curve of 50 points inside reward_range, for the layers the scheme shows. The prediction pre-fills the timeline and is stored once the annotator edits the timeline. The failure cause and the instruction are left to the annotator. The held-out headline is phase macro-F1 on a 0.1-second grid. The run also reports phase segment F1 at IoU 0.5, outcome accuracy, and Pearson's r of the reward curve where a curve was drawn. A scheme without phases takes outcome accuracy as the headline, and then reward correlation. The run needs at least 5 annotated episodes in the train split. Potato has no trainer for rollout_evaluation, because a rollout preference compares videos and episode-model reads no video.

Start a run and check the result

Restart the server after adding model_training:

potato start config.yaml -p 8000

Open /admin/training, pick the scheme and the trainer, and click Train now. The page offers only the trainers that fit each scheme, and a trainer that is not installed is listed as "not installed", with the install command. When the run finishes, Potato lists the held-out score in the Runs table, with the split and the metric in a tooltip. The run page lists every metric and the split of each item. The write-back report in GET /admin/training/api/status counts the predictions published and the ones each check held back, e.g., n_skipped_low_confidence.

Pre-labels anchor annotators to the model's answer, so turn on include_unlabeled in a new study only once the held-out score supports showing the model's answers. Review model pre-labels in the loop covers serving the least-confident pre-labels first and retraining on corrections, and Configuration lists every key, the backends, and the write-back checks.

Further reading

• Model training overview
• All trainers
• Chemical structures, spectra, and time series
• Robot episode annotation and the robot episode guide
• Potato for chemistry, physiology, and robotics
• Model-assisted labeling
• Upstream reference for rdkit-property, peak-cnn, signal-cnn, and episode-model

References

• Ismail Fawaz, H., Lucas, B., Forestier, G., Pelletier, C., Schmidt, D. F., Weber, J., Webb, G. I., Idoumghar, L., Muller, P.-A., and Petitjean, F. (2020). InceptionTime: Finding AlexNet for Time Series Classification. Data Mining and Knowledge Discovery, 34(6), 1936-1962. https://doi.org/10.1007/s10618-020-00710-y
• Rogers, D., and Hahn, M. (2010). Extended-Connectivity Fingerprints. Journal of Chemical Information and Modeling, 50(5), 742-754. https://doi.org/10.1021/ci100050t