Differentially Private Fine-Tuning with Opacus

New in v2.14.0

This tutorial trains a classifier on a study's own labels with differential privacy, so the released model reveals little about any one annotated item. The page is for study owners whose items or labels are sensitive (e.g., health records or private messages) and who plan to share or publish the model. You will add a dp block to the parameters of hf-text-classification, run the trainer on the product-review example, and read the privacy spent from the run log and the model card. The same block works on hf-image-classification, and no other trainer takes dp.

Epsilon and delta in differential privacy

Differential privacy is a mathematical guarantee about a computation on a dataset. Adding or removing one record changes the probability of any output by a bounded amount, so the output reveals little about whether that record was present. Epsilon (ε) is the bound. A smaller epsilon makes the outputs with and without the record harder to tell apart, and a smaller epsilon therefore gives stronger privacy. Delta (δ) is the probability that the epsilon bound fails, and Potato's documentation describes delta as "the chance the guarantee does not hold".

DP-SGD trains a neural network under this guarantee, by the method of Abadi et al. (2016). Each training step clips every item's gradient to a maximum norm and adds Gaussian noise to the sum. The clip bounds how far one item can move the model, and the noise masks that bounded move. Opacus is a PyTorch library that implements DP-SGD, and Potato hands the training loop to Opacus whenever a run carries dp. Opacus's RDP accountant adds up the epsilon that each step spends.

Install the trainer and Opacus

The transformer trainers need torch and transformers from Potato's training extras. No Potato extra includes Opacus, because Opacus pins specific torch versions. Install both on the machine where the run trains:

pip install "potato-annotation[train-text]"   # [train-vision] for hf-image-classification
pip install opacus

A run with dp checks for Opacus on the training machine before training starts. On an ssh, SLURM, or agent backend, Opacus therefore goes on the GPU server or compute node that trains, rather than on the annotation server. Remote training covers those backends.

Configure the trainer

The config below is the product-review example from Train your first model, with private training set as the default for hf-text-classification. Copy data/reviews.json and data/gold.json from Potato's examples/ai-assisted/model-training/ into a project folder, and save this config there as config.yaml:

annotation_task_name: "Review Sentiment (private training)"
 
data_files:
  - "data/reviews.json"
 
item_properties:
  id_key: "id"
  text_key: "text"
 
task_dir: "."
output_annotation_dir: "annotation_output"
 
require_password: false
admin_api_key: "demo-admin-key"
 
num_annotators_per_item: 1
 
annotation_schemes:
  - annotation_type: "radio"
    name: "sentiment"
    description: "Is this review positive or negative?"
    labels:
      - "positive"
      - "negative"
    sequential_key_binding: true
 
model_training:
  enabled: true
  trainer_params:
    hf-text-classification:
      epochs: 5
      batch_size: 16
      dp:
        epsilon: 8
        max_grad_norm: 1.0
  writeback:
    schemas: ["sentiment"]
    include_unlabeled: true
    min_confidence: 0.6

trainer_params sets the defaults for every run of hf-text-classification, so the Training page's Parameters field opens with the dp entry already filled in. epsilon: 8 sits inside the range of 1 to 10 that Potato's documentation calls usual. delta is left out, so Potato sets delta to 1 / (10 × train items). max_grad_norm: 1.0 repeats the default clip. epochs and batch_size repeat the trainer's defaults, because Opacus sizes the noise from both. require_password: false and the plain admin key suit a local trial only.

dp takes three keys:

The config passes python -m potato.validate_cli config.yaml --strict. The validator accepts a mapping, null, or false for dp and refuses dp: true. The validator also checks each key inside dp and the key's range. Opacus and the checks that depend on the number of train items run when a run starts, on the machine that trains, as Errors at the start of a run lists.

Label items and start a run

hf-text-classification refuses to fit fewer than 20 train items (min_instances). Start the server, then label reviews by hand or with three simulated annotators from the user simulator, who answer from the gold file:

potato start config.yaml -p 8000
 
python -m potato.simulator --server http://localhost:8000 --users 3 \
  --strategy gold_standard --competence good=1.0 --max-annotations 40 \
  --gold-file data/gold.json --sequential --fast-mode --no-export

The simulator labels 120 of the 160 reviews. With the default splits (70% train, 15% val, 15% test) and split_seed: 0, our run put 76 labeled reviews in train, 19 in val, and 25 in test. Delta therefore defaults to 1 / 760, about 0.00132, and any delta you set must be below 1 / 76, about 0.0132.

Open http://localhost:8000/admin/training with the admin key, pick hf-text-classification and the sentiment scheme, check that the Parameters field shows the dp entry, and click Train now. The same run starts from the API:

curl -X POST -H "X-API-Key: demo-admin-key" -H "Content-Type: application/json" \
  http://localhost:8000/admin/training/api/runs \
  -d '{"trainer": "hf-text-classification", "schemas": ["sentiment"]}'

The response carries the run_id. A run's own params override trainer_params, so "params": {"dp": {"epsilon": 2}} trains one run at a smaller budget, and "params": {"dp": null} or "params": {"dp": false} trains one run without DP. Keep split_seed unchanged between the two runs, so both train and score on the same items and the held-out scores compare directly.

Run log lines under DP

The run page at /admin/training/runs/<run_id> lists the run's events and metrics. A DP run adds three kinds of line to the events.

The first line names the embedding tables Potato freezes. A text transformer's position and token-type embeddings take one row shared by the whole batch, and Opacus cannot compute a per-item gradient through a shared row. For the default model, distilbert/distilroberta-base, the frozen tables are roberta.embeddings.position_embeddings and roberta.embeddings.token_type_embeddings. Frozen tables do not train and spend no privacy, and the word embeddings and every other layer still train.

The second line starts with DP-SGD: and states the target epsilon, delta, the planned epochs, the noise multiplier Opacus chose, the per-item clip, and the sampling rate. Each step draws a batch by Poisson sampling, so every train item joins a batch independently with probability 1 / batches per epoch. With 76 train items in batches of 16, an epoch has 5 batches and the log gives a sampling rate of 0.2000. The noise multiplier sets the standard deviation of the noise, and Opacus's FAQ gives 0.1 to 2 as the usual range.

After each epoch, a line of the form Epoch <n>/<epochs>: loss <loss>, epsilon spent <epsilon> reports the epoch's mean loss and the epsilon spent so far. The same value is recorded per epoch as the dp_epsilon metric, and the run's final metrics carry dp_epsilon for the whole run. Opacus sizes the noise so the planned epochs spend at most the target epsilon. A run that stops early on patience (two epochs without a better val score, by default) spends less than the target.

The dp entry in the model card

The run writes model_card.json to artifacts/ in the run directory, <task_dir>/<output_annotation_dir>/training/runs/<run_id>/. A DP run's card has a dp entry:

epsilon_spent counts every epoch the run trained. Epochs after the kept checkpoint count toward the total. Publishing to the Hugging Face Hub uploads model_card.json with the model, with local paths replaced by file names, so the dp entry travels with the weights. The README.md model card on the Hub also has a "Differential privacy" row with the epsilon spent against the target, the delta, and the train split the guarantee covers. The row leaves out the target or the delta when the run did not record the value.

Choosing the privacy budget and the clip

Potato's documentation gives 1 to 10 as the usual range for epsilon and requires delta below 1 / train items. Opacus's FAQ gives the same rule of thumb for delta and recommends a small constant for epsilon with a tiny delta. The default delta of 1 / (10 × train items) sits under the ceiling of 1 / train items, so most studies can leave delta out.

The clip, max_grad_norm, trades two errors. Clipping changes every gradient larger than the threshold, and Opacus's FAQ says clipping impairs convergence. The noise is scaled to the clip, so a larger clip adds more noise. The FAQ adds that a wrong threshold hurts linear models more than deep ones. Start from the default of 1.0 and compare held-out scores before tuning the clip.

Opacus sizes the noise for the planned epochs, so the same epsilon spread over fewer epochs means less noise in each step. Opacus's FAQ notes that a larger batch raises the sampling rate and with the sampling rate the privacy cost. The FAQ suggests a larger learning rate and earlier stopping to offset the cost. Compare a DP run's held-out score with the score of a run without dp on the same split. The gap between the two scores measures what the guarantee costs on your data.

Scope of the guarantee

The guarantee covers the train split. Changing one train item changes what the released model could be by at most the stated epsilon. The item counts together with the label resolved from the item's annotators. Potato states the guarantee per item, and the documentation states no guarantee per annotator.

The val split is outside the guarantee. After each epoch, the trainer scores the model on the val items and keeps the checkpoint from the best-scoring epoch. That choice reads the val items' labels with no noise added, so the epsilon on the model card makes no promise about val items. When val items need the same protection as train items, set the val fraction to 0 and move the fraction to test, e.g., splits: {train: 0.7, val: 0, test: 0.3}. With no val items, the trainer keeps the last epoch, patience never stops the run early, and the held-out score comes from test. Leaving the train fraction unchanged keeps the same items in train, because Potato assigns splits by a seeded hash of each item's id.

The model's label list is outside the guarantee. Potato reads the label list from the labels that appear in the data, so a label annotators used on a single item still appears among the model's outputs.

Training differences under DP

DP-SGD needs a gradient for each item, so a DP run differs from an ordinary run of the same trainer in these ways:

• Training runs in full precision, whatever precision says, because mixed precision is off under DP.
• gradient_accumulation must be 1. Each step trains one Poisson-sampled batch.
• A quantized base model (lora.quantize) is refused.
• A text transformer's position and token-type embedding tables are frozen, as the run log reports.
• A model with a layer Opacus does not support, such as BatchNorm, fails at the start of training with the layer named.

Only hf-text-classification and hf-image-classification take dp. Every other trainer refuses a parameter the trainer does not take, so dp on sklearn-text, setfit, or hf-token-classification fails at boot under trainer_params and with a 409 when a run starts.

Errors at the start of a run

potato validate reports a key dp does not take and a value out of range before the server starts. A run checks the dp block again when the run starts, together with Opacus and the limits that depend on the train split. A run that fails those checks ends in an error before training, with the code validation_failed and every problem in one message. The messages come from Potato's checks:

An empty mapping, dp: {}, asks for DP with no budget and fails with the missing-epsilon message. When trainer_params sets dp, a run's "dp": null or "dp": false turns DP off for that run. The status route, GET /admin/training/api/status, lists the same messages under errors.

Further reading

• Model training overview, for the Training page and Hub publishing
• Trainers, for what hf-text-classification and hf-image-classification fit
• Model training configuration, for splits, held-out scores, and write-back
• Train your first model, for the same project without a GPU or DP
• Remote training, for running DP fine-tuning on a GPU server
• Differential privacy (dp) in Potato's model training reference

References

• Abadi, M., Chu, A., Goodfellow, I., McMahan, H. B., Mironov, I., Talwar, K., and Zhang, L. (2016). Deep learning with differential privacy. Proceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security, 308-318. https://doi.org/10.1145/2976749.2978318
• Opacus. FAQ. https://opacus.ai/docs/faq