# Fine-Tuning an LLM on Your Annotations

Source: https://www.potatoannotator.com/docs/model-training/fine-tuning-llms

*New in v2.14.0*

This tutorial trains four kinds of language model on the answers a Potato study collects. The `llm-sft` trainer fine-tunes an LLM on written answers or corrected agent traces, `llm-dpo` tunes an LLM on pairwise choices, `reward-model` fits a scalar reward model on comparisons, and `prm-reward` fits a process reward model on step ratings. The page is written for teams that already collect preference or trace data in Potato and want a first model from those labels without exporting the data to a separate training script. Each section names the scheme that produces the data, gives a config that passes Potato's validator, lists the parameters worth changing, and says what the held-out score measures and whether the predictions go back to annotators as pre-labels. The [model training overview](/docs/model-training/overview) covers the Training page, and [Train Your First Model](/docs/model-training/first-model) runs the whole loop with a classifier that needs no GPU.

## Trainers by annotation scheme

Every trainer on this page learns from one annotation scheme per run, and the Training page offers a trainer only for the schemes the trainer can learn. The table below lists the schemes each trainer learns from:

| Trainer | Scheme | What the model learns | Held-out headline | Shown to annotators |
|---|---|---|---|---|
| `llm-sft` | `text` | to write the answer annotators wrote | token F1 | yes, the generated answer fills the text box |
| `llm-sft` | `trajectory_edit` | to rewrite an agent's steps into the corrected steps | `closer_to_correction` | no, stored and scored only |
| `llm-dpo` | `pairwise` in `binary` mode | to prefer the response annotators chose | accuracy, ties not counted | yes, the preferred response's card is selected |
| `llm-dpo` | `trajectory_edit` | to prefer the corrected steps over the original | `closer_to_correction` | no, stored and scored only |
| `reward-model` | `pairwise` (`binary` and `scale`), `ranking`, `bws` | a score per response | accuracy, direction accuracy, Kendall's tau, or best-worst accuracy | yes, as the scheme's own answer |
| `prm-reward` | `process_reward` | a rating of correct, neutral, or wrong for each step | `step_macro_f1` | offered through the "AI pre-label steps" button |

## Before you start

Potato's training extras install PyTorch, Transformers, and PEFT for the four trainers:

```bash
pip install "potato-annotation[train-llm]"
pip install "potato-annotation[train-text]"
```

The first extra covers `llm-sft` and `llm-dpo`, and the second covers `reward-model` and `prm-reward`. `pip install "potato-annotation[train]"` installs every training extra at once. A trainer whose packages are missing is listed on the Training page as "not installed", with the command that installs the packages.

Each trainer refuses to fit with fewer than 20 items in the train split (the `min_instances` parameter). The default split puts 70% of labeled items in train, 15% in val, and 15% in test, so plan on about 30 labeled items before the first run. The val and test items are held out of training and score the model.

`llm-sft` and `llm-dpo` start from `Qwen/Qwen2.5-0.5B-Instruct`, `reward-model` from `distilbert/distilroberta-base`, and `prm-reward` from `answerdotai/ModernBERT-base`. The LLM trainers are built to run on a GPU. When the machine running Potato has no GPU, [Remote Training](/docs/model-training/remote-training) sends runs to a GPU server over ssh or to a SLURM cluster.

## A study that collects preferences and rewrites

The first config asks two questions about each customer-support prompt. Annotators pick the better of two responses and write the answer an annotator would want from the assistant. The choice trains `llm-dpo` and `reward-model`, and the written answer trains `llm-sft`. Each line of `data/answers.jsonl` holds a prompt and the two responses:

```json
{"id": "q001", "prompt": "How do I reset my router to factory settings?", "responses": ["Hold the reset button on the back for 10 seconds until the lights blink, then wait two minutes for the router to restart.", "Routers can be reset. Check the manual."]}
```

The config shows the prompt as the item's text and the two responses as cards A and B:

```yaml
annotation_task_name: "Support answers: preferences and rewrites"
task_dir: .
data_files:
  - data/answers.jsonl
item_properties:
  id_key: id
  text_key: prompt
output_annotation_dir: annotation_output
user_config:
  allow_all_users: true

annotation_schemes:
  - annotation_type: pairwise
    name: preferred
    description: "Which response answers the question better?"
    mode: binary
    items_key: responses
    allow_tie: true
    tie_label: "No preference"

  - annotation_type: text
    name: rewrite
    description: "Write the answer you would want the assistant to give."
    multiline: true
    rows: 4

model_training:
  enabled: true
  writeback:
    schemas: [preferred]
    min_confidence: 0.7
```

`text_key: prompt` matters twice, because the prompt is what annotators read and what `llm-sft` puts after the question. `items_key: responses` names the list the pairwise cards come from, in data order, so A is the first response. `writeback.schemas` lists only `preferred`, so annotators see the model's choice and not a generated rewrite. Pre-labels [anchor annotators](/docs/guides/detecting-rubber-stamped-prelabels), so add `rewrite` to the list only once the `llm-sft` run's held-out score supports showing the model's text. `min_confidence: 0.7` keeps predictions the model is less sure of off the page while still storing the predictions. Check the config with Potato's validator before starting the server:

```bash
python -m potato.validate_cli config.yaml --strict
```

Start Potato, label at least 30 prompts, and open `/admin/training` as an admin. The runs below can also start from the API with the config's `admin_api_key` in `$KEY`.

## Supervised fine-tuning with `llm-sft`

`llm-sft` turns each labeled item into one training example. The prompt is the scheme's `description` (the question annotators saw) followed by the item's text, and the target is the answer the annotator wrote. The loss covers only the answer's tokens, so the model learns to answer rather than to repeat the prompt. A [LoRA](https://arxiv.org/abs/2106.09685) adapter is trained by default instead of every weight.

Pick `llm-sft` and the `rewrite` scheme on the Training page and click **Train now**, or start the same run from the API:

```bash
curl -X POST -H "X-API-Key: $KEY" -H "Content-Type: application/json" \
  http://localhost:8000/admin/training/api/runs \
  -d '{"trainer": "llm-sft", "schemas": ["rewrite"], "params": {"max_new_tokens": 256}}'
```

Most runs change only the parameters in this table:

| Parameter | Default | Meaning |
|---|---|---|
| `model_name` | `Qwen/Qwen2.5-0.5B-Instruct` | A causal language model on the Hugging Face Hub, or a local directory. |
| `lora` | `true` | `false` trains every weight and needs several times the memory. `{quantize: 4bit}` loads the base model in 4 bits (see [QLoRA](#fitting-a-larger-model-with-qlora)). |
| `max_length` | 1024 | Tokens per example. A long prompt loses tokens from the start, and the answer is kept. |
| `max_new_tokens` | 256 | The longest answer the model generates. |
| `instruction` | the scheme's `description` | Text put before each item's text. |
| `system_prompt` | none | A system message for models with a chat template. |
| `gradient_checkpointing` | `false` | Saves activation memory at the cost of about a third more time. |
| `learning_rate` | 2e-4 with LoRA, 1e-5 without | |

The run keeps the checkpoint with the lowest val loss. Because the run chose the checkpoint on val, the held-out score is read from test. The score is token F1, the overlap of words between the generated and the written answer regardless of order, with exact match reported beside the score. Token F1 credits an answer that uses the right words in the wrong places, so read the two metrics together.

The model generates an answer greedily for every val and test item. With `writeback.include_unlabeled: true`, the model also answers the items nobody has labeled. A prediction's confidence is the model's mean probability for the tokens the model generated, and the [model review](/docs/model-training/model-in-the-loop) queue sorts by that confidence. With `rewrite` under `writeback.schemas`, the generated answer fills the text box when an annotator opens the item.

Free text is a weak fit for several annotators per item. The `aggregation` setting resolves several answers to one by comparing answers word for word, and two written answers rarely match word for word. For `llm-sft` on a `text` scheme, one writer per item avoids the word-for-word comparison.

## Preference tuning with `llm-dpo`

[Direct Preference Optimization](https://arxiv.org/abs/2305.18290) (DPO) trains an LLM to raise the likelihood of the preferred response relative to the other response, measured against a frozen reference copy of the starting model. `llm-dpo` builds one example from each item answered A or B. The prompt is the item's `prompt` field, or the first of `question`, `instruction`, `query`, `input`, and `context` that the item has. Items answered with the tie option are left out, since a tie does not mark either response as better.

Pick `llm-dpo` and the `preferred` scheme, or send the same request as above with `"trainer": "llm-dpo"` and `"schemas": ["preferred"]`. With the default LoRA adapter, the reference model is the same model with the adapter switched off, so no second copy is held in memory. `lora: false` trains every weight and keeps a frozen copy as the reference, and the frozen copy doubles the memory.

| Parameter | Default | Meaning |
|---|---|---|
| `model_name` | `Qwen/Qwen2.5-0.5B-Instruct` | A causal language model. |
| `beta` | 0.1 | How far the model may move from the reference. A lower value lets the model move further. |
| `epochs` | 2 | |
| `learning_rate` | 5e-5 with LoRA, 5e-7 without | |
| `max_length` | 1024 | Tokens per prompt and response. |
| `instruction` | none | Text put before each item's prompt. |

The run keeps the checkpoint with the lowest DPO loss on val, and the run log reports how often the model prefers the annotators' choice on val. The held-out score is accuracy on test, over the items not answered as a tie, with macro-F1 beside the accuracy. A prediction is the response the trained model prefers, with the probability of that preference as the confidence. As a pre-label, the preferred response's card is selected when the annotator opens the item. With `randomize_order` on the scheme, the selection follows the response to wherever the annotator sees the response. The saved adapter is a policy as well and generates text when loaded with PEFT. For background on collecting the preference data itself, see [Collecting RLHF and Preference Data](/docs/guides/rlhf-preference-data) and [Pairwise Comparison](/docs/annotation-types/pairwise-comparison).

## A reward model with `reward-model`

A reward model maps a response to a number, for RLHF, for best-of-n sampling, or for ranking new outputs. `reward-model` trains a transformer with one output so that the response annotators preferred scores higher. The loss is the [Bradley–Terry](https://en.wikipedia.org/wiki/Bradley%E2%80%93Terry_model) loss, `-log sigmoid(r_winner - r_loser)`, the reward-modeling loss of [InstructGPT](https://arxiv.org/abs/2203.02155). The `preferred` scheme above trains `reward-model` with no change to the config. Each comparative scheme gives a different training signal:

| Scheme | Training signal | Held-out headline |
|---|---|---|
| `pairwise`, `binary` | The chosen response beats the other. Ties are skipped. | accuracy |
| `pairwise`, `scale` | The side the slider leans to wins, weighted by how far the slider leans. The midpoint is skipped. | direction accuracy |
| `ranking` | The whole order, by the Plackett-Luce loss, or every pair in the order with `ranking_loss: pairs`. | Kendall's tau |
| `bws` | The best item beats every other item in the tuple, and every other item beats the worst. | best-worst accuracy |

When an item has a prompt field, each reward is of the response given the prompt. A ranking's options are the scheme's labels, each scored given the item's text. The server builds BWS tuples from `bws_config`, so a run from the server's own data has the tuples, and the data files do not need the tuples.

| Parameter | Default | Meaning |
|---|---|---|
| `model_name` | `distilbert/distilroberta-base` | An encoder, or an LLM with `lora`. |
| `ranking_loss` | `plackett_luce` | `ranking` schemes only: `plackett_luce` or `pairs`. |
| `max_length` | 512 | Tokens per scored text, prompt and response together. |
| `learning_rate` | 2e-5, or 2e-4 with LoRA | |

Predictions answer the scheme's own question, so the predictions are scored and written back like any other scheme's. The pre-label is the selected card for a binary choice, the slider position for a scale, the list's order for a ranking, and the selected best and worst for BWS. Every prediction also stores the `rewards` the prediction came from. Only differences between rewards are meaningful. [Pairwise and Best-Worst Scaling](/docs/guides/pairwise-and-best-worst) compares the comparative formats for collecting the data.

## A study that collects step ratings and corrections

The second config works on agent traces. Annotators rate each step of the trace and then fix the wrong steps. The step ratings train `prm-reward`, and the corrections train `llm-sft` or `llm-dpo`. Each line of `data/traces.jsonl` holds the task, the agent's steps, and the final answer:

```json
{"id": "traj_001", "task_description": "Find the current weather in San Francisco and report the temperature.", "steps": [{"thought": "I should look up the current weather.", "action": "web_search(queyr='San Francisco weather')"}, {"thought": "Open the first result.", "action": "open_url(results[0])"}, {"thought": "Read the temperature from the page.", "action": "extract_text('.temp')"}], "final_answer": "It is sunny."}
```

The trainers read the task from the item's `task_description`, `task`, or `text` field, in that order:

```yaml
annotation_task_name: "Agent traces: step ratings and corrections"
task_dir: .
data_files:
  - data/traces.jsonl
item_properties:
  id_key: id
  text_key: task_description
output_annotation_dir: annotation_output
user_config:
  allow_all_users: true

annotation_schemes:
  - annotation_type: process_reward
    name: step_quality
    description: "Rate each step of the agent's trace"
    steps_key: steps
    step_text_key: action
    mode: per_step
    allow_neutral: true
    ai_prelabel: true

  - annotation_type: trajectory_edit
    name: corrected_trace
    description: "Fix the wrong steps, then correct the final answer"
    steps_key: steps
    step_text_key: action
    editable_fields: [action]
    show_diff: true
    edit_final_answer: true
    final_answer_key: final_answer

model_training:
  enabled: true
  writeback:
    schemas: [step_quality]
    include_unlabeled: true
```

`mode: per_step` with `allow_neutral: true` gives the three ratings `prm-reward` learns. `ai_prelabel: true` puts the "AI pre-label steps" button on the scheme, and the trained ratings appear through the button. `writeback.schemas` names only `step_quality`. Listing `corrected_trace` there stops the server from starting, and the validator gives the reason:

```text
model_training.writeback.schemas names 'corrected_trace', a trajectory_edit scheme, which has no prelabel display: its predictions are stored and scored but annotators would never see them. Leave it out of writeback.schemas.
```

## Repairing traces with `llm-sft` and `llm-dpo`

A `trajectory_edit` answer is a pair of traces, the agent's original steps and the steps after the annotator's corrections. [Trajectory Editing for SFT/DPO](/docs/agent-evaluation/trajectory-correction) covers the editor, and [the trajectory editing post](/blog/trajectory-editing-sft-dpo-training-data) covers how the pairs become training data. Both trainers leave out traces nobody edited.

- **`llm-sft`** takes the task and the original steps as the prompt and the corrected steps as the target, so the model learns to repair a trace. Each step is written as `Step N` followed by the step's fields (`action: ...`, `thought: ...`), and the final answer follows the last step when the scheme has `final_answer_key`.
- **`llm-dpo`** takes the task as the prompt, the corrected steps as the preferred response, and the original steps as the other response. A prediction says whether the model prefers the correction (`A`) or the original (`B`), and is made only for held-out traces that have a correction.

Both runs report `closer_to_correction` as the held-out score. For `llm-dpo`, the score is the share of held-out traces where the model prefers the correction. For `llm-sft`, the score is the share where the written steps are closer to the correction than to the original by token F1, with `token_f1` against the correction reported beside the score.

The predictions from a `trajectory_edit` run are stored and scored, and annotators never see the predictions. The editor always starts from the agent's original trace, so the page has no place to show a predicted rewrite. Read the predictions from the run page or from `artifacts/predictions.jsonl` in the run's directory.

## Step ratings with `prm-reward`

A process reward model rates the steps of a trace rather than only the final answer, as in [PRM800K](https://arxiv.org/abs/2305.20050). `prm-reward` feeds the whole trace to a transformer encoder as one input, the task followed by `Step 1`, `Step 2`, and so on. The encoder learns a three-way rating at each step's last token, so each rating is made with the steps before the step in view. Steps nobody rated are left out of the loss. The default model, `answerdotai/ModernBERT-base`, was chosen for the model's 8,192-token context.

| Parameter | Default | Meaning |
|---|---|---|
| `model_name` | `answerdotai/ModernBERT-base` | An encoder with a fast tokenizer. |
| `max_length` | 1024 | Tokens per trace. |
| `epochs` | 6 | |
| `batch_size` | 8 | Traces per step. |
| `learning_rate` | 5e-5, or 3e-4 with LoRA | |

Long traces lose their last steps. A trace longer than `max_length` tokens is cut from the end, and the steps past the cut are neither trained on nor rated. The run log counts the steps lost. When `max_length` is more than the model holds, the run lowers `max_length` to the model's limit with a warning, and the limit is 512 tokens for most BERT-class encoders. When the run log reports lost steps, raise `max_length` and choose a model whose context holds the longer traces.

The held-out score is `step_macro_f1`, macro-F1 over the three ratings on every rated step. Most steps in most traces are correct, so accuracy would flatter a model that never says "wrong". The run also reports `step_accuracy` and `first_error_accuracy`, the share of traces where the model's first wrong step is the annotators' first wrong step.

A prediction rates every step that fits in `max_length`, with each rating's probability as the rating's confidence and the least confident step as the trace's confidence. With the scheme in `writeback.schemas` and `ai_prelabel: true`, the "AI pre-label steps" button offers the trained ratings for the annotator to verify. An LLM judge configured under `ai_support` is not called for an item that has trained ratings. [Process Reward Annotation](/docs/features/process-reward-annotation) covers the scheme, and [the process reward models guide](/docs/guides/process-reward-models) covers the data.

## Fitting a larger model with QLoRA

[QLoRA](https://arxiv.org/abs/2305.14314) loads the base model's weights in 4 bits and trains a LoRA adapter on top. `llm-sft`, `llm-dpo`, `hf-text-classification`, `vlm-sft`, and `reward-model` take `lora: {quantize: 4bit}`, and `prm-reward` refuses the setting before loading the model. Plan the memory from the "Peak GPU memory" line that every transformer run logs at the end. The [QLoRA section of the configuration reference](/docs/model-training/configuration#qlora) describes how the base weights load and the 8-bit option.

Add this block to the config above to set QLoRA as the default for `llm-sft`:

```yaml
model_training:
  enabled: true
  trainer_params:
    llm-sft:
      model_name: Qwen/Qwen2.5-7B-Instruct
      lora: {quantize: 4bit}
      gradient_checkpointing: true
```

QLoRA needs a CUDA GPU and a `bitsandbytes` build for that machine's CUDA, installed with `pip install bitsandbytes`. Without a working GPU or build, the run stops at validation before loading the model.

## Publishing a run to the Hugging Face Hub

A successful run whose model is on the Potato server has **Publish to the Hugging Face Hub** on the run page. The push uploads the model and a `README.md` model card written from the run, with the trainer, the base model and adapter, the schemes and labels, the split sizes, the held-out score and what the score was measured on, and the run and bundle ids. A LoRA run saved only the adapter, so the Hub repository holds the adapter, and the card names the base model to load the adapter onto.

The token stays out of the browser. Create a write token in your [Hugging Face settings](https://huggingface.co/docs/hub/security-tokens) and put the token in the server's environment before starting Potato:

```bash
export HF_TOKEN=hf_your_write_token
potato start config.yaml
```

The form names the environment variable, `HF_TOKEN` by default, asks for the repository name twice, and keeps **Private** ticked by default. [Publishing to the Hugging Face Hub](/docs/model-training/configuration#publishing-to-the-hugging-face-hub) lists the files that go up, the checks before a push, and the license on the model card. The same push from the API:

```bash
curl -X POST -H "X-API-Key: $KEY" -H "Content-Type: application/json" \
  http://localhost:8000/admin/training/api/runs/RUN_ID/push-to-hub \
  -d '{"repo_id": "your-org/support-dpo", "confirm_repo_id": "your-org/support-dpo", "private": true, "licence": "apache-2.0"}'
```

The route returns `201` with the commit. A model left on a remote host with `fetch_artifacts: false` cannot be pushed from the Potato server, and the run page says where the model is. To move on from a published model to an active-learning loop, continue with [Model in the Loop](/docs/model-training/model-in-the-loop).

## Further reading

- [Model Training Configuration](/docs/model-training/configuration): every `model_training` key, the held-out scores, write-back, and the routes.
- [Trainers](/docs/model-training/trainers): all 38 trainers, grouped by task.
- [Remote Training](/docs/model-training/remote-training): GPU servers over ssh, SLURM clusters, and the polling agent.
- [Agent Trajectory Annotation](/docs/guides/agent-trajectory-annotation): collecting trace data in Potato.
- Upstream reference sections for [`llm-sft`](https://github.com/davidjurgens/potato/blob/master/docs/ai-intelligence/model_training.md#llm-sft), [`llm-dpo`](https://github.com/davidjurgens/potato/blob/master/docs/ai-intelligence/model_training.md#llm-dpo), [agent traces](https://github.com/davidjurgens/potato/blob/master/docs/ai-intelligence/model_training.md#agent-traces), [`prm-reward`](https://github.com/davidjurgens/potato/blob/master/docs/ai-intelligence/model_training.md#prm-reward), [`reward-model`](https://github.com/davidjurgens/potato/blob/master/docs/ai-intelligence/model_training.md#reward-model), [QLoRA](https://github.com/davidjurgens/potato/blob/master/docs/ai-intelligence/model_training.md#qlora-4-bit-base-weights), and [Hub publishing](https://github.com/davidjurgens/potato/blob/master/docs/ai-intelligence/model_training.md#publish-a-model-to-the-hugging-face-hub).

## References

- Dettmers, T., Pagnoni, A., Holtzman, A., and Zettlemoyer, L. (2023). QLoRA: Efficient Finetuning of Quantized LLMs. *arXiv:2305.14314*. [arxiv.org/abs/2305.14314](https://arxiv.org/abs/2305.14314)
- Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., and Chen, W. (2021). LoRA: Low-Rank Adaptation of Large Language Models. *arXiv:2106.09685*. [arxiv.org/abs/2106.09685](https://arxiv.org/abs/2106.09685)
- Lightman, H., Kosaraju, V., Burda, Y., Edwards, H., Baker, B., Lee, T., Leike, J., Schulman, J., Sutskever, I., and Cobbe, K. (2023). Let's Verify Step by Step. *arXiv:2305.20050*. [arxiv.org/abs/2305.20050](https://arxiv.org/abs/2305.20050)
- Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., Schulman, J., Hilton, J., Kelton, F., Miller, L., Simens, M., Askell, A., Welinder, P., Christiano, P., Leike, J., and Lowe, R. (2022). Training Language Models to Follow Instructions with Human Feedback. *arXiv:2203.02155*. [arxiv.org/abs/2203.02155](https://arxiv.org/abs/2203.02155)
- Rafailov, R., Sharma, A., Mitchell, E., Ermon, S., Manning, C. D., and Finn, C. (2023). Direct Preference Optimization: Your Language Model is Secretly a Reward Model. *arXiv:2305.18290*. [arxiv.org/abs/2305.18290](https://arxiv.org/abs/2305.18290)