Fine-Tuning an LLM on Your Annotations

New in v2.14.0

This tutorial trains four kinds of language model on the answers a Potato study collects. The llm-sft trainer fine-tunes an LLM on written answers or corrected agent traces, llm-dpo tunes an LLM on pairwise choices, reward-model fits a scalar reward model on comparisons, and prm-reward fits a process reward model on step ratings. The page is written for teams that already collect preference or trace data in Potato and want a first model from those labels without exporting the data to a separate training script. Each section names the scheme that produces the data, gives a config that passes Potato's validator, lists the parameters worth changing, and says what the held-out score measures and whether the predictions go back to annotators as pre-labels. The model training overview covers the Training page, and Train Your First Model runs the whole loop with a classifier that needs no GPU.

Trainers by annotation scheme

Every trainer on this page learns from one annotation scheme per run, and the Training page offers a trainer only for the schemes the trainer can learn. The table below lists the schemes each trainer learns from:

Before you start

Potato's training extras install PyTorch, Transformers, and PEFT for the four trainers:

pip install "potato-annotation[train-llm]"
pip install "potato-annotation[train-text]"

The first extra covers llm-sft and llm-dpo, and the second covers reward-model and prm-reward. pip install "potato-annotation[train]" installs every training extra at once. A trainer whose packages are missing is listed on the Training page as "not installed", with the command that installs the packages.

Each trainer refuses to fit with fewer than 20 items in the train split (the min_instances parameter). The default split puts 70% of labeled items in train, 15% in val, and 15% in test, so plan on about 30 labeled items before the first run. The val and test items are held out of training and score the model.

llm-sft and llm-dpo start from Qwen/Qwen2.5-0.5B-Instruct, reward-model from distilbert/distilroberta-base, and prm-reward from answerdotai/ModernBERT-base. The LLM trainers are built to run on a GPU. When the machine running Potato has no GPU, Remote Training sends runs to a GPU server over ssh or to a SLURM cluster.

A study that collects preferences and rewrites

The first config asks two questions about each customer-support prompt. Annotators pick the better of two responses and write the answer an annotator would want from the assistant. The choice trains llm-dpo and reward-model, and the written answer trains llm-sft. Each line of data/answers.jsonl holds a prompt and the two responses:

{"id": "q001", "prompt": "How do I reset my router to factory settings?", "responses": ["Hold the reset button on the back for 10 seconds until the lights blink, then wait two minutes for the router to restart.", "Routers can be reset. Check the manual."]}

The config shows the prompt as the item's text and the two responses as cards A and B:

annotation_task_name: "Support answers: preferences and rewrites"
task_dir: .
data_files:
  - data/answers.jsonl
item_properties:
  id_key: id
  text_key: prompt
output_annotation_dir: annotation_output
user_config:
  allow_all_users: true
 
annotation_schemes:
  - annotation_type: pairwise
    name: preferred
    description: "Which response answers the question better?"
    mode: binary
    items_key: responses
    allow_tie: true
    tie_label: "No preference"
 
  - annotation_type: text
    name: rewrite
    description: "Write the answer you would want the assistant to give."
    multiline: true
    rows: 4
 
model_training:
  enabled: true
  writeback:
    schemas: [preferred]
    min_confidence: 0.7

text_key: prompt matters twice, because the prompt is what annotators read and what llm-sft puts after the question. items_key: responses names the list the pairwise cards come from, in data order, so A is the first response. writeback.schemas lists only preferred, so annotators see the model's choice and not a generated rewrite. Pre-labels anchor annotators, so add rewrite to the list only once the llm-sft run's held-out score supports showing the model's text. min_confidence: 0.7 keeps predictions the model is less sure of off the page while still storing the predictions. Check the config with Potato's validator before starting the server:

python -m potato.validate_cli config.yaml --strict

Start Potato, label at least 30 prompts, and open /admin/training as an admin. The runs below can also start from the API with the config's admin_api_key in $KEY.

Supervised fine-tuning with llm-sft

llm-sft turns each labeled item into one training example. The prompt is the scheme's description (the question annotators saw) followed by the item's text, and the target is the answer the annotator wrote. The loss covers only the answer's tokens, so the model learns to answer rather than to repeat the prompt. A LoRA adapter is trained by default instead of every weight.

Pick llm-sft and the rewrite scheme on the Training page and click Train now, or start the same run from the API:

curl -X POST -H "X-API-Key: $KEY" -H "Content-Type: application/json" \
  http://localhost:8000/admin/training/api/runs \
  -d '{"trainer": "llm-sft", "schemas": ["rewrite"], "params": {"max_new_tokens": 256}}'

Most runs change only the parameters in this table:

The run keeps the checkpoint with the lowest val loss. Because the run chose the checkpoint on val, the held-out score is read from test. The score is token F1, the overlap of words between the generated and the written answer regardless of order, with exact match reported beside the score. Token F1 credits an answer that uses the right words in the wrong places, so read the two metrics together.

The model generates an answer greedily for every val and test item. With writeback.include_unlabeled: true, the model also answers the items nobody has labeled. A prediction's confidence is the model's mean probability for the tokens the model generated, and the model review queue sorts by that confidence. With rewrite under writeback.schemas, the generated answer fills the text box when an annotator opens the item.

Free text is a weak fit for several annotators per item. The aggregation setting resolves several answers to one by comparing answers word for word, and two written answers rarely match word for word. For llm-sft on a text scheme, one writer per item avoids the word-for-word comparison.

Preference tuning with llm-dpo

Direct Preference Optimization (DPO) trains an LLM to raise the likelihood of the preferred response relative to the other response, measured against a frozen reference copy of the starting model. llm-dpo builds one example from each item answered A or B. The prompt is the item's prompt field, or the first of question, instruction, query, input, and context that the item has. Items answered with the tie option are left out, since a tie does not mark either response as better.

Pick llm-dpo and the preferred scheme, or send the same request as above with "trainer": "llm-dpo" and "schemas": ["preferred"]. With the default LoRA adapter, the reference model is the same model with the adapter switched off, so no second copy is held in memory. lora: false trains every weight and keeps a frozen copy as the reference, and the frozen copy doubles the memory.

The run keeps the checkpoint with the lowest DPO loss on val, and the run log reports how often the model prefers the annotators' choice on val. The held-out score is accuracy on test, over the items not answered as a tie, with macro-F1 beside the accuracy. A prediction is the response the trained model prefers, with the probability of that preference as the confidence. As a pre-label, the preferred response's card is selected when the annotator opens the item. With randomize_order on the scheme, the selection follows the response to wherever the annotator sees the response. The saved adapter is a policy as well and generates text when loaded with PEFT. For background on collecting the preference data itself, see Collecting RLHF and Preference Data and Pairwise Comparison.

A reward model with reward-model

A reward model maps a response to a number, for RLHF, for best-of-n sampling, or for ranking new outputs. reward-model trains a transformer with one output so that the response annotators preferred scores higher. The loss is the Bradley–Terry loss, -log sigmoid(r_winner - r_loser), the reward-modeling loss of InstructGPT. The preferred scheme above trains reward-model with no change to the config. Each comparative scheme gives a different training signal:

When an item has a prompt field, each reward is of the response given the prompt. A ranking's options are the scheme's labels, each scored given the item's text. The server builds BWS tuples from bws_config, so a run from the server's own data has the tuples, and the data files do not need the tuples.

Predictions answer the scheme's own question, so the predictions are scored and written back like any other scheme's. The pre-label is the selected card for a binary choice, the slider position for a scale, the list's order for a ranking, and the selected best and worst for BWS. Every prediction also stores the rewards the prediction came from. Only differences between rewards are meaningful. Pairwise and Best-Worst Scaling compares the comparative formats for collecting the data.

A study that collects step ratings and corrections

The second config works on agent traces. Annotators rate each step of the trace and then fix the wrong steps. The step ratings train prm-reward, and the corrections train llm-sft or llm-dpo. Each line of data/traces.jsonl holds the task, the agent's steps, and the final answer:

{"id": "traj_001", "task_description": "Find the current weather in San Francisco and report the temperature.", "steps": [{"thought": "I should look up the current weather.", "action": "web_search(queyr='San Francisco weather')"}, {"thought": "Open the first result.", "action": "open_url(results[0])"}, {"thought": "Read the temperature from the page.", "action": "extract_text('.temp')"}], "final_answer": "It is sunny."}

The trainers read the task from the item's task_description, task, or text field, in that order:

annotation_task_name: "Agent traces: step ratings and corrections"
task_dir: .
data_files:
  - data/traces.jsonl
item_properties:
  id_key: id
  text_key: task_description
output_annotation_dir: annotation_output
user_config:
  allow_all_users: true
 
annotation_schemes:
  - annotation_type: process_reward
    name: step_quality
    description: "Rate each step of the agent's trace"
    steps_key: steps
    step_text_key: action
    mode: per_step
    allow_neutral: true
    ai_prelabel: true
 
  - annotation_type: trajectory_edit
    name: corrected_trace
    description: "Fix the wrong steps, then correct the final answer"
    steps_key: steps
    step_text_key: action
    editable_fields: [action]
    show_diff: true
    edit_final_answer: true
    final_answer_key: final_answer
 
model_training:
  enabled: true
  writeback:
    schemas: [step_quality]
    include_unlabeled: true

mode: per_step with allow_neutral: true gives the three ratings prm-reward learns. ai_prelabel: true puts the "AI pre-label steps" button on the scheme, and the trained ratings appear through the button. writeback.schemas names only step_quality. Listing corrected_trace there stops the server from starting, and the validator gives the reason:

model_training.writeback.schemas names 'corrected_trace', a trajectory_edit scheme, which has no prelabel display: its predictions are stored and scored but annotators would never see them. Leave it out of writeback.schemas.

Repairing traces with llm-sft and llm-dpo

A trajectory_edit answer is a pair of traces, the agent's original steps and the steps after the annotator's corrections. Trajectory Editing for SFT/DPO covers the editor, and the trajectory editing post covers how the pairs become training data. Both trainers leave out traces nobody edited.

• llm-sft takes the task and the original steps as the prompt and the corrected steps as the target, so the model learns to repair a trace. Each step is written as Step N followed by the step's fields (action: ..., thought: ...), and the final answer follows the last step when the scheme has final_answer_key.
• llm-dpo takes the task as the prompt, the corrected steps as the preferred response, and the original steps as the other response. A prediction says whether the model prefers the correction (A) or the original (B), and is made only for held-out traces that have a correction.

Both runs report closer_to_correction as the held-out score. For llm-dpo, the score is the share of held-out traces where the model prefers the correction. For llm-sft, the score is the share where the written steps are closer to the correction than to the original by token F1, with token_f1 against the correction reported beside the score.

The predictions from a trajectory_edit run are stored and scored, and annotators never see the predictions. The editor always starts from the agent's original trace, so the page has no place to show a predicted rewrite. Read the predictions from the run page or from artifacts/predictions.jsonl in the run's directory.

Step ratings with prm-reward

A process reward model rates the steps of a trace rather than only the final answer, as in PRM800K. prm-reward feeds the whole trace to a transformer encoder as one input, the task followed by Step 1, Step 2, and so on. The encoder learns a three-way rating at each step's last token, so each rating is made with the steps before the step in view. Steps nobody rated are left out of the loss. The default model, answerdotai/ModernBERT-base, was chosen for the model's 8,192-token context.

Long traces lose their last steps. A trace longer than max_length tokens is cut from the end, and the steps past the cut are neither trained on nor rated. The run log counts the steps lost. When max_length is more than the model holds, the run lowers max_length to the model's limit with a warning, and the limit is 512 tokens for most BERT-class encoders. When the run log reports lost steps, raise max_length and choose a model whose context holds the longer traces.

The held-out score is step_macro_f1, macro-F1 over the three ratings on every rated step. Most steps in most traces are correct, so accuracy would flatter a model that never says "wrong". The run also reports step_accuracy and first_error_accuracy, the share of traces where the model's first wrong step is the annotators' first wrong step.

A prediction rates every step that fits in max_length, with each rating's probability as the rating's confidence and the least confident step as the trace's confidence. With the scheme in writeback.schemas and ai_prelabel: true, the "AI pre-label steps" button offers the trained ratings for the annotator to verify. An LLM judge configured under ai_support is not called for an item that has trained ratings. Process Reward Annotation covers the scheme, and the process reward models guide covers the data.

Fitting a larger model with QLoRA

QLoRA loads the base model's weights in 4 bits and trains a LoRA adapter on top. llm-sft, llm-dpo, hf-text-classification, vlm-sft, and reward-model take lora: {quantize: 4bit}, and prm-reward refuses the setting before loading the model. Plan the memory from the "Peak GPU memory" line that every transformer run logs at the end. The QLoRA section of the configuration reference describes how the base weights load and the 8-bit option.

Add this block to the config above to set QLoRA as the default for llm-sft:

model_training:
  enabled: true
  trainer_params:
    llm-sft:
      model_name: Qwen/Qwen2.5-7B-Instruct
      lora: {quantize: 4bit}
      gradient_checkpointing: true

QLoRA needs a CUDA GPU and a bitsandbytes build for that machine's CUDA, installed with pip install bitsandbytes. Without a working GPU or build, the run stops at validation before loading the model.

Publishing a run to the Hugging Face Hub

A successful run whose model is on the Potato server has Publish to the Hugging Face Hub on the run page. The push uploads the model and a README.md model card written from the run, with the trainer, the base model and adapter, the schemes and labels, the split sizes, the held-out score and what the score was measured on, and the run and bundle ids. A LoRA run saved only the adapter, so the Hub repository holds the adapter, and the card names the base model to load the adapter onto.

The token stays out of the browser. Create a write token in your Hugging Face settings and put the token in the server's environment before starting Potato:

export HF_TOKEN=hf_your_write_token
potato start config.yaml

The form names the environment variable, HF_TOKEN by default, asks for the repository name twice, and keeps Private ticked by default. Publishing to the Hugging Face Hub lists the files that go up, the checks before a push, and the license on the model card. The same push from the API:

curl -X POST -H "X-API-Key: $KEY" -H "Content-Type: application/json" \
  http://localhost:8000/admin/training/api/runs/RUN_ID/push-to-hub \
  -d '{"repo_id": "your-org/support-dpo", "confirm_repo_id": "your-org/support-dpo", "private": true, "licence": "apache-2.0"}'

The route returns 201 with the commit. A model left on a remote host with fetch_artifacts: false cannot be pushed from the Potato server, and the run page says where the model is. To move on from a published model to an active-learning loop, continue with Model in the Loop.

Further reading

• Model Training Configuration: every model_training key, the held-out scores, write-back, and the routes.
• Trainers: all 38 trainers, grouped by task.
• Remote Training: GPU servers over ssh, SLURM clusters, and the polling agent.
• Agent Trajectory Annotation: collecting trace data in Potato.
• Upstream reference sections for llm-sft, llm-dpo, agent traces, prm-reward, reward-model, QLoRA, and Hub publishing.

References

• Dettmers, T., Pagnoni, A., Holtzman, A., and Zettlemoyer, L. (2023). QLoRA: Efficient Finetuning of Quantized LLMs. arXiv:2305.14314. arxiv.org/abs/2305.14314
• Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., and Chen, W. (2021). LoRA: Low-Rank Adaptation of Large Language Models. arXiv:2106.09685. arxiv.org/abs/2106.09685
• Lightman, H., Kosaraju, V., Burda, Y., Edwards, H., Baker, B., Lee, T., Leike, J., Schulman, J., Sutskever, I., and Cobbe, K. (2023). Let's Verify Step by Step. arXiv:2305.20050. arxiv.org/abs/2305.20050
• Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., Schulman, J., Hilton, J., Kelton, F., Miller, L., Simens, M., Askell, A., Welinder, P., Christiano, P., Leike, J., and Lowe, R. (2022). Training Language Models to Follow Instructions with Human Feedback. arXiv:2203.02155. arxiv.org/abs/2203.02155
• Rafailov, R., Sharma, A., Mitchell, E., Ermon, S., Manning, C. D., and Finn, C. (2023). Direct Preference Optimization: Your Language Model is Secretly a Reward Model. arXiv:2305.18290. arxiv.org/abs/2305.18290