# Decision Models as a Cheap First Labeling Tier

Source: https://www.potatoannotator.com/blog/decision-models-first-labeling-tier

**A decision model labels a fixed-choice question in one forward pass and returns a probability for every option, with no text to generate. Use one as the first tier of a labeling cascade on one-of-N questions with a handful of options, send the items it is unsure about to an LLM, and certify its threshold on a blind human sample, because its confidence can stay high on inputs it gets wrong.**

A *decision model* is a classifier that takes a piece of text and a set of typed questions and returns a probability for each answer. It generates no text, so it cannot write a prompt, a rationale or a rule. Laya is one open model of this kind. Its [model card](https://huggingface.co/convaiinnovations/laya) lists an English checkpoint on a ModernBERT-large encoder with 421M parameters under the Apache 2.0 license, and a multilingual checkpoint with 322M parameters. The `/v1/systemone` API serves the same kind of answer over HTTP, from hosted Jev, OpenJev or Laya's own server.

## Where a first tier saves money

A labeling *cascade* tries models in order, usually from cheapest to most capable, and passes an item on only when the current model is not confident enough. When a corpus has a large share of easy items, a cheap model that labels those confidently leaves the LLM a smaller, harder remainder. The saving depends on that share, so measure it on your own data before planning around it.

A decision model suits the first tier because it answers every option in one pass, so a question with five labels costs one forward pass and no generated text. Its output probabilities are the confidence score, so it needs no follow-up question or repeated sampling. It also runs on a CPU. The model card reports 193 to 464 ms per request on CPU with the checkpoints preloaded, and 39.5 ms for the English checkpoint on a T4 GPU.

## Limits reported for Laya

Four of these limits come from the Laya model card, and the fifth, on rating scales, comes from Potato's documentation:

- **Many options.** On the Banking77 intent benchmark the card reports Laya at 0.425 accuracy on 77 labels, against 0.870 for Jev on 72. Options share a fixed token budget, so long label lists crowd each other out. Potato's documentation also notes that Laya warns its temperature is out of range on questions with 11 or more options and clamps it.
- **Other scripts on the English checkpoint.** The card reports the English checkpoint at 0.000 accuracy and 0.952 confidence on Khmer text. A confidence threshold cannot catch an error the model is sure about.
- **Zero-shot accuracy.** The card reports the base checkpoints near chance on its typed-decisions benchmark zero-shot, at 0.362 against a 0.318 random and a 0.461 majority-class baseline, and calls Laya "a fast base to specialise, not a zero-shot decision engine".
- **Raw calibration.** The card reports that `laya` ships over-confident, with a mean expected calibration error of 0.466 that falls to 0.081 after refitting temperatures on your own data. Calibration varies by domain, so a threshold chosen on someone else's data does not transfer.
- **Rating scales.** Potato's documentation finds that on `likert` and `slider` schemes Laya's scores can barely follow the scale, whatever the wording of the question.

Taken together, the limits suggest keeping the decision model on one-of-N questions with a few options, in the language its checkpoint was trained for. Choose its threshold from human labels on your own corpus.

## A decision model in Potato

Potato 2.11 adds two endpoint types for decision models. `laya` runs Laya inside the Potato process and needs the `decision` extra, `pip install 'potato-annotation[decision]'`, which pulls in torch and transformers. `systemone` talks to any server with the `/v1/systemone` API and needs only a `base_url`. Both work on `radio`, `select`, `likert` and `slider` schemes, and Solo Mode does not start when one is set to label a `multiselect`, span or free-text scheme.

In the [labeling engine](/docs/labeling-engine/overview), a decision model is a `decision` source placed first in the cascade. This excerpt from the text example that ships with Potato puts Laya ahead of a small Ollama LLM and certifies the pair against a 10% error target at Potato's default 90% confidence:

```yaml
labeling_engine:
  enabled: true
  sources:
    - id: laya
      kind: decision
      endpoint_type: laya
      model: convaiinnovations/laya
      options:
        device: cpu
    - id: llm
      kind: llm
      endpoint_type: ollama
      model: llama3.2:3b
      base_url: http://localhost:11434
      max_tokens: 200
  cascade: [laya, llm]
  contracts:
    - id: accuracy
      kind: selective_risk
      schema: sentiment
      error: 0.1
      confidence: 0.9
```

The engine chooses Laya's confidence threshold on half of a blind human sample and tests it on the other half, so the threshold comes from your corpus rather than from the model card. Until the dev half of the sample fixes a threshold for Laya, the LLM labels every item. Afterwards the LLM sees only the items Laya is not sure about, plus the calibration sample.

In the run documented in the example's README, with Gemma 12B as the LLM, Laya's threshold was never fixed and the LLM delivered all of the accepted labels, so measure what the first tier contributes on your own corpus before counting on it. Because the engine tests each tier's accepted labels separately, a confident failure such as the Khmer case shows up as errors in the certification sample, provided the sample covers that kind of input. The `distributional` contract can also use Laya's full probability vector, which suits subjective questions where annotators split.

In [Solo Mode](/docs/features/solo-mode#decision-models), a decision model sits in `labeling_models` and as the first tier of `confidence_routing`. Solo Mode also needs a generative model under `revision_models`, because decision models cannot write the prompts and rules its revision loop produces.

## Testing a decision model before a full run

Run the model on a few dozen items you have already labeled before routing a whole corpus through it. Solo Mode's **Try on 10 items** button does this for a prompt, and in the engine `potato label run` followed by `potato label status` shows how many more labels each tier's threshold needs and which source produced errors. A tier that fails certification accepts nothing, and it sends the items above its threshold to people instead of to the LLM, which spends the human-label budget on items the LLM might have labeled. If the decision model's tier keeps failing certification, drop it from the cascade.

## Further reading

- [Labeling engine configuration](/docs/labeling-engine/configuration), for every `decision` source option
- [Certifying LLM Labels with a Blind Human Sample](/blog/certifying-llm-labels), on certifying a threshold against a blind sample
- [Solo Mode](/docs/features/solo-mode), for confidence routing with one expert