Skip to content
Guides6 min read

Decision Models as a Cheap First Labeling Tier

A decision model scores every option in one forward pass and writes no text. It works as a first labeling tier before an LLM on questions with few options.

Potato Team

A decision model labels a fixed-choice question in one forward pass and returns a probability for every option, with no text to generate. Use one as the first tier of a labeling cascade on one-of-N questions with a handful of options, send the items it is unsure about to an LLM, and certify its threshold on a blind human sample, because its confidence can stay high on inputs it gets wrong.

A decision model is a classifier that takes a piece of text and a set of typed questions and returns a probability for each answer. It generates no text, so it cannot write a prompt, a rationale or a rule. Laya is one open model of this kind. Its model card lists an English checkpoint on a ModernBERT-large encoder with 421M parameters under the Apache 2.0 license, and a multilingual checkpoint with 322M parameters. The /v1/systemone API serves the same kind of answer over HTTP, from hosted Jev, OpenJev or Laya's own server.

Where a first tier saves money

A labeling cascade tries models in order, usually from cheapest to most capable, and passes an item on only when the current model is not confident enough. When a corpus has a large share of easy items, a cheap model that labels those confidently leaves the LLM a smaller, harder remainder. The saving depends on that share, so measure it on your own data before planning around it.

A decision model suits the first tier because it answers every option in one pass, so a question with five labels costs one forward pass and no generated text. Its output probabilities are the confidence score, so it needs no follow-up question or repeated sampling. It also runs on a CPU. The model card reports 193 to 464 ms per request on CPU with the checkpoints preloaded, and 39.5 ms for the English checkpoint on a T4 GPU.

Limits reported for Laya

Four of these limits come from the Laya model card, and the fifth, on rating scales, comes from Potato's documentation:

  • Many options. On the Banking77 intent benchmark the card reports Laya at 0.425 accuracy on 77 labels, against 0.870 for Jev on 72. Options share a fixed token budget, so long label lists crowd each other out. Potato's documentation also notes that Laya warns its temperature is out of range on questions with 11 or more options and clamps it.
  • Other scripts on the English checkpoint. The card reports the English checkpoint at 0.000 accuracy and 0.952 confidence on Khmer text. A confidence threshold cannot catch an error the model is sure about.
  • Zero-shot accuracy. The card reports the base checkpoints near chance on its typed-decisions benchmark zero-shot, at 0.362 against a 0.318 random and a 0.461 majority-class baseline, and calls Laya "a fast base to specialise, not a zero-shot decision engine".
  • Raw calibration. The card reports that laya ships over-confident, with a mean expected calibration error of 0.466 that falls to 0.081 after refitting temperatures on your own data. Calibration varies by domain, so a threshold chosen on someone else's data does not transfer.
  • Rating scales. Potato's documentation finds that on likert and slider schemes Laya's scores can barely follow the scale, whatever the wording of the question.

Taken together, the limits suggest keeping the decision model on one-of-N questions with a few options, in the language its checkpoint was trained for. Choose its threshold from human labels on your own corpus.

A decision model in Potato

Potato 2.11 adds two endpoint types for decision models. laya runs Laya inside the Potato process and needs the decision extra, pip install 'potato-annotation[decision]', which pulls in torch and transformers. systemone talks to any server with the /v1/systemone API and needs only a base_url. Both work on radio, select, likert and slider schemes, and Solo Mode does not start when one is set to label a multiselect, span or free-text scheme.

In the labeling engine, a decision model is a decision source placed first in the cascade. This excerpt from the text example that ships with Potato puts Laya ahead of a small Ollama LLM and certifies the pair against a 10% error target at Potato's default 90% confidence:

yaml
labeling_engine:
  enabled: true
  sources:
    - id: laya
      kind: decision
      endpoint_type: laya
      model: convaiinnovations/laya
      options:
        device: cpu
    - id: llm
      kind: llm
      endpoint_type: ollama
      model: llama3.2:3b
      base_url: http://localhost:11434
      max_tokens: 200
  cascade: [laya, llm]
  contracts:
    - id: accuracy
      kind: selective_risk
      schema: sentiment
      error: 0.1
      confidence: 0.9

The engine chooses Laya's confidence threshold on half of a blind human sample and tests it on the other half, so the threshold comes from your corpus rather than from the model card. Until the dev half of the sample fixes a threshold for Laya, the LLM labels every item. Afterwards the LLM sees only the items Laya is not sure about, plus the calibration sample.

In the run documented in the example's README, with Gemma 12B as the LLM, Laya's threshold was never fixed and the LLM delivered all of the accepted labels, so measure what the first tier contributes on your own corpus before counting on it. Because the engine tests each tier's accepted labels separately, a confident failure such as the Khmer case shows up as errors in the certification sample, provided the sample covers that kind of input. The distributional contract can also use Laya's full probability vector, which suits subjective questions where annotators split.

In Solo Mode, a decision model sits in labeling_models and as the first tier of confidence_routing. Solo Mode also needs a generative model under revision_models, because decision models cannot write the prompts and rules its revision loop produces.

Testing a decision model before a full run

Run the model on a few dozen items you have already labeled before routing a whole corpus through it. Solo Mode's Try on 10 items button does this for a prompt, and in the engine potato label run followed by potato label status shows how many more labels each tier's threshold needs and which source produced errors. A tier that fails certification accepts nothing, and it sends the items above its threshold to people instead of to the LLM, which spends the human-label budget on items the LLM might have labeled. If the decision model's tier keeps failing certification, drop it from the cascade.

Further reading