Docs/Annotation Types

Multirate (Matrix Rating)

Rate multiple items on the same scale in a matrix format.

Multirate (Matrix Rating) Annotation

The multirate type displays multiple items in a matrix format where each item is rated on the same scale. Perfect for evaluating multiple dimensions of a single item.

Basic Configuration

yaml

annotation_schemes:
  - name: "aspect_ratings"
    description: "Rate each aspect of the response"
    annotation_type: "multirate"
    labels:
      - name: "Accuracy"
      - name: "Clarity"
      - name: "Helpfulness"
    options:
      - name: "1"
      - name: "2"
      - name: "3"
      - name: "4"
      - name: "5"

Configuration Options

Field	Type	Required	Description
`name`	string	Yes	Unique identifier for the annotation
`description`	string	Yes	Instructions shown to annotators
`annotation_type`	string	Yes	Must be `"multirate"`
`labels`	array	Yes	Items to be rated (rows)
`options`	array	Yes	Rating scale options (columns)
`size`	number	No	Alternative to options: number of scale points
`min_label`	string	No	Label for lowest rating
`max_label`	string	No	Label for highest rating
`randomize`	boolean	No	Randomize item order
`compact`	boolean	No	Use compact layout

Examples

Response Quality Assessment

yaml

- name: "quality_assessment"
  description: "Rate each aspect of the AI response"
  annotation_type: "multirate"
  labels:
    - name: "Accuracy"
      tooltip: "Is the information factually correct?"
    - name: "Completeness"
      tooltip: "Does it fully address the question?"
    - name: "Clarity"
      tooltip: "Is it easy to understand?"
    - name: "Relevance"
      tooltip: "Does it stay on topic?"
  size: 5
  min_label: "Poor"
  max_label: "Excellent"

Translation Quality

yaml

- name: "translation_quality"
  description: "Evaluate the translation quality"
  annotation_type: "multirate"
  labels:
    - name: "Fluency"
    - name: "Adequacy"
    - name: "Terminology"
    - name: "Style"
  options:
    - name: "1 - Unacceptable"
    - name: "2 - Poor"
    - name: "3 - Acceptable"
    - name: "4 - Good"
    - name: "5 - Excellent"

Product Review Dimensions

yaml

- name: "product_dimensions"
  description: "Rate each aspect of the product"
  annotation_type: "multirate"
  labels:
    - name: "Build Quality"
    - name: "Value for Money"
    - name: "Ease of Use"
    - name: "Customer Support"
    - name: "Documentation"
  size: 5
  min_label: "Very Poor"
  max_label: "Excellent"
  randomize: true

Output Format

The multirate annotation outputs a dictionary mapping each item to its rating:

json

{
  "id": "item_1",
  "annotations": {
    "aspect_ratings": {
      "Accuracy": "4",
      "Clarity": "5",
      "Helpfulness": "3"
    }
  }
}

Use Cases

LLM Evaluation: Rate responses on multiple quality dimensions
Translation Assessment: Evaluate fluency, adequacy, and terminology
Product Reviews: Capture ratings across different product aspects
Survey Research: Likert-style matrix questions
Peer Review: Rate papers on multiple criteria

Best Practices

Limit the number of items - 3-7 items works best; more causes fatigue
Use consistent scales - All items should use the same rating scale
Order items logically - Group related dimensions together
Provide clear definitions - Use tooltips to explain each dimension
Consider randomization - Prevents order bias in responses

PreviousSelect (Dropdown)NextPure Display