# Medical Text Annotation Services vs In-House Annotation Tools

Source: https://www.potatoannotator.com/docs/guides/medical-text-annotation-services-vs-tools

**A medical text annotation service provides annotators with clinical training and returns labeled clinical notes for a price quoted per project. An in-house annotation tool provides only the software, and your team recruits the clinicians, obtains the approvals, and runs the quality checks. Pay a vendor when your clinicians cannot label the volume and the notes may go to a third party, and annotate in-house when the notes cannot leave your institution or the labels need your own clinicians' judgment.**

Medical text annotation is the labeling of clinical notes, discharge summaries, radiology reports, and other [electronic health record](https://en.wikipedia.org/wiki/Electronic_health_record) text. The labels train clinical language models or score the models' output. The common tasks are listed below, with the shared task that set a benchmark where one exists:

- **Clinical [named-entity recognition](https://en.wikipedia.org/wiki/Named-entity_recognition).** Annotators mark each mention of a problem, test, treatment, or medication. The [2010 i2b2/VA challenge](https://doi.org/10.1136/amiajnl-2011-000203) covered concepts, assertions, and relations in clinical text.
- **[Relation extraction](https://en.wikipedia.org/wiki/Relationship_extraction).** Annotators link two mentions, such as a drug and the drug's dose, or a finding and the body site of the finding.
- **Assertion and negation.** Annotators record whether a mentioned problem is present, absent, or uncertain, and whether the problem belongs to the patient or to a family member. In "denies chest pain", chest pain is absent.
- **Temporal annotation.** Annotators mark [temporal expressions](https://en.wikipedia.org/wiki/Temporal_expressions) and clinical events and order the events in time, as in the [2012 i2b2 temporal relations challenge](https://doi.org/10.1136/amiajnl-2013-001628).
- **[De-identification](https://en.wikipedia.org/wiki/De-identification).** Annotators mark [protected health information](https://en.wikipedia.org/wiki/Protected_health_information) such as names, dates, and medical record numbers, as in [Track 1 of the 2014 i2b2/UTHealth shared task](https://doi.org/10.1016/j.jbi.2015.06.007).
- **Coding and normalization.** Annotators map each mention to a concept in a terminology such as the [UMLS](https://en.wikipedia.org/wiki/Unified_Medical_Language_System), [SNOMED CT](https://en.wikipedia.org/wiki/SNOMED_CT), or [ICD-10](https://en.wikipedia.org/wiki/ICD-10).

A medical text annotation service is a company that staffs annotators for the tasks above and delivers the labels. The same phrase also names automatic concept taggers such as [DrNote](https://doi.org/10.1371/journal.pdig.0000086), an open-source service that links medical text to Wikidata entities. The guide below covers labeling by people.

Potato is a free, open-source annotation tool (GPL-3.0-or-later) that runs on your own server, and the Potato team does not provide annotators. [Medical Image and Clinical Text Annotation Tools Compared](/docs/guides/medical-annotation-tools-compared) compares the tools for the in-house route. The comparison includes INCEpTION and MedCATtrainer for clinical text. [Expert vs Crowd Annotation](/docs/guides/expert-vs-crowd-annotation) covers who should label, and [Build or Buy an Annotation Tool](/docs/guides/build-or-buy-annotation-tool) covers the software decision for any data type.

## Clinical text services named on vendor pages

Vendor pages for clinical text name the tasks on offer, the workforce, and the security standards the vendor claims. The Potato team read each vendor's own medical or healthcare text annotation page on November 17, 2026. The table below lists the vendors in alphabetical order and records what each page states, with each compliance claim given as the page words the claim:

| Vendor | Clinical text work named on the page | Workforce, as stated | Compliance, as stated | Price on the page |
|---|---|---|---|---|
| [AnnotationBox](https://annotationbox.com/text-annotation/healthcare/) | Medical entity recognition, radiology report annotation, health record annotation, and healthcare text classification | 1000+ trained experts | No compliance standard named | By quote |
| [Cogito Tech](https://www.cogitotech.com/industries/medical-text-annotation/) | Clinical and biomedical entity recognition, linking to UMLS, ICD-10, and SNOMED, de-identification, and medical record annotation | 1500+ data and industry experts, with language and subject matter experts in 100+ languages and dialects | The page states that Cogito Tech adheres to GDPR, HIPAA, and ISO 27001 standards | No price, contact form |
| [iMerit](https://imerit.ai/medical-text-and-audio-ai/) | Localizing, classifying, and linking patients, providers, diseases, treatments, and outcomes in longitudinal medical records | US board-certified physicians for validation and benchmarking | The page states that iMerit workflows meet HIPAA, SOC-2, CE, and FDA regulatory compliance needs | No price |
| [LXT](https://www.lxt.ai/services/text-annotation/) | Clinical documentation annotation, symptom detection in notes, and de-identification of patient records | 10 million+ contributors across LXT and subsidiary clickworker, including 250K+ domain experts across 100+ fields | The page states ISO 27001-certified facilities and compliance with GDPR and HIPAA standards | No price |
| [Shaip](https://www.shaip.com/solutions/medical-data-annotation/) | Clinical entity recognition, relations, assertion (status, negation, and subject), temporal and section annotation, and coding to SNOMED, RxNorm, ICD-10-CM, and CPT with the supporting text marked | Credentialed domain experts, from a 10,000+ in-house workforce and 500K+ crowd contributors | The page lists GDPR, HIPAA, ISO 9001:2015, SOC 2 Type II, and ISO 27001, and states HIPAA Safe Harbor and Expert Determination de-identification | Customized pricing, no figures |

LXT's entry comes from a general text annotation page with a one-line healthcare item, and the other four pages are about healthcare work. Every compliance entry above is the vendor's own statement. Ask for the audit report, the certificate, or the signed agreement behind any claim the project relies on. iMerit's page reports 99% annotation quality from one case study, and AnnotationBox's page reports 95%+ accuracy. Neither page says how the figure was measured.

## Questions to ask a medical text annotation vendor

The answers to the questions below determine whether the notes may go to the vendor at all and whether the returned labels can be trusted. Ask the questions before signing:

- **Annotator credentials.** Ask who labels and who reviews, how many annotators label each note, and each person's clinical role (nurse, medical coder, physician, or trained non-clinician).
- **Where the notes are processed.** Ask which country and which facility the notes reach, whether annotators work from home or from a controlled site, whether subcontractors or a crowd platform see the text, and when the vendor deletes the notes after delivery.
- **A business associate agreement.** Under [HIPAA](https://en.wikipedia.org/wiki/Health_Insurance_Portability_and_Accountability_Act), a covered entity needs a business associate agreement (BAA) with a contractor that receives protected health information on the entity's behalf. Confirm with your privacy office whether the project needs a BAA or whether only de-identified notes leave the institution.
- **Guideline ownership.** Ask who writes the annotation guidelines and who owns the guidelines and the labels afterward. Confirm that the contract lets you publish the guidelines and bars reuse of your notes in the vendor's own datasets.
- **Agreement reporting.** Ask which agreement coefficient the vendor reports, on which task, and between how many annotators per note. Request each annotator's labels on a double-annotated sample instead of one merged answer.
- **Gold-set checks.** Ask whether the vendor scores annotators against a gold set that you write, and whether you see the scores.
- **Sample work and pricing unit.** Send your hardest notes with your guidelines and score the returned labels before committing the budget. Ask whether the price is per note, per entity, per code, or per annotator hour, and what a rework round costs.

The Partnership on AI publishes a [responsible sourcing library](https://partnershiponai.org/responsible-sourcing-library/) for teams buying annotation work. The library includes guidance on worker pay and a vendor comparison template.

## In-house clinical text annotation with Potato

Running the work in-house means your team owns each part of the study described below. Potato covers the software, and your team covers the people and the time.

**Approvals and de-identification.** An [institutional review board](https://en.wikipedia.org/wiki/Institutional_review_board) protocol or a data use agreement usually governs who may read the notes. De-identify the notes before anyone outside the approved team sees the text. HIPAA recognizes two de-identification methods, Safe Harbor (removing or generalizing 18 listed elements) and Expert Determination (a documented risk analysis). Potato does not de-identify notes, and Potato has no HIPAA compliance feature.

**The annotators.** Clinician time is the main cost. Run a short pilot to measure how long each note takes, and turn the per-note time into a budget with the [annotation cost calculator](/tools/annotation-cost). A [training phase](/docs/features/training-phase) gives annotators practice notes with feedback and admits only annotators who pass. [Gold items](/docs/guides/gold-standards-and-attention-checks) mixed into the real task score accuracy throughout the study, and [category-based assignment](/docs/features/category-assignment) sends each annotator only the note types the annotator qualified for in training.

**The tool.** A `span` scheme marks entities, assertion cues, and de-identification targets, and a `span_link` scheme draws relations between spans ([Span Linking](/docs/annotation-types/span-linking)). [Entity linking](/docs/annotation-types/entity-linking) attaches a concept to each span from the UMLS, from Wikidata, or from a terminology service on your own network through the `rest` type. The `umls` type searches NLM's UTS service with your UMLS API key, and each search string then leaves the network. Potato ships no SNOMED CT or ICD-10 files. To code to either system, point the `rest` type at a terminology service your institution runs. For OBO ontologies such as the Human Phenotype Ontology, a `label_source` block searches a local `.obo` file offline, or searches EMBL-EBI's OLS4 only when the scheme sets `allow_network: true` ([Ontology Term Labels](/docs/features/ontology-labels)).

**Agreement.** `num_annotators_per_item` sets how many annotators label each note. For a `span` scheme, the agreement report at `/admin/iaa` gives token-level kappa, exact and partial span F1, a character-level Krippendorff's alpha, and a simplified form of Mathet's gamma, as [Agreement for Spans](/docs/guides/agreement-for-spans) explains.

**Data handling.** The notes stay on the server you run, and the whole setup runs air-gapped once any model weights are on the machine ([Air-Gapped Annotation](/docs/guides/air-gapped-annotation)). Model suggestions through [AI Support](/docs/features/ai-support) stay on your server with `endpoint_type: ollama` or `vllm`. With `labeling_engine.features.pii_redaction` on, the [labeling engine](/docs/labeling-engine/configuration) masks emails, URLs, IP addresses, card numbers, US Social Security numbers, phone numbers, and @handles in text sent to a third-party model API, and `labeling_engine.redaction.categories` picks which of the seven to mask. The patterns do not find patient names, dates, or record numbers. Potato's source code states that the redaction patterns are not a compliance tool, and a study whose notes must not leave the site should list only local model sources.

## Choosing between a service and in-house annotation

Each common situation below is paired with the route that usually fits the situation:

| Situation | Usual fit |
|---|---|
| More notes than your clinicians can label in time, and notes cleared to go to a vendor under a BAA or after de-identification | Service |
| A fixed deadline weeks away and no time to recruit or train | Service |
| Notes that your IRB protocol or data use agreement keeps inside the institution | In-house |
| Labels that need your own specialists, such as oncology staging or a local coding policy | In-house |
| Guidelines that will change as the team reads the notes | In-house |
| A published corpus that must report agreement and how the annotators were trained | In-house, or a service that delivers per-annotator labels |
| Vendor capacity with in-house control over quality | Both, combined as below |

## Checking a vendor's labels yourself

A vendor's agreement figure describes the vendor's annotators against each other. Your own check measures the vendor's labels against your clinicians' reading of the same notes. Run the check on a sample in five steps:

1. **Draw a random sample** of delivered notes, stratified by note type when the delivery mixes discharge summaries, radiology reports, and progress notes. [How Many Annotators Do You Need?](/docs/guides/how-many-annotators) covers sample size.
2. **Label the sample independently.** Two of your clinicians label the sample in Potato with the vendor's guidelines, without seeing the vendor's spans.
3. **Compute agreement.** Put the vendor's spans and your annotators' spans into one CSV with item, annotator, label, start, and end columns, and the [span agreement calculator](/tools/span-and-box-agreement) reports exact-match F1, partial-match F1, and a character-level Krippendorff's alpha in the browser. For assertion or document labels, the [agreement calculator](/tools/inter-annotator-agreement) computes Cohen's kappa and Krippendorff's alpha.
4. **Compare against your own pair.** Agreement between your two clinicians shows how consistently the task can be done on your notes. Vendor-to-clinician agreement well below the clinicians' own agreement points to vendor errors or to a guideline the vendor reads differently.
5. **Read the disagreements by label.** Fix a confusion between two labels or a run of missed negations with a guideline revision before the next batch.

Character offsets have to refer to the same note text on both sides. Confirm that the vendor's delivery did not change whitespace, line endings, or encoding before computing agreement.

## Hybrid setups with a vendor and your own server

Two hybrid patterns let a team buy capacity and keep control of quality.

**Correct the vendor's delivery.** `potato import` reads brat standoff, CoNLL, doccano, and Prodigy files and builds a Potato project with the vendor's spans as pre-annotations for your clinicians to correct. brat relations and events are listed in the import warnings rather than imported. A delivery with relations needs the relations redrawn in Potato ([Importing a Project](/docs/tools/importing-projects#brat)).

**Bring the vendor's annotators into your own Potato server.** Potato's expert invite provider gives each contracted annotator a private link with a token. The vendor's annotators then label on your server, and your training phase, gold items, and agreement report apply to the vendor's annotators in the same way as to your own. The vendor's staff still read the notes, and the BAA and de-identification questions above apply in the same way. The following minimal config marks problems, tests, treatments, and negation cues in discharge summaries, links each negation cue to the problem the cue negates, and links each span to a terminology service inside the hospital network:

```yaml
annotation_task_name: "Discharge summary problems and negation"
task_dir: "."
data_files:
  - "data/notes.jsonl"
item_properties:
  id_key: id
  text_key: text
output_annotation_dir: "annotation_output/"

login:
  type: url_direct
  url_argument: invite

crowdsourcing:
  provider: expert
  expert:
    invites_file: "expert_invites.yaml"

# Two annotators per note, so every note has a pair to compare
num_annotators_per_item: 2

annotation_schemes:
  - annotation_type: span
    name: concepts
    description: "Highlight each problem, test, and treatment, and each negation cue such as 'denies'."
    labels: [Problem, Test, Treatment, Negation cue]
    entity_linking:
      enabled: true
      knowledge_bases:
        # A terminology service inside the hospital network
        - name: local_terms
          type: rest
          base_url: "http://terminology.internal:8080"
      auto_search: false

  - annotation_type: span_link
    name: negations
    description: "Link each negation cue to the problem it negates."
    span_schema: concepts
    link_types:
      - name: negates
        directed: true
        allowed_source_labels: ["Negation cue"]
        allowed_target_labels: ["Problem"]
```

`expert_invites.yaml` maps each token to a display name, and Potato rejects any token missing from the file. Send each annotator `https://your-server/?invite=<token>`. With `auto_search: false`, an annotator opens the concept search from a span's link icon, and a negation cue triggers no search unless the annotator asks for one. The admin dashboard reports annotations and time on task per annotator, and the [expert annotator documentation](https://github.com/davidjurgens/potato/blob/master/docs/deployment/expert_annotators.md) covers token generation.

## Frequently asked questions

### What is medical text annotation?

Medical text annotation is the labeling of clinical notes, radiology reports, and other health record text with entities, relations, assertion status, time expressions, protected health information, or codes from a terminology such as the UMLS, SNOMED CT, or ICD-10. The labels train or evaluate clinical natural language processing models. Clinicians, medical coders, or trained annotators do the labeling, either in-house with an annotation tool or through a paid annotation service.

### Which companies offer medical text annotation services?

AnnotationBox, Cogito Tech, iMerit, LXT, and Shaip are among the companies whose own pages offer clinical or medical text annotation. The section "Clinical text services named on vendor pages" records the tasks, workforce, compliance claims, and pricing each page stated on November 17, 2026. None of the five pages publishes a price. Before choosing, send sample notes with your guidelines and score the returned labels against your own clinicians' labels.

## Further reading

- [Medical Image and Clinical Text Annotation Tools Compared](/docs/guides/medical-annotation-tools-compared), for choosing the in-house tool
- [Potato for healthcare and clinical annotation](/use-cases/healthcare)
- [Audio Annotation Services vs In-House Annotation Tools](/docs/guides/audio-annotation-services-vs-tools)
- [Named Entity Recognition](/docs/guides/named-entity-recognition)
- [Relation and Event Extraction](/docs/guides/relation-and-event-extraction)
- [Entity Linking](/docs/guides/entity-linking)
- [Inter-Annotator Agreement Explained](/docs/guides/inter-annotator-agreement)
- [Adjudication and Resolving Disagreement](/docs/guides/adjudication-and-disagreement)
- [Expert vs Crowd Annotation](/docs/guides/expert-vs-crowd-annotation)

## References

1. Uzuner, Ö., South, B. R., Shen, S., & DuVall, S. L. (2011). 2010 i2b2/VA challenge on concepts, assertions, and relations in clinical text. *Journal of the American Medical Informatics Association*, 18(5), 552-556. https://doi.org/10.1136/amiajnl-2011-000203
2. Sun, W., Rumshisky, A., & Uzuner, O. (2013). Evaluating temporal relations in clinical text: 2012 i2b2 Challenge. *Journal of the American Medical Informatics Association*, 20(5), 806-813. https://doi.org/10.1136/amiajnl-2013-001628
3. Stubbs, A., Kotfila, C., & Uzuner, Ö. (2015). Automated systems for the de-identification of longitudinal clinical narratives: Overview of 2014 i2b2/UTHealth shared task Track 1. *Journal of Biomedical Informatics*, 58, S11-S19. https://doi.org/10.1016/j.jbi.2015.06.007
4. Frei, J., Soto-Rey, I., & Kramer, F. (2022). DrNote: An open medical annotation service. *PLOS Digital Health*, 1(8), e0000086. https://doi.org/10.1371/journal.pdig.0000086
