Skip to content
Tutorials16 min read

Correct Transkribus and eScriptorium HTR Output Line by Line

Have two transcribers check each HTR line against the page image in Potato, export PAGE XML for retraining, and compute the model's character error rate.

Potato Team

Archivists, historians, and editors with handwritten text recognition (HTR) output from Transkribus or eScriptorium can follow this tutorial to have two transcribers correct every recognized line in Potato 2.13.1, write the corrections back into PAGE XML for retraining, and compute the model's character error rate from the corrections. PAGE XML is the XML format in which Transkribus, eScriptorium, and the Kraken engine store one page image's regions, text lines, and recognized text. Potato makes one item per text line, crops the line out of the page image, and puts the recognized text in an editable box below the crop. The worked example uses four pages of 17th-century council minutes with a ground-truth transcription of every line. The error rates on this page can therefore be checked against an answer key. Digital humanities formats documents every import and export option, and this page follows one correction project from the PAGE files to the corrected files.

The pages and the recognized text

The example pages come from the Ratsprotokolle of Bozen, the minutes of the town council's meetings from 1470 to 1805. The minutes are handwritten in Early Modern German. The READ project released 450 of the pages and a ground-truth transcription of each line in PAGE XML as the HTR Dataset ICFHR 2016, under CC BY 4.0. The test set is Test-ICFHR-2016.tgz (59 MB), with 50 page images and a page/ folder of PAGE files, one per image. Potato's importer expects that folder layout from a Transkribus export. The tutorial uses pages 405 to 408. The four pages have 88 text lines and 1,732 characters of ground truth.

The dataset's PAGE files contain the ground truth rather than a model's reading. The Potato team therefore ran an HTR model over the dataset's own line boundaries to make the text to correct, and did not use Transkribus. The model is Stefan Weil's German handwriting model for Kraken, and Weil released the model under CC BY-SA 4.0. Kraken is the recognition engine inside eScriptorium. Kraken looks for the page image beside the XML file, so copy the four PAGE files out of page/ first:

bash
pip install kraken==7.1.1
cd Test-ICFHR-2016
mkdir -p ../htr/page
for n in 0405 0406 0407 0408; do
  cp page/Seite$n.xml .
  kraken -f page -x -i Seite$n.xml ../htr/page/Seite$n.xml ocr -m ../german_handwriting.mlmodel
  cp Seite$n.JPG ../htr/
done

-f page reads the line polygons from the PAGE file instead of finding lines again, and -x writes PAGE XML. The htr/ folder then has the Transkribus layout, with the images at the top and the PAGE files in htr/page/. Kraken wrote a confidence for each line, and the confidence came to 0.95 or more on 52 of the 88 lines. Against the ground truth, Kraken's text had a character error rate of 8.4% over all 50 test pages. The model's training data includes the dataset's 400 training pages, and the 50 test pages are not among the training pages. Kraken has therefore seen this collection's hands, and a collection new to the model can show a higher error rate. If you have your own Transkribus or eScriptorium export, start from the next step with that export in place of htr/.

Importing PAGE XML into Potato

Install Potato and import the folder:

bash
pip install potato-annotation==2.13.1
potato import -f page_xml -i htr/ -o htr-correction/
text
Imported 88 item(s), 1 question(s), 0 answer(s) from 0 annotator(s), and 0 model suggestion(s) from PAGE XML.
  wrote htr-correction/data/htr.jsonl
  wrote htr-correction/page_xml/page/Seite0405.xml
  wrote htr-correction/media/Seite0405.JPG
  ...
  wrote htr-correction/config.yaml
  wrote htr-correction/README.md

-f transkribus names the same importer. The import searches the folder for PAGE files, looks for each page's image beside the XML and one folder up, and copies the image into media/. For images stored elsewhere, add --image-dir with the folder's path. When a PAGE file's imageFilename is a web address, such as a IIIF image URL, the item keeps the address and the browser loads the image from that server. The import also keeps a copy of each PAGE file under page_xml/, and the export later writes the corrections into those copies.

Each line became one item in data/htr.jsonl, in the page's reading order. The second line of page 405 shows the fields:

json
{"id": "Seite0405-line_b0d3dab2-88ff-4574-9081-768f21788681",
 "htr_text": "schofft. Aūch das Er der",
 "line_image": {"image": "/media/Seite0405.JPG", "width": 2322, "height": 3511,
                "region": [700.0, 500.0, 1055.0, 97.0],
                "polygon": [[1755.0, 507.0], [1405.0, 500.0], [700.0, 508.0], [700.0, 597.0], [1405.0, 589.0], [1755.0, 596.0]],
                "baseline": [[700.0, 568.0], [1405.0, 560.0], [1755.0, 567.0]]},
 "page": "Seite0405.xml", "page_xml": "page_xml/page/Seite0405.xml",
 "region_id": "region_2a649214-6882-4084-978c-59e74c45047f",
 "line_id": "line_b0d3dab2-88ff-4574-9081-768f21788681",
 "line_number": 2, "htr_confidence": "0.9855"}

htr_text is Kraken's reading, line_image locates the line on the page, and htr_confidence is Kraken's confidence for the line. line_id ties the item back to the TextLine element in the PAGE file. For correcting the regions and lines themselves, --page-xml-mode regions imports each page as one item with every region and line as a polygon.

The correction config

The import writes a working config with one text_edit scheme and the image_region display. The config below makes five changes to the import's config:

  • A task name replaces htr (imported).
  • num_annotators_per_item: 2 sends every line to two transcribers.
  • assignment_strategy: fixed_order serves the lines in reading order, so a transcriber reads the lines of a page in the scribe's order.
  • A second display field shows Kraken's confidence for the line.
  • The scheme's description states the transcription convention, because two transcribers who follow different conventions produce different text for the same line.

The config is complete and passes python -m potato.validate_cli config.yaml --strict on 2.12.1 and on 2.13.1:

yaml
port: 8000
annotation_task_name: Ratsprotokolle line correction
task_dir: .
output_annotation_dir: annotation_output/
data_files:
- data/htr.jsonl
item_properties:
  id_key: id
  text_key: htr_text
user_config:
  allow_all_users: true
  users: []
site_dir: default
assignment_strategy: fixed_order
num_annotators_per_item: 2
instance_display:
  fields:
  - key: line_image
    type: image_region
    label: Line
  - key: htr_confidence
    type: text
    label: Kraken confidence
annotation_schemes:
- annotation_type: text_edit
  name: transcription
  description: Correct the transcription of this line. Keep the scribe's spelling and abbreviations.
  source_field: htr_text
  show_diff: true
  show_edit_distance: true
  allow_reset: true

show_diff shows a word diff against Kraken's text, show_edit_distance shows counts of changed words and characters, and allow_reset adds a button that restores Kraken's text. A real project's description would name each special character, such as the ū and ¬ in the next section.

Correcting a line

Start the server from the project folder, and have each transcriber register with a username and password:

bash
cd htr-correction
potato start config.yaml -p 8000

The page shows the line cropped from the page image with the line's outline drawn over the crop, and "Show the whole image" opens the full page with the line marked. Below the crop and Kraken's confidence, "Original" shows Kraken's text, and the box under "Edit below" starts with the same text. While the transcriber types, "Words changed" and "Chars changed" count the edits, and "Changes" strikes out each replaced word and shows the new word beside the old one. The diff works on whole words. Kraken read the first line of page 405 as 263 instead of 203, and the diff strikes out the whole number and adds the new number, while the character count shows one change.

On a line that is already correct, click Next. Potato 2.13.1 saves an unchanged line as accepted, with an edit distance of 0, so the export tells the line apart from a line nobody opened. Potato 2.12.1 saved a text_edit answer only after the text in the box changed or the reset button ran. On 2.12.1 a line left untouched was saved with no answer, and the progress counter did not count the line, so on that version click "Reset to Original" before Next. In the browser check for this page on 2.12.1, a line passed with Next alone left the counter at 1 of 88, and "Reset to Original" on the next line moved the counter to 2.

The READ transcription follows two conventions to put in the scheme's description. The first is ū for the hook a scribe wrote over a u to tell u from n, and the second is ¬ at the end of a line where a word breaks. The ground truth for the four pages has 37 ū and 18 ¬.

The test run behind this page

The numbers on this page come from a run on Potato 2.12.1 with two scripted transcribers who used the ground truth as the answer key. The script saved each line through the same requests the annotation page sends. The first transcriber, alice, typed the ground truth of each line exactly. The second transcriber, bob, typed the ground truth with a plain u for every ū. Bob's rule stands for a transcriber who follows a different convention from the ground truth's. On a line whose answer matched Kraken's text, each transcriber saved the line the way "Reset to Original" saves the line. Alice changed 56 of the 88 lines and Bob changed 69, because Bob also removed the hook from lines that Kraken had read correctly. A real transcriber also misreads lines, and the script makes no misreadings.

Exporting the corrected PAGE XML

The page_xml export writes each transcriber's corrections into the project's copies of the PAGE files:

bash
python -m potato.export -c config.yaml -f page_xml -o export/
text
export/alice/page_xml/page/Seite0405.xml ... Seite0408.xml
export/bob/page_xml/page/Seite0405.xml ... Seite0408.xml

With two transcribers, each transcriber gets a folder, and the export reported 176 lines written and 125 lines changed. --option annotator=alice writes one transcriber's version without the user folder. In each saved line, the export sets the line's TextEquiv text and removes Kraken's confidence for the line. On a line whose text changed, the export also removes Kraken's Word elements and the Glyph elements inside them. The old word and glyph positions no longer match the new text, and --option keep_words=true keeps the elements anyway. The export updates the LastChange date and leaves the rest of each file's content as Kraken wrote it. Alice's four files matched the ground truth with no character differences.

The corrected files are training data for the next model. eScriptorium imports PAGE XML transcriptions and trains models from a document's transcriptions. Kraken's own tools read the files once the page images sit beside them, and ketos test reports the model's error rate against the corrected text:

bash
cp media/*.JPG export/alice/page_xml/page/
ketos test -f page -m german_handwriting.mlmodel export/alice/page_xml/page/*.xml
text
=== report  ===
1732	Characters
151	Errors
91.28%	Character Accuracy

A character accuracy of 91.28% is a character error rate of 8.72%. The same files fine-tune the model:

bash
ketos train -f page -i german_handwriting.mlmodel --resize union -o finetuned export/alice/page_xml/page/*.xml

-i starts from the existing model, --resize union adds any characters the corrections use that the model lacks, and -o names the folder for the new model. A one-epoch run on the four pages split the 88 lines into 79 for training and 9 for validation, and wrote a model file to the folder. Four pages are a test of the route rather than enough data for a better model, so collect corrections from many more pages before retraining.

Character error rate before and after correction

Character error rate (CER) is the number of character insertions, deletions, and substitutions needed to turn the model's text into the corrected text, divided by the length of the corrected text. CER is the character-level form of word error rate, and the count of edits is the Levenshtein distance. Potato 2.12.1 computes no agreement between transcribers on text_edit, so CER between the two transcribers stands in for an agreement figure. The JSONL export has each transcriber's text for each line:

bash
python -m potato.export -c config.yaml -f jsonl -o export/

Each record stores the line's answer as a JSON string with edited_text, original_text, edit_distance_chars, and edit_distance_words. The script below compares Kraken's text with each transcriber's text, compares the two transcribers, and splits Kraken's errors by Kraken's confidence. Run the script in htr-correction/:

python
import collections
import json
 
 
def distance(a, b):
    """Levenshtein distance between two strings, in characters."""
    previous = list(range(len(b) + 1))
    for i, char_a in enumerate(a, 1):
        current = [i]
        for j, char_b in enumerate(b, 1):
            current.append(min(previous[j] + 1, current[j - 1] + 1,
                               previous[j - 1] + (char_a != char_b)))
        previous = current
    return previous[-1]
 
 
def cer(pairs):
    """Character error rate of the first text in each pair against the second."""
    errors = sum(distance(hyp, ref) for hyp, ref in pairs)
    return errors / sum(len(ref) for _, ref in pairs), errors
 
 
# The HTR text and Kraken's confidence for every line
lines = {}
for row in open("data/htr.jsonl"):
    item = json.loads(row)
    lines[item["id"]] = item
 
# Each transcriber's corrected text for each line
corrected = collections.defaultdict(dict)
for row in open("export/annotations.jsonl"):
    record = json.loads(row)
    blob = record["labels"].get("transcription", {}).get("transcription")
    if blob:
        corrected[record["user_id"]][record["instance_id"]] = json.loads(blob)["edited_text"]
 
for user, texts in sorted(corrected.items()):
    rate, errors = cer([(lines[i]["htr_text"], texts[i]) for i in texts])
    print(f"HTR against {user}: CER {rate:.3f} ({errors} edits, {len(texts)} lines)")
 
first, second = sorted(corrected)
shared = sorted(set(corrected[first]) & set(corrected[second]))
rate, errors = cer([(corrected[second][i], corrected[first][i]) for i in shared])
differ = [i for i in shared if corrected[first][i] != corrected[second][i]]
print(f"{second} against {first}: CER {rate:.3f} ({errors} edits, {len(differ)} of {len(shared)} lines differ)")
 
# How wrong the HTR text was at each confidence level, against the first transcriber
bands = collections.defaultdict(list)
for i in shared:
    confidence = float(lines[i].get("htr_confidence", 0))
    band = "0.95 and up" if confidence >= 0.95 else "0.90 to 0.95" if confidence >= 0.9 else "below 0.90"
    bands[band].append((lines[i]["htr_text"], corrected[first][i]))
for band in ("below 0.90", "0.90 to 0.95", "0.95 and up"):
    rate, errors = cer(bands[band])
    print(f"confidence {band:13} {len(bands[band]):3} lines  CER {rate:.3f}")
text
HTR against alice: CER 0.087 (150 edits, 88 lines)
HTR against bob: CER 0.106 (184 edits, 88 lines)
bob against alice: CER 0.021 (37 edits, 29 of 88 lines differ)
confidence below 0.90     15 lines  CER 0.255
confidence 0.90 to 0.95   21 lines  CER 0.140
confidence 0.95 and up    52 lines  CER 0.042

Kraken's text had a CER of 8.7% against Alice's corrections and 10.6% against Bob's. The script's 150 edits and ketos test's 151 differ by one, because one of Kraken's lines ends in a space and the two counts treat the space differently. The 37 edits between the two transcribers are the 37 hooks Bob left out, and the convention alone moved the model's measured CER by 1.9 points. Agree on the convention before the first line is corrected, and report CER against one transcriber's text or against text the two transcribers settled together.

Lines below a confidence of 0.90 had a CER of 25.5%, against 4.2% for lines at 0.95 and above. One line has no confidence, because Kraken read nothing on a line with a single letter, and the script counts the line below 0.90. Even so, 24 of the 52 lines at 0.95 and above had at least one error. A project that skips high-confidence lines leaves those errors in the training data.

TEI from the corrected text

Potato 2.13.1 has no route from a text_edit project to TEI. The tei export writes span annotations as TEI name elements, and on this project the export stops with an error:

text
Export failed!
  ERROR: Cannot export: No span annotations to export

The tei exporter also writes each item's original text rather than a corrected text, so adding a span scheme to the config would still produce Kraken's reading. TEI from the corrected lines therefore needs a conversion outside Potato. The Transkribus help center lists a TEI export for the Scholar, Team, and Organisation plans, with page2tei as one of two stylesheets. The page2tei stylesheet is MIT-licensed and runs outside Transkribus. The stylesheet starts from the METS file of a Transkribus export, and Potato's export writes PAGE files and no METS file. For a plain reading text, a short script can read each corrected TextLine from the PAGE files in reading order and write a TEI <lb/> before each line.

Other tools for HTR correction

Transkribus and eScriptorium each have a correction editor, and the two platforms also recognize text and train models. Transkribus is a hosted service from READ-COOP. The Transkribus plans page lists recognition at 1 credit per page, a free plan with 50 credits a month and basic exports that include PAGE XML, and a Scholar plan at €99 a year with 900 credits and every export format. eScriptorium is MIT-licensed, runs Kraken on a server the project hosts or on an institution's instance, and exports PAGE XML, ALTO, and plain text. For one person correcting a model's output before retraining, the editor in the tool that ran the model is the shorter route.

Potato fits when several transcribers share the correction, when each line goes to more than one transcriber, and when the page images stay on your own server. Digital humanities transcription tools compared sets Transkribus, eScriptorium, FromThePage, Zooniverse, and Potato beside each other.

Limits of Potato for HTR correction

  • No recognition or training. Potato reads PAGE XML from Transkribus, eScriptorium, OCR-D, or Kraken and runs no HTR model.
  • No agreement figure on transcriptions. Potato 2.13.1 computes no agreement on text_edit answers, and the CER script above is the substitute.
  • No ALTO. Potato 2.13.1 reads and writes PAGE XML and no ALTO XML. Export PAGE XML from Transkribus or eScriptorium for the import.
  • No TEI from corrected lines. The tei export writes spans over the original text, as the TEI section above describes.
  • Whole-word diff. The "Changes" view marks whole words, and "Chars changed" gives the character count.

Where to go next

References

  1. Sánchez, J. A., Romero, V., Toselli, A. H., & Vidal, E. (2016). ICFHR2016 Competition on Handwritten Text Recognition on the READ Dataset. 15th International Conference on Frontiers in Handwriting Recognition (ICFHR), 630-635. https://doi.org/10.1109/ICFHR.2016.0120
  2. Toselli, A. H., Romero, V., Villegas, M., Vidal, E., & Sánchez, J. A. (2018). HTR Dataset ICFHR 2016 (Version 1.2.0). Zenodo. https://doi.org/10.5281/zenodo.1297399
  3. Weil, S. (2023). HTR model for German manuscripts trained from several datasets. Zenodo. https://doi.org/10.5281/zenodo.7933463