Linguistic Fieldwork in Potato: Glossing with FLEx, Toolbox, Praat and ELAN

Potato 2.12.0 imports interlinear texts from FieldWorks Language Explorer (FLEx) and Toolbox, lets annotators review every sentence in an interlinear_gloss scheme that checks the Leipzig alignment rules, and exports the corrected glosses back to the same format. Praat TextGrid and ELAN files import the same way, as time-aligned tiers.

The analysis usually already lives in FLEx or Toolbox, and the work is checking and correcting it, often with several people reviewing different texts. Potato adds the review step: assignment of texts to reviewers, an alignment check on every word, and a record of who confirmed each sentence.

Import FLEx or Toolbox texts

# A FLEx interlinear export (FLExText)
potato import -i stories.flextext -o glossing-project/
 
# A Toolbox or Shoebox text database
potato import -i texts.txt -f toolbox -o glossing-project/

-i can also be a directory of .flextext files, or of Toolbox files (.txt, .sfm, .db, .tbx, .toolbox). The format is detected from the content: a FLExText file has <interlinear-text> elements, and a Toolbox file has \ref records with a text line and a morpheme or gloss line. -f flextext or -f toolbox chooses explicitly.

A FLExText file that cannot be read is left out with the reason: it is binary or not UTF-8 text, it is not well-formed XML, or it defines XML entities, which uploaded files may not.

Each FLEx phrase or Toolbox record becomes one item. The existing analysis is written into the item's own fields (words, morphemes, glosses, translation, and pos when the source has parts of speech), and the generated scheme reads them as its starting point. Nothing counts as a person's answer until an annotator presses Next or edits the item, so an export before review contains only the sentences someone has checked.

Toolbox records start at \ref, and the interlinear lines are read by column, as Toolbox writes them. Hand-edited files are often not aligned, so two readings by order are tried first: morphemes grouped into words by their own marks, and glosses paired with morphemes in order. The importer reads \tx or \t for the text, \mb or \m for morphemes, \ge or \g for the gloss, \ps or \p for part of speech, and \ft or \f for the free translation. Other markers in a record, such as \nt, are kept in the item's sfm_extra field and written back on export.

Review the glosses

python potato/flask_server.py start glossing-project/config.yaml -p 8000

Annotators see each sentence with the imported analysis in place. Any word whose segmentation and gloss do not line up is marked when the page opens, so errors already in the source show on the first review. Interlinear Glossed Text lists the checks and the keyboard controls.

The checks follow the Leipzig Glossing Rules by Bernard Comrie, Martin Haspelmath and Balthasar Bickel:

• The example line and the gloss line are aligned word by word.
• Within a word, - separates affixes and = separates clitics, and the gloss has the same separators in the same places, so morpheme and gloss can be paired by counting.
• When one morpheme stands for several meanings, they are joined with a period: NOM.SG, put.in.bag. A period is not a boundary and is not counted.
• Person and number may be written together without a period, as in 3SG.
• Grammatical categories are abbreviations in capitals, printed in small caps.
• An infix is set off with angle brackets in both lines, and its gloss goes next to the stem's gloss.

Export back to FLEx, Toolbox or LaTeX

python -m potato.export --config glossing-project/config.yaml --format flextext --output out/
python -m potato.export --config glossing-project/config.yaml --format toolbox --output out/
python -m potato.export --config glossing-project/config.yaml --format igt --output out/

• flextext writes <annotator>.flextext, one per annotator, grouped into the texts and paragraphs the import recorded. Morph types are worked out from the boundaries: the first morpheme with a lowercase (lexical) gloss is the stem, morphemes before it are prefixes or proclitics, and those after it are suffixes or enclitics.
• toolbox writes <annotator>.txt with column-aligned lines and the original \ref values.
• igt writes igt.txt, with three aligned lines and the translation per example, and igt.tex for the gb4e LaTeX package.

Re-importing an exported file gives the same words, segmentations, glosses, translations and parts of speech. What does not survive the trip:

• FLEx GUIDs and lexicon links, because the export has no lexicon.
• FLEx punctuation words, which stay in the item's sentence text but are not written back as punct words.
• The morph type of a morpheme that is neither the stem nor next to a marked boundary, such as a second root in a compound.
• Tiers other than pos in FLExText. In Toolbox they are written as extra lines named after the tier.

Praat TextGrids and ELAN files

potato import -i recordings/ -o my-project

recordings/ holds .TextGrid or .eaf files, each with an audio or video file of the same name beside it. Each file becomes an item with its media copied into media/, each tier becomes a tier of one tiered_annotation scheme, and every non-empty interval becomes a segment that annotators can correct. A tier with at most 30 distinct values, such as phones or speaker codes, gets them as its labels. The textgrid and eaf exporters write the corrected tiers back out. Importing Projects has the details.

Further Reading

• Interlinear Glossed Text - Scheme options, checks and the stored answer
• Transcript Formats - Time-aligned transcripts and tiers
• Perceptual Dialectology - Draw-a-map tasks for dialect perception
• Multilingual and Low-Resource Annotation - Annotating in many languages

For implementation details, see the source documentation.