Skip to content
यह पृष्ठ अभी आपकी भाषा में उपलब्ध नहीं है। अंग्रेज़ी संस्करण दिखाया जा रहा है।

How to Annotate Threaded Conversations

Render reply structure from reply_to, annotate whole threads and individual comments at once, link spans across comments, and import a ConvoKit corpus both ways.

A forum thread flattened into a transcript loses the thing most conversational research is about: who was replying to whom. Rendering reply structure is what makes a thread annotatable as a thread rather than as a list.

Render the structure

yaml
instance_display:
  fields:
    - key: conversation
      type: dialogue
      label: "Thread"
      display_options:
        indent_replies: true

Nesting derives from each turn's reply_to. Turn identity is read from turn_id, step_id or plain id, so forum exports, chat logs and mailing lists work without reshaping the data.

Use stable ids. Falling back to position means a re-fetched or re-ordered thread re-points every existing annotation, which is the kind of corruption that is invisible until someone checks.

One field, several questions at once

The useful property is that a single conversation field can carry, simultaneously:

  • Whole-thread schemes — did this thread go off the rails?
  • Per-comment radio, likert, select or text — rate each comment
  • Spans inside comments — mark the sentence that did it
  • span_link between spans in different comments — link a claim to the reply that rebuts it

That last one is what most conversational annotation actually needs and what flat renderings cannot express. A rebuttal is a relation between two positions in two different turns.

For branching conversations, conversation_tree gains per-node widgets keyed by node id, so a branching tree and a flat thread can refer to the same messages.

ConvoKit, both directions

ConvoKit (Cornell) is the standard corpus format for conversational research in CS and computational linguistics.

bash
potato convokit conversations-gone-awry-corpus     # by name
potato convokit ./my-corpus/                       # by directory
potato convokit corpus.zip                         # by zip

Import at conversation or utterance granularity. Export back with --format convokit, either as info.<field>.jsonl overlays that drop into an existing corpus, or as a full dump.

Two properties worth knowing:

  • Each turn carries the real utterance id, so per-comment annotations round-trip by direct lookup rather than by position.
  • No convokit dependency. The format is read and written with the standard library, so importing a corpus does not pull in a research stack.

Every format variant in the wild is read, down to the pre-rename user/root/users.json layout.

Measuring agreement

Thread-level labels use ordinary categorical agreement. Per-comment labels are the interesting case: agreement should be computed per comment, not pooled across the thread, or a long thread of easy comments will swamp the disagreement on the one comment that mattered.

Spans inside comments use span agreement, with the caveat that offsets must be computed against the same normalisation the server applies. Potato had a bug here where the server collapsed whitespace and the client did not, drifting every offset in a dialogue.

Further reading