Cultural Domain Analysis in Potato: Free Lists, Triads, Pile Sorts and Consensus

Cultural domain analysis studies how members of a group think about a set of things, such as kinds of fruit, illnesses or kin terms. Potato 2.12.0 runs free lists, triad tests, pile sorts and true/false questionnaires as questions in a study, and its cultural_domain export computes salience indices, similarity matrices and a cultural consensus model from the answers.

The methods are the ones Weller and Romney describe in Systematic Data Collection (1988):

Free lists

annotation_schemes:
  - annotation_type: free_list
    name: fruit_freelist
    description: List all the kinds of fruit you can think of.
    instructions: >-
      Type one kind of fruit, then press Enter. Keep going until you cannot
      think of any more.
    min_items: 3
    max_items: 60
    normalize: true
    label_requirement:
      required: true

Each item is stored with the time since the list started. The clock runs only while the page is showing, and a timed list saves its elapsed time every five seconds, so leaving the page gives back at most that much. A free list usually sits on a survey page; put the same entry in the page's question file.

Triad tests

The domain comes from the config, or from a list in each item:

annotation_schemes:
  - annotation_type: triad_test
    name: triads
    description: Triads
    labels: [apple, banana, cherry, grape, lemon, lime, mango]

annotation_schemes:
  - annotation_type: triad_test
    name: triads
    description: Triads
    items_field: items

A domain needs 3 to 60 items, and a triad test on a survey page needs labels, because a survey page has no item. question defaults to "Which is most different?", and max_triads caps the number asked.

steiner is a balanced incomplete block design with λ = 1 (a Steiner triple system). It exists only for those n, so asking for it elsewhere is a validation error when the domain is in labels, and falls back to sample for an items_field domain. sample picks triads one at a time, each step taking the unused triad whose pairs have appeared least so far. With a fixed max_triads, pair counts stay close to even but can differ by two.

Every participant answers the same triads, because the design depends only on the domain. The order of the triads and the position of the items within each triad are shuffled per participant with a seed, so a reload shows the same sequence. Participants click an item or press 1, 2 or 3.

A study with all three methods

examples/advanced/cultural-domain/ runs the steps in one study:

1. a prestudy page with a free list ("List all the kinds of fruit you can think of");
2. two items, nine fruits and seven vegetables, each with a triad test and an open pile sort over the item's list;
3. a poststudy page of eight true/false statements for a consensus analysis.

python potato/flask_server.py start examples/advanced/cultural-domain/config.yaml -p 8000

In a real study the free lists usually come first, and the items for the triads and pile sorts are chosen from them afterwards, which means two deployments. The example puts all three in one so it can be tried end to end.

Exporting the analysis

python -m potato.export --config config.yaml --format cultural_domain --output out/ \
    --option consensus_schemes=cc_tomato,cc_lemon_sweet,cc_banana_tree \
    --option synonyms=data/synonyms.csv

Free-list salience

freelist_items.csv has one row per listed item, with the user, the normalized item, what was typed, its rank, the list length and the time it was added. salience.csv has one row per item and domain, most salient first, with N (the non-empty lists), F (the lists that mention the item), the percentage of informants, the mean rank, Smith's S and Sutrop's (2001) cognitive salience index. For item j, where list i has length L_i and mentions j at rank r_ij:

S_j   = (1/N) × Σ_i (L_i − r_ij + 1) / L_i     (sum over lists that mention j)
CSI_j = F_j / (N × mean rank of j)

Before counting, each item is normalized if asked and mapped through the synonym table. An item that then appears twice in one list counts once, at its first rank. Add plural forms to the synonyms file, such as apples,apple, so they are not counted separately.

Pile-sort and triad similarity

From each card_sort answer the export builds an item-by-item matrix with a 1 where two items share a pile. pilesort_cooccurrence_<scheme>_<item>.csv holds the share of informants who put each pair in one pile, out of the informants who sorted both items, and pilesort_informants_<scheme>_<item>.csv holds the proportion of item pairs each two informants treated alike.

In a triad (a, b, c) where c was chosen as most different, the pair (a, b) is judged similar. triads_similarity_<scheme>.csv gives, for each pair, the number of times it was judged similar divided by the number of times it was shown together. With a Steiner design every pair appears in one triad per informant, so a pair's similarity is the proportion of informants who left it together.

Cultural consensus

The consensus model (Romney, Weller & Batchelder, 1986) asks whether a group of informants shares one body of knowledge, how much of it each informant has (their competence), and what the culturally correct answers are. A radio, select or likert scheme named in consensus_schemes gives one question per item, or one question when it sits on a survey page.

1. Agreement. For informants i and j, the proportion of matching answers is corrected for guessing among L options.
2. Competence. Competences are the loadings on the first factor of the corrected agreement matrix, estimated by iterated principal axis factoring.
3. One culture. The eigenvalues are reported. A first-to-second ratio above 3 is the usual rule of thumb for one shared culture, and it is a rule of thumb, not a test.
4. Answer key. Following Batchelder and Romney (1988), the export gives each question's most probable culturally correct answer with its posterior probability.

consensus_informants.csv, consensus_answers.csv and consensus_summary.json hold the results. At least three informants are needed, and with few questions the competence estimates are noisy, because the model is meant for questionnaires of tens of questions. multiselect schemes are refused for consensus, since an informant can give several answers to one question, so use one radio per statement.

References

• Weller, S. C., & Romney, A. K. (1988). Systematic Data Collection. Sage.
• Romney, A. K., Weller, S. C., & Batchelder, W. H. (1986). Culture as consensus: A theory of culture and informant accuracy. American Anthropologist, 88(2), 313-338.
• Batchelder, W. H., & Romney, A. K. (1988). Test theory without an answer key. Psychometrika, 53(1), 71-92.
• Sutrop, U. (2001). List task and a cognitive salience index. Field Methods, 13(3), 263-276.

Further Reading

• Card Sorting - Open and closed pile sorts
• Ego Networks - Personal networks from the same informants
• Survey Import - Bringing questionnaires in from other tools

For implementation details, see the source documentation.