# Cultural Domain Analysis in Potato: Free Lists, Triads, Pile Sorts and Consensus Source: https://www.potatoannotator.com/docs/guides/cultural-domain-analysis **Cultural domain analysis studies how members of a group think about a set of things, such as kinds of fruit, illnesses or kin terms. Potato 2.12.0 runs free lists, triad tests, pile sorts and true/false questionnaires as questions in a study, and its `cultural_domain` export computes salience indices, similarity matrices and a cultural consensus model from the answers.** The methods are the ones Weller and Romney describe in *Systematic Data Collection* (1988): | Method | Potato scheme | What it gives | |---|---|---| | Free listing | `free_list` | Which items belong to the domain, and how salient each is | | Triads | `triad_test` | Item-by-item similarity | | Pile sorts | [`card_sort`](/docs/annotation-types/card-sort) | Item-by-item similarity, and agreement between informants | | Questionnaires | `radio`, `select`, `likert` | The input to a [cultural consensus](https://en.wikipedia.org/wiki/Cultural_consensus_theory) analysis | ## Free lists ```yaml annotation_schemes: - annotation_type: free_list name: fruit_freelist description: List all the kinds of fruit you can think of. instructions: >- Type one kind of fruit, then press Enter. Keep going until you cannot think of any more. min_items: 3 max_items: 60 normalize: true label_requirement: required: true ``` | Option | Default | Meaning | |---|---|---| | `instructions` | none | Text shown under the prompt | | `min_items` | none | The list is incomplete until it has this many items | | `max_items` | none | The entry box is disabled once the list has this many items | | `time_limit_s` | none | A countdown in seconds. The participant presses Start, and the list locks when the time runs out | | `allow_duplicates` | `false` | When false, an item already on the list (ignoring case and spaces) is refused with a message | | `normalize` | `false` | `true`, or a list of `lowercase` and `trim`. Applied by the export, never to what is stored | | `placeholder` | "Type an item, then press Enter" | Placeholder text in the entry box | Each item is stored with the time since the list started. The clock runs only while the page is showing, and a timed list saves its elapsed time every five seconds, so leaving the page gives back at most that much. A free list usually sits on a survey page; put the same entry in the page's question file. ## Triad tests The domain comes from the config, or from a list in each item: ```yaml annotation_schemes: - annotation_type: triad_test name: triads description: Triads labels: [apple, banana, cherry, grape, lemon, lime, mango] ``` ```yaml annotation_schemes: - annotation_type: triad_test name: triads description: Triads items_field: items ``` A domain needs 3 to 60 items, and a triad test on a survey page needs `labels`, because a survey page has no item. `question` defaults to "Which is most different?", and `max_triads` caps the number asked. | `design` | Triads | When `auto` uses it | |---|---|---| | `complete` | Every one of the C(n, 3) triads | While there are at most 20 (n ≤ 6) and `max_triads` allows it | | `steiner` | n(n - 1)/6 triads, with every pair of items together in exactly one | When n is 1 or 3 mod 6 (7, 9, 13, 15, 19, 21, ...) and `max_triads` allows it | | `sample` | `max_triads` triads, or without it, enough for every pair to appear at least once | Otherwise | `steiner` is a balanced incomplete block design with λ = 1 (a Steiner triple system). It exists only for those n, so asking for it elsewhere is a validation error when the domain is in `labels`, and falls back to `sample` for an `items_field` domain. `sample` picks triads one at a time, each step taking the unused triad whose pairs have appeared least so far. With a fixed `max_triads`, pair counts stay close to even but can differ by two. Every participant answers the same triads, because the design depends only on the domain. The order of the triads and the position of the items within each triad are shuffled per participant with a seed, so a reload shows the same sequence. Participants click an item or press 1, 2 or 3. ## A study with all three methods `examples/advanced/cultural-domain/` runs the steps in one study: 1. a prestudy page with a free list ("List all the kinds of fruit you can think of"); 2. two items, nine fruits and seven vegetables, each with a triad test and an open pile sort over the item's list; 3. a poststudy page of eight true/false statements for a consensus analysis. ```bash python potato/flask_server.py start examples/advanced/cultural-domain/config.yaml -p 8000 ``` In a real study the free lists usually come first, and the items for the triads and pile sorts are chosen from them afterwards, which means two deployments. The example puts all three in one so it can be tried end to end. ## Exporting the analysis ```bash python -m potato.export --config config.yaml --format cultural_domain --output out/ \ --option consensus_schemes=cc_tomato,cc_lemon_sweet,cc_banana_tree \ --option synonyms=data/synonyms.csv ``` | Option | Default | Meaning | |---|---|---| | `normalize` | each scheme's `normalize` | `true`, `false`, or `lowercase` and `trim` (comma-separated), applied to free-list items before counting | | `synonyms` | none | A CSV of `variant,canonical` rows, applied after normalizing | | `consensus_schemes` | none | Comma-separated scheme names for the consensus analysis | | `consensus_choices` | the number of labels | The number of options L for questions whose scheme has fewer labels than that | | `consensus_min_answers` | `2` | Informants with fewer answers are left out of the consensus analysis | ### Free-list salience `freelist_items.csv` has one row per listed item, with the user, the normalized item, what was typed, its rank, the list length and the time it was added. `salience.csv` has one row per item and domain, most salient first, with N (the non-empty lists), F (the lists that mention the item), the percentage of informants, the mean rank, Smith's S and Sutrop's (2001) cognitive salience index. For item j, where list i has length L_i and mentions j at rank r_ij: ```text S_j = (1/N) × Σ_i (L_i − r_ij + 1) / L_i (sum over lists that mention j) CSI_j = F_j / (N × mean rank of j) ``` Before counting, each item is normalized if asked and mapped through the synonym table. An item that then appears twice in one list counts once, at its first rank. Add plural forms to the synonyms file, such as `apples,apple`, so they are not counted separately. ### Pile-sort and triad similarity From each `card_sort` answer the export builds an item-by-item matrix with a 1 where two items share a pile. `pilesort_cooccurrence__.csv` holds the share of informants who put each pair in one pile, out of the informants who sorted both items, and `pilesort_informants__.csv` holds the proportion of item pairs each two informants treated alike. In a triad (a, b, c) where c was chosen as most different, the pair (a, b) is judged similar. `triads_similarity_.csv` gives, for each pair, the number of times it was judged similar divided by the number of times it was shown together. With a Steiner design every pair appears in one triad per informant, so a pair's similarity is the proportion of informants who left it together. ### Cultural consensus The consensus model (Romney, Weller & Batchelder, 1986) asks whether a group of informants shares one body of knowledge, how much of it each informant has (their competence), and what the culturally correct answers are. A `radio`, `select` or `likert` scheme named in `consensus_schemes` gives one question per item, or one question when it sits on a survey page. 1. **Agreement.** For informants i and j, the proportion of matching answers is corrected for guessing among L options. 2. **Competence.** Competences are the loadings on the first factor of the corrected agreement matrix, estimated by iterated principal axis factoring. 3. **One culture.** The eigenvalues are reported. A first-to-second ratio above 3 is the usual rule of thumb for one shared culture, and it is a rule of thumb, not a test. 4. **Answer key.** Following Batchelder and Romney (1988), the export gives each question's most probable culturally correct answer with its posterior probability. `consensus_informants.csv`, `consensus_answers.csv` and `consensus_summary.json` hold the results. At least three informants are needed, and with few questions the competence estimates are noisy, because the model is meant for questionnaires of tens of questions. `multiselect` schemes are refused for consensus, since an informant can give several answers to one question, so use one `radio` per statement. ## References - Weller, S. C., & Romney, A. K. (1988). *Systematic Data Collection*. Sage. - Romney, A. K., Weller, S. C., & Batchelder, W. H. (1986). Culture as consensus: A theory of culture and informant accuracy. *American Anthropologist*, 88(2), 313-338. - Batchelder, W. H., & Romney, A. K. (1988). Test theory without an answer key. *Psychometrika*, 53(1), 71-92. - Sutrop, U. (2001). List task and a cognitive salience index. *Field Methods*, 13(3), 263-276. ## Further Reading - [Card Sorting](/docs/annotation-types/card-sort) - Open and closed pile sorts - [Ego Networks](/docs/annotation-types/ego-network) - Personal networks from the same informants - [Survey Import](/docs/tools/survey-import) - Bringing questionnaires in from other tools For implementation details, see the [source documentation](https://github.com/davidjurgens/potato/blob/master/docs/guides/cultural-domain-analysis.md).