# Survey Research Tools Compared

Source: https://www.potatoannotator.com/docs/guides/survey-research-tools-compared

**LimeSurvey is a free, open-source replacement for a Qualtrics questionnaire, with quotas and exports to SPSS, R, and Stata, while REDCap suits clinical research databases, KoboToolbox suits offline field interviews, SurveyJS suits forms inside your own application, and oTree suits interactive economic experiments. Potato is a free, self-hosted alternative to Qualtrics for survey experiments. Potato imports Qualtrics, LimeSurvey, and REDCap questionnaires and adds balanced random assignment, factorial vignettes, list experiments, and Prolific or SONA recruitment.**

A survey experiment gives each respondent one version of a questionnaire by [random assignment](https://en.wikipedia.org/wiki/Random_assignment) and compares answers across the versions. A [vignette](https://en.wikipedia.org/wiki/Vignette_(psychology)) is a short scenario whose details change from one condition to another, and a factorial vignette design crosses several such details. [Conjoint analysis](https://en.wikipedia.org/wiki/Conjoint_analysis) asks respondents to choose between profiles built from attribute levels and estimates how much each attribute changes the choice. A [list experiment](https://en.wikipedia.org/wiki/List_experiment) and [randomized response](https://en.wikipedia.org/wiki/Randomized_response) let respondents answer a sensitive question without revealing the respondent's own answer. [Data residency](https://en.wikipedia.org/wiki/Data_sovereignty) is the country and the server where the responses are stored.

Many researchers start this comparison when a university limits Qualtrics. [Radboud University's Faculty of Social Sciences](https://www.ru.nl/en/staff/news/phasing-out-qualtrics) chose not to renew the faculty's Qualtrics license, first for cost and then on the advice of the university's Data Protection Officer. The faculty asks researchers to move to LimeSurvey by December 1, 2026, and to delete Qualtrics data and materials by the same date. [Survey research in Potato](/use-cases/survey-research) covers Potato's survey features in one place, and [Behavioral Experiment Tools Compared](/docs/guides/behavioral-experiment-tools-compared) covers software for reaction-time tasks.

## Which survey tools are worth comparing?

Costs come from each vendor's own pricing page in October 2026, and papers describing [REDCap](https://doi.org/10.1016/j.jbi.2008.08.010) (2009), [oTree](https://doi.org/10.1016/j.jbef.2015.12.001) (2016), and [Potato 2.0](https://aclanthology.org/2026.acl-demo.37/) (2026) are listed under References.

| Tool | Runs as | License | Cost | Suits |
|---|---|---|---|---|
| **[Qualtrics](https://www.qualtrics.com/)** | Hosted survey platform | Proprietary | Free account with 3 active surveys, 30 questions per survey, and 500 responses; Strategic Research from $420 a month; larger plans by quote | Questionnaires with many question types, plus conjoint and MaxDiff projects |
| **[LimeSurvey](https://www.limesurvey.org/)** | PHP web app you host (Community Edition), or LimeSurvey Cloud | GPL-2.0-or-later | Community Edition free; Cloud has a free package and paid plans from $39 a month | Questionnaires in several languages, with quotas and exports to SPSS, R, and Stata |
| **[REDCap](https://projectredcap.org/)** | Web app your institution installs | End-user license from Vanderbilt; not open source | Free to non-profit members of the REDCap consortium | Clinical and longitudinal research data, with audit trails |
| **[SurveyJS](https://surveyjs.io/)** | JavaScript library inside your own web application | MIT (Form Library); Survey Creator under a commercial license | Form Library free; Survey Creator from $589 per developer, one-time | Survey forms embedded in your own application |
| **[KoboToolbox](https://www.kobotoolbox.org/)** | Hosted web app on a Global or an EU server, plus an Android app | AGPL-3.0 | Free Community plan with usage limits, and paid plans | Interviews in the field on phones and tablets, online or offline |
| **[oTree](https://www.otree.org/)** | Python framework, deployed through oTree Hub to Heroku or to your own server | Open source; the license requires citing the oTree paper | Free | Interactive experiments with groups, markets, and payoffs |
| **[Potato](/)** | Python web server configured in YAML, on your own server | GPL-3.0-or-later | Free | Vignette and list experiments, item ratings, and coding open-ended answers with several coders |

## Which tool for which survey

The table pairs each kind of study with the tools built for that kind of study.

| Study | Good fits |
|---|---|
| A standard questionnaire with skip logic | Qualtrics, LimeSurvey, REDCap, SurveyJS, and Potato |
| A between-subjects survey experiment with balanced cells | Qualtrics, oTree, and Potato |
| A factorial vignette whose wording changes by condition | Qualtrics and Potato |
| A choice-based conjoint | Qualtrics (an additional purchase) and Potato |
| List experiments and randomized response with prevalence estimates | Potato |
| Quotas on respondent characteristics | Qualtrics and LimeSurvey |
| One questionnaire offered in several languages | LimeSurvey |
| Interviews on phones or tablets without a connection | KoboToolbox and REDCap (mobile app) |
| Longitudinal panels with repeated instruments | REDCap |
| Group games, markets, and bargaining | oTree |
| Survey forms inside your own web application | SurveyJS |
| Ratings of many items split across respondents, with agreement | Potato |
| Coding open-ended answers | ATLAS.ti, NVivo, and Potato (see [coding open-ended answers](#coding-open-ended-answers)) |

Qualtrics appears in many rows because one hosted product covers randomization and quotas as well as questionnaires. Potato appears where a study varies what respondents see, asks sensitive questions, or puts items in front of respondents to rate or label.

## Hosting, data residency, and cost

Qualtrics hosts every survey on its own servers. LimeSurvey's Community Edition stores responses on your own server or your provider's, and LimeSurvey Cloud hosts accounts in Germany, Finland, the United Kingdom, the United States, Canada, and Australia. LimeSurvey recommends Germany or Finland for EU customers. REDCap runs on your institution's servers. A member institution signs an end-user license agreement with Vanderbilt University and supports REDCap through its own IT department. KoboToolbox hosts accounts on a Global server or on a European Union server in Ireland.

SurveyJS sends responses to a backend you build, and its pricing page states that all data stays on your own servers. oTree's documentation describes deployment to Heroku through oTree Hub and to Ubuntu or Windows servers you manage. Potato runs on your own machine or your institution's server and writes each participant's answers to `user_state.json` in the study's output directory. Potato contacts an outside service only when the config names the service, as with Prolific's API or a hosted AI endpoint.

## Free Qualtrics alternatives

LimeSurvey's Community Edition, REDCap for consortium members, the SurveyJS Form Library, oTree, and Potato cost nothing to license. Running any of the five means paying for a server and for the staff time to maintain the server. Qualtrics, LimeSurvey Cloud, and KoboToolbox's paid plans charge for the hosting instead. Each free tool covers part of what a Qualtrics account does:

- **LimeSurvey Community Edition** comes closest for an ordinary questionnaire, with quotas, several languages in one survey, and exports to SPSS, R, and Stata. Random assignment takes `rand()` in an equation question, and we found no conjoint module in LimeSurvey's documentation.
- **REDCap** is free only to non-profit members of the REDCap consortium. REDCap suits clinical and longitudinal research and randomizes from an allocation table you upload.
- **The SurveyJS Form Library** is free under the MIT license, and the Survey Creator costs from $589 per developer. Responses go to a backend you build.
- **KoboToolbox** has a free Community plan with usage limits and collects interviews offline. We found no random assignment to conditions in KoboToolbox's documentation.
- **oTree** is free and runs interactive experiments with groups and payoffs, and every assignment rule is Python code you write.
- **Potato** imports a Qualtrics `.qsf` and runs balanced assignment, vignettes, conjoint, and list experiments. Potato has no quotas, no multi-language surveys, and no SPSS files.

## Survey experiments, recruitment, and exports by tool

The second table compares the parts of a study around the questions. SurveyJS is left out because every row depends on the application you build around the library.

| | Qualtrics | LimeSurvey | REDCap | KoboToolbox | oTree | Potato |
|---|---|---|---|---|---|---|
| Random assignment to conditions | Randomizer, with Evenly Present Elements for balance | `rand()` in an equation question, with relevance on question groups | Randomization module, from an allocation table you upload | — | Python code in `creating_session`, with `itertools.cycle` for balance | `conditions`, balanced on assigned or on completed participants, with optional cell caps and weights |
| Question and choice order | Randomizer | Randomization groups | — | `randomize=true` on choices | Python code | `randomize_questions` per page, `randomize_order` per question |
| List experiments and randomized response | — | — | — | — | — | `list_experiment` and `randomized_response`, with estimates in the export |
| Conjoint | Conjoint project, an additional purchase | — | — | — | — | `conjoint`, with the profiles written into each item |
| Published instruments | — | — | Shared Library of curated instruments for partners | — | — | 55 [instruments by name](/docs/features/survey-instruments), from the PHQ-9 to the ANES demographics battery |
| Prolific | Prolific's own integration guide | Panel integration with URL parameters and an end URL | — | — | oTree HR handles start links, completion URLs, and payments | `crowdsourcing.provider: prolific`, with Potato in Prolific's integration guides |
| SONA | Sona's credit-granting guide | Sona's credit-granting guide | Sona's credit-granting guide | — | Sona's credit-granting guide | `crowdsourcing.provider: sona`, with credit granted from the server |
| Response exports | CSV, TSV, Excel, XML, and SPSS | CSV, Excel, SPSS, R, Stata, and PDF | Excel, PDF, SPSS, SAS, Stata, and R | Excel, CSV, GeoJSON, and SPSS labels | CSV and Excel | CSV or TSV with `participants.csv`, JSONL, Parquet, Frictionless Data Package, and Psych-DS |

A dash means we found nothing for the row in the tool's own documentation, checked on 2026-10-28. Prolific states that most survey software works with Prolific when the study link records the Prolific ID and sends the participant back at the end. A dash in the Prolific row therefore means no published guide. A list experiment can be built in any tool with random assignment and display logic, with the estimate computed afterward. If a cell is wrong, [tell us](https://github.com/davidjurgens/potato/issues/new).

Potato's sensitive-question designs and assignment are covered in [Sensitive Questions](/docs/annotation-types/sensitive-questions) and [Experiment Flow](/docs/experiments/experiment-flow). The `survey_experiments` export writes a difference-in-means estimate for each list experiment and a prevalence estimate for each randomized-response question, with a standard error and a 95% interval for every estimate. `participants.csv` writes each participant's condition, page order, question order, and any screen-out. The analysis file therefore records what each respondent was assigned and shown.

## Moving a questionnaire into Potato

`potato import` turns the questions, pages, and show/hide logic of a questionnaire into Potato survey pages and rules. The command reads seven formats:

- a Qualtrics `.qsf` file;
- a LimeSurvey `.lss` file;
- a REDCap data dictionary;
- SurveyJS JSON;
- an XLSForm, as used by ODK, KoboToolbox, SurveyCTO, and Ona;
- a Beiwe survey; and
- a Network Canvas protocol.

```bash
potato import -i my-survey.qsf -o survey-project/
```

A `.qsf` file contains a survey's questions, blocks, flow, and logic, and Qualtrics exports the file from **Survey → Tools → Import/Export → Export Survey**. None of the seven formats contains responses. Export the response data separately before an account closes. Everything the importer could not translate, such as Qualtrics quotas, piped text, or REDCap calculated fields, is listed in `IMPORT_REPORT.md`. [Survey Import](/docs/tools/survey-import) lists what imports from each format, and [Coming from Qualtrics](/docs/guides/coming-from-qualtrics) maps Qualtrics blocks, randomizers, and End of Survey elements to Potato keys.

## A factorial vignette experiment in Potato

The config below runs a 2×2 vignette experiment recruited on Prolific. Each respondent is assigned one combination of sponsor and funding source on arrival, and `{condition.sponsor}` and `{condition.funding}` in the item text are filled with the respondent's levels. `balance_on: completed` counts respondents who finish and respondents still working within `in_progress_minutes` of their last activity, so dropouts and screen-outs do not unbalance the four cells. The screener ends the study for anyone outside the United States and returns the respondent to Prolific with the screen-out code. Two published instruments follow the vignettes, with the efficacy items in a different order for each respondent.

```yaml
annotation_task_name: Local policy vignette experiment
task_dir: .
output_annotation_dir: annotation_output/
data_files:
  - data/vignettes.jsonl
item_properties:
  id_key: id
  text_key: text

login:
  type: url_direct
  url_argument: PROLIFIC_PID

conditions:
  factors:
    sponsor: ["The city council", "A neighborhood association"]
    funding: ["existing city funds", "a new property tax"]
  assignment: balanced
  balance_on: completed

phases:
  order: [consent, screener, annotation, efficacy, demographics]
  consent:
    type: consent
    file: surveys/consent.json
  screener:
    type: prestudy
    file: surveys/screener.json
    exit_when:
      show_when:
        - {schema: us_resident, operator: equals, value: "No"}
      message: This study is for people who live in the United States. Thank you for your time.
  efficacy:
    type: poststudy
    instrument: political-efficacy
    randomize_questions: true
  demographics:
    type: poststudy
    instrument: anes-demographics

crowdsourcing:
  provider: prolific
  prolific:
    completion:
      code: C1VIGNETTE
      screened_out_code: C1SCREENED

annotation_schemes:
  - annotation_type: likert
    name: support
    description: How strongly do you support or oppose this proposal?
    size: 7
    min_label: Strongly oppose
    max_label: Strongly support
    label_requirement:
      required: true
  - annotation_type: radio
    name: sponsor_check
    description: Who made this proposal?
    labels: ["The city council", "A neighborhood association", "I don't remember"]
    label_requirement:
      required: true

# Exports leave out survey answers unless export_include_phase_data is true
export_include_phase_data: true
```

Each line of `data/vignettes.jsonl` is one vignette, with the placeholders in its text:

```json
{"id": "v1", "text": "{condition.sponsor} proposes turning the parking lot on Mill Street into a public park, paid for by {condition.funding}."}
```

The screener page in `surveys/screener.json` asks one question. With `required_label`, the server keeps the respondent on the page until the respondent picks an answer, and the screen-out rule always has an answer to test:

```json
[
  {"name": "us_resident", "annotation_type": "radio",
   "description": "Do you currently live in the United States?",
   "labels": ["Yes", "No"],
   "label_requirement": {"required_label": ["Yes", "No"]}}
]
```

`consent.json` follows the same format, with `required_label: ["I agree"]` on the consent question. The sponsor question after each vignette is a manipulation check. Run `python -m potato.export --config config.yaml --format csv --output results/` for `annotations.csv` with the ratings, `participants.csv` with each respondent's cell and screen-out, and `phase_responses.csv` with the consent, screener, and instrument answers.

## Coding open-ended answers

Open-ended answers are where many survey projects bring in qualitative data analysis software. NVivo and ATLAS.ti are sold under paid licenses with student pricing. The tools below differ in how each reads a survey export. [Qualitative Data Analysis Software Compared](/docs/guides/qualitative-coding-software-compared) compares the same tools as coding software.

- **NVivo** imports a Qualtrics survey directly. Each respondent becomes a case, closed questions become case attributes, and the answers to each open-ended question are grouped into one code. A survey exported as CSV can be imported as a spreadsheet instead, with fewer features.
- **ATLAS.ti** imports a survey from an Excel, CSV, or ODS file. Each open-ended question becomes a code, and each answer becomes a quotation coded with the question, the respondent, and the respondent's closed answers.
- **Usercall** runs AI-moderated interviews and turns transcripts and imported open-ended answers into themes with supporting excerpts. Plans start at $89 a month.
- **Potato** loads exported answers as items in a data file, splits the items among several coders working from a codebook, and reports Krippendorff's alpha per question with a bootstrap interval. [Content Analysis](/docs/guides/content-analysis) walks through the setup.

## What Potato does not do for survey research

- **No panel of its own.** Potato recruits through Prolific, CloudResearch Connect, SONA, and other panels that pass an ID in the URL. A study that needs a probability sample or a national panel buys access from a panel provider.
- **No quotas on respondent answers.** `max_per_cell` caps how many respondents each condition receives, and Potato has no cap on respondents by age, gender, or region. Qualtrics and LimeSurvey have quotas.
- **No offline survey pages.** Consent, screener, and survey pages need a connection to the server. [Pocket Mode](/docs/features/pocket-mode) lets annotators label prefetched items offline on a phone, but the mode applies to the annotation phase only. KoboToolbox and the REDCap mobile app collect whole interviews offline.
- **No field data types.** GPS points, photos, audio recordings, and barcodes from an XLSForm are reported by the import and not brought over.
- **No payments of its own.** Potato computes bonuses from price lists, BDM bids, and belief elicitation. Prolific pays the bonuses, from an uploaded bulk-bonus file or a request sent from Potato's admin route. [Incentive Payments](/docs/experiments/incentive-payments) covers the rules.
- **One language per study.** `ui_language` translates Potato's interface into one of ten bundled languages besides English, and the questions are written in that language. A survey offered in three languages is three configs.
- **No questionnaire without items.** Every Potato study has an annotation phase over items from a data file, and from version 2.13.1 `potato validate` reports an error for a config without one. A plain questionnaire uses one item with a single-label `radio` scheme such as Finish, with the annotation phase placed last. A `pure_display` item does not work for the purpose, because it never completes.
- **No SPSS, SAS, or Stata files.** Exports are CSV, TSV, JSONL, Parquet, a Frictionless Data Package, or Psych-DS. SPSS, SAS, Stata, and R all read the CSV files.
- **Scheduling fixed at the first visit.** From version 2.13, a `schedule:` block runs follow-up waves, experience-sampling prompts, and Delphi rounds, and a `notifications:` block sends reminders by email, SMS, or Prolific message. Each participant's schedule is computed when the first visit ends and does not change afterwards. REDCap schedules repeated instruments and events.

Checked against each project's own documentation in October 2026.

## Frequently asked questions

### Is there a free, self-hosted alternative to Qualtrics?

Yes. LimeSurvey's Community Edition and Potato are free, open-source survey servers you run yourself, and REDCap is free and installed by your institution if the institution belongs to the REDCap consortium. LimeSurvey is the tool Radboud's Faculty of Social Sciences chose for general questionnaires, with quotas and SPSS, R, and Stata exports. Potato imports a Qualtrics `.qsf` and adds balanced assignment, vignettes, list experiments, and Prolific or SONA recruitment, without quotas or SPSS files.

### Should I use REDCap or Qualtrics for a research survey?

Use REDCap for clinical, health, or longitudinal studies at an institution that already runs REDCap. REDCap keeps an audit trail, schedules repeated instruments and events, collects data offline in its mobile app, and costs consortium members nothing. Qualtrics suits general questionnaires and survey experiments, with quotas, a Randomizer for balanced conditions, specialty question types such as heat maps, and conjoint projects as an additional purchase. Potato imports questionnaires from both.

### What should I do before my university's Qualtrics license ends?

Export the survey and the responses separately before the account closes. Qualtrics writes the questionnaire to a `.qsf` file that contains the questions, logic, and flow but no responses, and the response data comes out of the Data & Analysis tab as CSV, SPSS, or another format. The `.qsf` can be imported into Potato with `potato import`, and the response files go to your institution's archive or data repository. Finish both exports before any deletion date your university sets, such as December 1, 2026, at Radboud's Faculty of Social Sciences.

## Further reading

- [Survey research in Potato](/use-cases/survey-research), for the survey features in one place
- [Survey Import](/docs/tools/survey-import) and [Coming from Qualtrics](/docs/guides/coming-from-qualtrics)
- [Move a Qualtrics Survey to Potato Before Your License Ends](/blog/moving-a-qualtrics-survey-to-potato), for one survey moved from the `.qsf` export to a Prolific study and a CSV export
- [Experiment Flow](/docs/experiments/experiment-flow), for conditions, page rules, screen-outs, and counterbalancing
- [Sensitive Questions](/docs/annotation-types/sensitive-questions), for list experiments and randomized response
- [Survey Instruments](/docs/features/survey-instruments)
- [Running a Crowdsourced Study](/docs/guides/crowdsourcing-prolific-mturk)
- [Export Formats](/docs/features/export-formats)
- [Behavioral Experiment Tools Compared](/docs/guides/behavioral-experiment-tools-compared)
- [Annotation Tools Compared](/docs/guides/annotation-tools-compared)

## References

1. Harris, P. A., Taylor, R., Thielke, R., Payne, J., Gonzalez, N., & Conde, J. G. (2009). Research electronic data capture (REDCap): A metadata-driven methodology and workflow process for providing translational research informatics support. *Journal of Biomedical Informatics*, 42(2), 377-381. https://doi.org/10.1016/j.jbi.2008.08.010
2. Chen, D. L., Schonger, M., & Wickens, C. (2016). oTree: An open-source platform for laboratory, online, and field experiments. *Journal of Behavioral and Experimental Finance*, 9, 88-97. https://doi.org/10.1016/j.jbef.2015.12.001
3. Jurgens, D., Chen, M., & Iyer, L. (2026). Potato 2.0: A Comprehensive Annotation Platform with AI-in-the-Loop Support. *Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations)*, 374-386. https://aclanthology.org/2026.acl-demo.37/