Make Qualtrics survey work with Amazon Mechanical Turk smoothly. Yes, I mean smoothly!
I use Qualtrics to do advanced surveys that requires "drag and drop", "flow logic", etc. And then I use Amazon Mechanical Turk to recruit subjects for really cheap (I usually do $0.15/subject). This post is about how to make the two services work together.
It seems that all other existing approaches require subjects to copy a "confirmation code" here and there. But what I'll show you here is a very smooth process: Mturk workers would see the survey inside of Mturk, they work on it, submit, and get paid (see below):
Before you begin, I assume you know how to run Mturk Command Line Tools. If you don't, download it here, and see how to use it. If you have to do lots of Mturk stuff, this is a tool you can't miss. To test whether your Mturk Command Line Tools is setup correctly, you can run getBalance.sh (for Linux) or getBalance (for Windows). Make sure it runs correctly before you continue.
Create a Qualtrics survey like you would normally do.
Go to "Survey Flow", add an "Embedded Data" block. Move it to the beginning of the flow. Create two fields "workerId" and "assignmentId".
Note that these two fields are case-sensitive. These two fields will be recorded by Qualtrics automatically for each response. See screenshots below:
Go to "Survey Options", find "Survey Termination". In "Redirect to a URL", paste the following:
Note that this URL is the same for any surveys. Basically, we ask Qualtrics to tell Mturk that a survey is done so Mturk can pay the worker. Note that at the end of the URL, "foo=bar" is needed because Mturk requires some form of input. See screenshots below.
Update 1: To test with sandbox, the URL is:
Update 2: You can employ the Qualtrics logic to disable the "Submit" button if a worker hasn't press "Accept HIT" yet. See MTurk ExternalQuestion document for more details. Basically, it employs checking whether the "assignmentId" embed data equals to "ASSIGNMENT_ID_NOT_AVAILABLE" to decide whether a worker has pressed the button or not. I haven't tried this approach yet.
Launch the Qualtrics survey as you would normally do. Copy the survey URL. Screenshot:
Download mturk_conf.zip file. Unzip it. You'll find three files: demo.csv, demo.txt, and external.xml.
Copy the survey URL to replace the URL in demo.csv. Note: Don't add an empty line at the end of demo.csv, because an empty line is considered as a URL too, even though empty.
You can also modify demo.txt to change how much you'll pay to each subjects, how many subjects you'll recruit, etc.
Bring up you command line, execute the following command:
loadHITs.sh -input demo.csv -label demo -properties demo.txt -question external.xml
You'll see some info like the following. If you use Windows, use loadHITs instead (I haven't tested yet, but it should work the same). If you don't know how to use the Mturk command line tool, read instructions here.
Question File: external.xml
Preview mode disabled
Start time: Mon Oct 15 21:46:19 UTC 2012
Created HIT 1: HITId=2U78PJWKUDD8INLBK30Z6B6VKO79KV
You may see your HIT(s) with HITTypeId '21GQY2RCJK63KWOBBRRK0IAQ9UY5UD' here:
End time: Mon Oct 15 21:46:20 UTC 2012
--[Done Loading HITs]----------
Total load time: 0 seconds.
Successfully loaded 1 HITs.
Update 3: To test with sandbox, you need to use the "-sandbox" option. Here's the code (note "-sandbox" at the end):
loadHITs.sh -input demo.csv -label demo -properties demo.txt -question external.xml -sandbox
The loadHITs command will tell you the URL of the HIT you just created. Preview it on Mturk, relax, and wait for the responses. To get 60 responses at $0.15/response, it usually takes about 4 hours. You'll get much faster responses if you relieve some qualification requirements in demo.txt.
Updated on 2013-05-08: Added a few notes about setting up Mturk Command Line Tools.
Updated on 2013-07-17: Added notes on sandbox and disable "Submit" button on Qualtrics.
Updated on 2014-01-30: Added notes on how to post tasks to sandbox using the "-sandbox" option in CLT.