Make Qualtrics survey work with Amazon Mechanical Turk smoothly. Yes, I mean smoothly!

I use Qualtrics to do advanced surveys that requires "drag and drop", "flow logic", etc. And then I use Amazon Mechanical Turk to recruit subjects for really cheap (I usually do $0.15/subject). This post is about how to make the two services work together.

It seems that all other existing approaches require subjects to copy a "confirmation code" here and there. But what I'll show you here is a very smooth process: Mturk workers would see the survey inside of Mturk, they work on it, submit, and get paid (see below):

mturk0.png

Before you begin, I assume you know how to run Mturk Command Line Tools. If you don't, download it here, and see how to use it. If you have to do lots of Mturk stuff, this is a tool you can't miss. To test whether your Mturk Command Line Tools is setup correctly, you can run getBalance.sh (for Linux) or getBalance (for Windows). Make sure it runs correctly before you continue.

Step 1:

Create a Qualtrics survey like you would normally do.

Step 2:

Go to "Survey Flow", add an "Embedded Data" block. Move it to the beginning of the flow. Create two fields "workerId" and "assignmentId".

Note that these two fields are case-sensitive. These two fields will be recorded by Qualtrics automatically for each response. See screenshots below:

mturk1.pngmturk2.png

Step 3:

Go to "Survey Options", find "Survey Termination". In "Redirect to a URL", paste the following:

https://www.mturk.com/mturk/externalSubmit?assignmentId=${e://Field/assignmentId}&foo=bar

Note that this URL is the same for any surveys. Basically, we ask Qualtrics to tell Mturk that a survey is done so Mturk can pay the worker. Note that at the end of the URL, "foo=bar" is needed because Mturk requires some form of input. See screenshots below.

mturk3.pngmturk4.png

Update 1: To test with sandbox, the URL is:

https://workersandbox.mturk.com/mturk/externalSubmit?assignmentId=${e://Field/assignmentId}&foo=bar

Update 2: You can employ the Qualtrics logic to disable the "Submit" button if a worker hasn't press "Accept HIT" yet. See MTurk ExternalQuestion document for more details. Basically, it employs checking whether the "assignmentId" embed data equals to "ASSIGNMENT_ID_NOT_AVAILABLE" to decide whether a worker has pressed the button or not. I haven't tried this approach yet.

Step 4:

Launch the Qualtrics survey as you would normally do. Copy the survey URL. Screenshot:

mturk5.png

Step 5:

Download mturk_conf.zip file. Unzip it. You'll find three files: demo.csv, demo.txt, and external.xml.

Copy the survey URL to replace the URL in demo.csv. Note: Don't add an empty line at the end of demo.csv, because an empty line is considered as a URL too, even though empty.

You can also modify demo.txt to change how much you'll pay to each subjects, how many subjects you'll recruit, etc.

Step 6:

Bring up you command line, execute the following command:

loadHITs.sh -input demo.csv -label demo -properties demo.txt -question external.xml

You'll see some info like the following. If you use Windows, use loadHITs instead (I haven't tested yet, but it should work the same). If you don't know how to use the Mturk command line tool, read instructions here.

--[Initializing]----------
Input: demo.csv
Properties: demo.txt
Question File: external.xml
Preview mode disabled
--[Loading HITs]----------
Start time: Mon Oct 15 21:46:19 UTC 2012
Created HIT 1: HITId=2U78PJWKUDD8INLBK30Z6B6VKO79KV
You may see your HIT(s) with HITTypeId '21GQY2RCJK63KWOBBRRK0IAQ9UY5UD' here:
https://www.mturk.com/mturk/preview?groupId=21GQY2RCJK63KWOBBRRK0IAQ9UY5UD
End time: Mon Oct 15 21:46:20 UTC 2012
--[Done Loading HITs]----------
Total load time: 0 seconds.
Successfully loaded 1 HITs.

Update 3: To test with sandbox, you need to use the "-sandbox" option. Here's the code (note "-sandbox" at the end):

loadHITs.sh -input demo.csv -label demo -properties demo.txt -question external.xml -sandbox

Step 7:

The loadHITs command will tell you the URL of the HIT you just created. Preview it on Mturk, relax, and wait for the responses. To get 60 responses at $0.15/response, it usually takes about 4 hours. You'll get much faster responses if you relieve some qualification requirements in demo.txt.

That's it.

Updated on 2013-05-08: Added a few notes about setting up Mturk Command Line Tools.
Updated on 2013-07-17: Added notes on sandbox and disable "Submit" button on Qualtrics.
Updated on 2014-01-30: Added notes on how to post tasks to sandbox using the "-sandbox" option in CLT.

Category: 

Comments

This looks interesting but I don't see where you modify the HTML code in MTurk's interface. Can you add that to this writeup?

You don't edit any HTML code in MTurk interface. Your survey is edited at Qualtrics side.

I found Amazon's CLI instructions really clumsy and stupid.

Put the unzipped aws-mturk-clt-1.3.1 folder into your home directory and add the following lines to your .profile

export MTURK_CMD_HOME=~/aws-mturk-clt-1.3.1/
export JAVA_HOME=`/usr/libexec/java_home`

As long as you are using OS X 10.5 or later, you'll be set.

Also, I couldn't use a normal AWS user key pair, I had to use my accounts key pair at http://aws.amazon.com/security-credentials

This looks like it will work perfectly for my study design. I'm having trouble with Java, though- I get the "bad version number in .class file" error. Any suggestions?

Just install Java 6 or above.

i got struck on step6. the command line said there is sth wrong with loadHITS.bat/ loadHITS.sh
So how can i fix it?

i installed Java version7, but I still get the "bad version number in .class file" error. How can i fix that? thanks!

for Java problems, you should consult Amazon mturk command line documentation. This post assumes you know how to do it.

Had this same problem when I started using MTurk a few weeks ago (running on Windows 7). Hopefully you've since figured it out, but in case you haven't, the problem is the result of the CLT being packaged with an outdated Java version. To fix this, you need to delete the old Java Runtime Environment folder from whatever your CLT installation directory was (default is C:\mech-turk-tools-1.3.1). Then, install the latest (jre7) version of Java. Finally, set the system variable JAVA_HOME to the directory where the latest java version is installed.

There's a good response in the AWS forums here that has more details.
https://forums.aws.amazon.com/message.jspa?messageID=388586#388585

This works perfectly. Thanks!

Hi,

I am new to Mechanical Turk and I am learning the cmd line tools. However, I am not a beginner and I was wondering if you can actually include some additional qualifications criteria before having turkers take the survey. For example, I have a question on whether the respondent has children or not, I would not wish to pay the respondent if they only answer with a no which practically ends the survey. How can you do that?

Thanks!
Mal

For simplicity, I would just include an honor code checkbox: "By checking this box, I attest ...". A more sophisticated 2-steps approach is to have people first take another 1 question survey and then grant them qualification accordingly, and then only allow those people who have certain qualifications to take the survey.

Is there an easy way to use incentive-based pay with Mturk and qualtrics?

How can I preview the survey before actually distribute it?

To preview the survey, you just need to use the sandbox mturk site. Please see the article about the URL for mturk sandbox.

Problem solved.
This is a great instruction. Thank you so much!

This was a great guide; saved me from that bulky "please enter your code here" format I was using before. Thanks so much!

I tried to use this guide, but it won't seem to work for me. I created the study, followed each section of this guide step by step, yet when anyone tries to submit their hit, they end up getting this error message:

"There was a problem submitting your results for this HIT.

This HIT is still assigned to you. To try this HIT again, click "HITs Assigned To You" in the navigation bar, then click "Continue work on this HIT" for the HIT. If this problem persists, you can contact the Requester for this HIT using the "Contact" link above.

To return this HIT and continue working on other HITs, click the "Return HIT" button."

Regardless of what my participants do, the error message still appears. Any thoughts on what the issue might be?

Derek -- Please make sure to follow the instructions strictly. The problem looks like you didn't setup the "submitting url" correctly in Qualtrics. I've done this 20+ times and so are other people, so the instructions can't be wrong. You can also test your stuff on Sandbox Mturk before actually posting it to the production site.

I tried downloading the command line tools, but have a mac. Is there a package that will work with osx, or should I try again on a windows machine?

I tried downloading the command line tools, but have a mac. Is there a package that will work with osx, or should I try again on a windows machine?

I am new to this, and to CLT... I think I made a couple of mistakes... are the variables workerId and assignmentId case sensitve? I accidently used workerid and assignmentid.

When I ran the CLT it generated the HIT but posted it on the production site - even though I used the Update 1 URL:

https://workersandbox.mturk.com/mturk/externalSubmit?assignmentId=${e://Field/assignmentId}&foo=bar

But perhaps that's because I used assignmentid and not assignmentId ??

When the HIT got posted the workers tried to do the survey but it was not working properly (again - maybe because of the error described above?)

You need to use the "-sandbox" option in the command line tools. Otherwise the HIT is posted to mturk production site. The URL is to tell Qualtrics to inform mturk the job is done.

There seems to be a small change in the Amazon API: it appears to expect an additional "title" field in demo.txt. Without it, the current solution will return a 400 error and complain about a missing 'title'.

To solve this, just I just added an extra line in demo.txt with (for example) title:Quick survey (6 questions) like so:

title = Quick survey (6 questions)
title:Quick survey (6 questions)
description:This survey studies people's news reading behaviors. Instant approval.
keywords:politics,survey,news

# how much you'll pay each subject
reward:0.12

# how many subjects do you want
assignments:60

... etc

Add new comment

CAPTCHA
This question is for testing whether or not you are a human visitor and to prevent automated spam submissions.
Image CAPTCHA
Enter the characters shown in the image.