04/29 - Wacky Wednesday!

Lol requestor here. We intended for this to take 5 min. The bonus is around $0.50 if you read the question. I can complete it in ~1.5 minutes (and so should most people) once you read the first question and know the format.

The problem is that there's some people (or bots?) who just seem to click through extremely fast, and get the one obvious anti-bot question wrong (and there's probably more bots that split through). We requested 20 people and have to throw out at least 5 of the responses.

My question is, how should I improve the survey/reward structure to get rid of this kind of behavior? We want people to actually think about this (this is a research survey) and should take at least 15 seconds per question. We're thinking about making the reward scaled from $0.10 to ~$2.

 

MTurk Responses [AM8EADG1K0864] PDF Viewer Evaluation (<5 min) - MUST USE DESKTOP/LAPTOP COMPUTER TO COMPLETE - $1.00 | PANDA

Generous

Unrated

Unrated

$42.86 / hour
00:01:24 / completion time

​

Pros:

Cons:

Catchability: Great (HIT Last Seen: one minute ago)
Qualifications: Total approved HITs >= 1; HIT approval rate (%) >= 90; Location EqualTo US;

To read MTurk Responses's full profile check out TurkerView!​

 

Pe: $52.

Today is good for m

 

@ChrisTurk

 

https://www.cambridge.org/core/jour...&utm_content=&utm_campaign=MNE_campaign_MMMYY

TurkPrime (now CloudResearch) is worth the 5% fee they tack on for the global worker blocklist & handful of other useful point/click features. But if you're using qualtrics, the iphub integration burleigh/etc propose in their work is really good (for now)

 

Title: Short fun research study(~ 15 minutes) | PANDA
Requester: phillab [A36MJBQYSQS7EB]
TurkerView: [ $24.63 / hour ]
Description: Read short stories and answer a few questions
Duration: 25 Min
Available: 7
Reward: $3.00
Qualifications:

• Total approved HITs GreaterThanOrEqualTo 500,
• HIT approval rate (%) GreaterThanOrEqualTo 95,
• Exc: [-651537366-215518] DoesNotExist
• Location In US,

[398WDAEDRHDP31A6ZM35R4XJPWTHUA]​

 

Yep, we've seen this. Definitely a good solution, but unfortunately we had to create a website since Qualtrics didn't allow for the randomization we wanted. We're currently concerned about turkers rushing through the survey (US-based or not) instead of reading/thinking about the question. Would you happen to have any ideas for this? Thanks!

 

You can use the same protocol just (preferably) with your serverside language (php? python? god forbid ruby? ) or javascript (i cri)

javascript will be the easiest for timed pages, hide the element and only show it after a setinterval / timeout. It's not foolproof, but no one is custom rigging against your unique elements for a one off task.

 

Sorry, I realize I've written this horribly unprofessionally it's late in the day & I'm not used to doing recommendations for requesters on the daily thread The above is how I'd seriously approach it, though. Happy to take a look at code if you want to shoot me a DM / email with your task link if you need help w/ the quick/dirty JS stuff.

 

For shame, ChrisTurk, so unprofessional.

 

Me: Y'all don't act up on the daily requesters sometimes read our shenanigans
Also me: hold my English thesaurus I'm going in with the 12-year old on xbox language pack.

 

Title: We’d like your feedback to help us choose a name for our new app. | PANDA
Requester: HiRoad Research [A19AME7AQEZ8C6]
TurkerView: [ $32.08 / hour ]
Description: Should take 5 minutes
Duration: 600 Min
Available: 0
Reward: $1.00
Qualifications:

• App pilot DoesNotExist
• HIT approval rate (%) GreaterThan 95,
• Total approved HITs GreaterThan 50,
• Location EqualTo US,

[3OLDKDGFDNFLMDOGKHCKUJO0WZ6DO7]​

 

GITGUD!

 

phillab [A36MJBQYSQS7EB] Short fun research study(~ 15 minutes) - $3.00 | PANDA

Generous

Unrated

Unrated

$33.13 / hour
00:05:26 / completion time

​

Pros:

Cons:

Catchability: Okay (HIT Last Seen: 8 minutes ago)
Qualifications: Total approved HITs >= 500; HIT approval rate (%) >= 95; Exc: [-651537366-215518] DoesNotExist ; Location In US;

To read phillab's full profile check out TurkerView!​

 

Ah I must have missed the part with timed elements, I thought the paper was more on tracing IPs/VPNs. Great idea, we should be able to code it up relatively quickly. Thanks for the offer though!

 

Noah Ringler [A1WTMPMGO4003K] Coping with Covid-19(~ 12 minutes) - $1.10 | PANDA

Fair

Unrated

Unrated

$7.76 / hour
00:08:30 / completion time

​

Pros:

Cons:

Catchability: Great (HIT Last Seen: a second ago)
Qualifications: Exc: [210052-209463] DoesNotExist ; TP Panel: -1318032489 N >= 100; Total approved HITs >= 50; HIT approval rate (%) >= 85; Location In US;

To read Noah Ringler's full profile check out TurkerView!​

 

UPenn Researcher [A2M70XENI0CWIR] Behavioral Survey (1-2 minutes)(~ 2 minutes) - $0.20 | PANDA

Fair

Unrated

Unrated

$9.60 / hour
00:01:15 / completion time

​

Pros:

Cons:

Catchability: Great (HIT Last Seen: 18 seconds ago)
Qualifications: Total approved HITs >= 100; Exc: [-2915153-215484] DoesNotExist ; HIT approval rate (%) >= 95; Location In US;

To read UPenn Researcher's full profile check out TurkerView!​

 

Fiona Munro [A1KQXJ2QKUCU2L] Climate Change Messaging Survey - $1.00 | PANDA

Generous

Unrated

Unrated

$33.03 / hour
00:01:49 / completion time

​

Pros:

Cons:

Catchability: Great (HIT Last Seen: 9 seconds ago)
Qualifications: Location EqualTo US;

To read Fiona Munro's full profile check out TurkerView!​

 

Oh no, the paper was only specifically in response to the "bots"/ foreign VPN accounts.

FWIW timed elements will only really force workers to sit there. Being honest, a few wont think any harder than they otherwise would have, they'll just alt/tab to something else.

Lazily pasting from another email I sent this week on that note:

Again, YMMV, you might also find some use in screening out the higher end workers (ie, limit to total approved < 500 -1,000) but that's really a total lazy shot in the dark attempt (and also someone is going to throw rocks at me for saying it here, but it's one of those quick/dirty ways of doing this kinda thing when you only need a teeny tiny sample anyway [20 p's is almost nothing for mturk studies\)