04/29 - Wacky Wednesday!

Discussion in 'Daily mTurk HITs Threads' started by Tripsa, Apr 29, 2020.

Tags:
Thread Status:
Not open for further replies.
  1. freeze0117

    freeze0117 New Turker

    Messages:
    3
    Ratings:
    +2
    Lol requestor here. We intended for this to take 5 min. The bonus is around $0.50 if you read the question. I can complete it in ~1.5 minutes (and so should most people) once you read the first question and know the format.

    The problem is that there's some people (or bots?) who just seem to click through extremely fast, and get the one obvious anti-bot question wrong (and there's probably more bots that split through). We requested 20 people and have to throw out at least 5 of the responses.

    My question is, how should I improve the survey/reward structure to get rid of this kind of behavior? We want people to actually think about this (this is a research survey) and should take at least 15 seconds per question. We're thinking about making the reward scaled from $0.10 to ~$2.
     
    • Today I Learned Today I Learned x 1
  2. awdrf

    awdrf Well-Known Turker

    Messages:
    6,605
    Ratings:
    +5,459
    MTurk Responses [AM8EADG1K0864] PDF Viewer Evaluation (<5 min) - MUST USE DESKTOP/LAPTOP COMPUTER TO COMPLETE - $1.00 | PANDA


    Generous

    Unrated

    Unrated
    $42.86 / hour
    00:01:24 / completion time
    Pros:
    Cons:


    Catchability: Great (HIT Last Seen: one minute ago)
    Qualifications: Total approved HITs >= 1; HIT approval rate (%) >= 90; Location EqualTo US;

    To read MTurk Responses's full profile check out TurkerView!
     
  3. samoth84

    samoth84 Survey Slinger

    Messages:
    6,744
    Gender:
    Male
    Ratings:
    +14,778
    Pe: $52.

    Today is good for m :)
     
    • 5/5 Pay 5/5 Pay x 6
    • Like Like x 1
  4. cheyenne

    cheyenne Survey Slinger

    Messages:
    6,583
    Gender:
    Female
    Ratings:
    +7,691
    @ChrisTurk
     
  5. ChrisTurk

    ChrisTurk Administrator

    Messages:
    56,724
    Ratings:
    +163,220
    https://www.cambridge.org/core/jour...&utm_content=&utm_campaign=MNE_campaign_MMMYY

    TurkPrime (now CloudResearch) is worth the 5% fee they tack on for the global worker blocklist & handful of other useful point/click features. But if you're using qualtrics, the iphub integration burleigh/etc propose in their work is really good (for now)
     
  6. ml50rsd

    ml50rsd Turker

    Messages:
    32
    Ratings:
    +6
    Title: Short fun research study(~ 15 minutes) | PANDA
    Requester: phillab [A36MJBQYSQS7EB]
    TurkerView: [ $24.63 / hour ]
    Description: Read short stories and answer a few questions
    Duration: 25 Min
    Available: 7
    Reward: $3.00
    Qualifications:
    • Total approved HITs GreaterThanOrEqualTo 500,
    • HIT approval rate (%) GreaterThanOrEqualTo 95,
    • Exc: [-651537366-215518] DoesNotExist
    • Location In US,
    [398WDAEDRHDP31A6ZM35R4XJPWTHUA]
     
  7. freeze0117

    freeze0117 New Turker

    Messages:
    3
    Ratings:
    +2
    Yep, we've seen this. Definitely a good solution, but unfortunately we had to create a website since Qualtrics didn't allow for the randomization we wanted. We're currently concerned about turkers rushing through the survey (US-based or not) instead of reading/thinking about the question. Would you happen to have any ideas for this? Thanks!
     
  8. ChrisTurk

    ChrisTurk Administrator

    Messages:
    56,724
    Ratings:
    +163,220
    You can use the same protocol just (preferably) with your serverside language (php? python? god forbid ruby? :p) or javascript (i cri)

    javascript will be the easiest for timed pages, hide the element and only show it after a setinterval / timeout. It's not foolproof, but no one is custom rigging against your unique elements for a one off task.
     
  9. ChrisTurk

    ChrisTurk Administrator

    Messages:
    56,724
    Ratings:
    +163,220
    Sorry, I realize I've written this horribly unprofessionally it's late in the day & I'm not used to doing recommendations for requesters on the daily thread :emoji_sweat_smile: The above is how I'd seriously approach it, though. Happy to take a look at code if you want to shoot me a DM / email with your task link if you need help w/ the quick/dirty JS stuff.
     
    • Love Love x 1
  10. Jbearly77

    Jbearly77 Active Turker

    Messages:
    200
    Gender:
    Male
    Ratings:
    +273
    For shame, ChrisTurk, so unprofessional.
     
    • LOL LOL x 5
  11. ChrisTurk

    ChrisTurk Administrator

    Messages:
    56,724
    Ratings:
    +163,220
    Me: Y'all don't act up on the daily requesters sometimes read our shenanigans
    Also me: hold my English thesaurus I'm going in with the 12-year old on xbox language pack.

    :emoji_sweat_smile:
     
    • LOL LOL x 5
    • 5/5 Pay 5/5 Pay x 1
  12. SunlitSunflower

    SunlitSunflower baby head

    Messages:
    22,494
    Ratings:
    +36,431
  13. samoth84

    samoth84 Survey Slinger

    Messages:
    6,744
    Gender:
    Male
    Ratings:
    +14,778
    GITGUD!
     
  14. Neven123

    Neven123 Well-Known Turker TurkerView Masters

    Messages:
    9,194
    Gender:
    Male
    Ratings:
    +7,688
    phillab [A36MJBQYSQS7EB] Short fun research study(~ 15 minutes) - $3.00 | PANDA


    Generous

    Unrated

    Unrated
    $33.13 / hour
    00:05:26 / completion time
    Pros:
    Cons:


    Catchability: Okay (HIT Last Seen: 8 minutes ago)
    Qualifications: Total approved HITs >= 500; HIT approval rate (%) >= 95; Exc: [-651537366-215518] DoesNotExist ; Location In US;

    To read phillab's full profile check out TurkerView!
     
  15. freeze0117

    freeze0117 New Turker

    Messages:
    3
    Ratings:
    +2
    Ah I must have missed the part with timed elements, I thought the paper was more on tracing IPs/VPNs. Great idea, we should be able to code it up relatively quickly. Thanks for the offer though!
     
    • Like Like x 1
  16. racketrye

    racketrye Well-Known Turker

    Messages:
    2,003
    Ratings:
    +715
    Noah Ringler [A1WTMPMGO4003K] Coping with Covid-19(~ 12 minutes) - $1.10 | PANDA


    Fair

    Unrated

    Unrated
    $7.76 / hour
    00:08:30 / completion time
    Pros:
    Cons:


    Catchability: Great (HIT Last Seen: a second ago)
    Qualifications: Exc: [210052-209463] DoesNotExist ; TP Panel: -1318032489 N >= 100; Total approved HITs >= 50; HIT approval rate (%) >= 85; Location In US;

    To read Noah Ringler's full profile check out TurkerView!
     
  17. racketrye

    racketrye Well-Known Turker

    Messages:
    2,003
    Ratings:
    +715
    UPenn Researcher [A2M70XENI0CWIR] Behavioral Survey (1-2 minutes)(~ 2 minutes) - $0.20 | PANDA


    Fair

    Unrated

    Unrated
    $9.60 / hour
    00:01:15 / completion time
    Pros:
    Cons:


    Catchability: Great (HIT Last Seen: 18 seconds ago)
    Qualifications: Total approved HITs >= 100; Exc: [-2915153-215484] DoesNotExist ; HIT approval rate (%) >= 95; Location In US;

    To read UPenn Researcher's full profile check out TurkerView!
     
  18. klee

    klee Active Turker

    Messages:
    2,202
    Gender:
    Female
    Ratings:
    +2,392
    Fiona Munro [A1KQXJ2QKUCU2L] Climate Change Messaging Survey - $1.00 | PANDA


    Generous

    Unrated

    Unrated
    $33.03 / hour
    00:01:49 / completion time
    Pros:
    Cons:


    Catchability: Great (HIT Last Seen: 9 seconds ago)
    Qualifications: Location EqualTo US;

    To read Fiona Munro's full profile check out TurkerView!
     
  19. AlexZ

    AlexZ Distinguished Butthat The Prophet

    Messages:
    15,503
    Gender:
    Male
    Ratings:
    +47,699
    :tinfoil:
     
    • LOL LOL x 5
  20. ChrisTurk

    ChrisTurk Administrator

    Messages:
    56,724
    Ratings:
    +163,220
    Oh no, the paper was only specifically in response to the "bots"/ foreign VPN accounts.

    FWIW timed elements will only really force workers to sit there. Being honest, a few wont think any harder than they otherwise would have, they'll just alt/tab to something else.

    Lazily pasting from another email I sent this week on that note:
    Again, YMMV, you might also find some use in screening out the higher end workers (ie, limit to total approved < 500 -1,000) but that's really a total lazy shot in the dark attempt (and also someone is going to throw rocks at me for saying it here, but it's one of those quick/dirty ways of doing this kinda thing when you only need a teeny tiny sample anyway [20 p's is almost nothing for mturk studies\)
     
    • LOL LOL x 1
Thread Status:
Not open for further replies.