10/02 - TurkerView - Return That HIT! Warning/Review API

Hey folks,

So this is a new approach for TV entirely. I feel like nowadays the website does a good job at collating/displaying data about Requesters to visitors who need it. The API generally provides folks a quick glance at the pay ratings/history of a Requester.

What we don't have is a good heads-up system of identifying all the crap we invariably throw back into the pool from day to day. TV has always been restricted to completed HIT reviews only - which has been very good for data quality but has definitely left some blind spots in the dataset.

Starting today you'll no longer have to fully complete a HIT in order to leave a note for other workers on why it isn't worth finishing.

Returned HIT Reviews
These are gonna be kind of weird since they wont go directly to the website, instead they'll only be displayed on the task completion page for a HIT in your queue.

The upside is that instead of having to finish a really crappy (or impossible) HIT in order to leave a note for others folks TurkerView will now facilitate an on-site communication network between workers to provide a heads up display about HITs that deserve a warning note to incoming users.

In order to try to keep data quality standards close to those for the on-site reviews these are tied to the action of returning a HIT out of your queue. This ensures that users who are leaving the reviews have at least looked over the HIT (probably even started doing it) before deciding it is indeed so bad its not even worth submitting.

The UI on the task/HIT page is fairly simple:


The Return Report Form


I started off by keeping this as simple as possible. A few checkboxes, an optional comment box & you can get on with your day. No need to leave MTurk or even your work queue, as soon as you submit the data TVJS will finish returning the HIT for you and MTurk will forward you along to the next HIT in your queue to keep you working & wasting less time. Anyone who loads the HIT after you return it & has TVJS installed with get a notification that the HIT has been reported as bad & they can give it a looksie.

In keeping with the "waste less time" theme we also don't want to force workers to come to the website to get a heads up on bad HITs:

The Returned Report Overview


The red warning buttons as seen in the top image will only appear on HITs that have active reports, when they do exist they'll pull up the above overview page of reports w/ comments (where available) and some stats about why workers returned the HITs previously.

The overview will show you how many users reported each specific category. On top of that if there is an unpaid screener in the HIT their progress in minutes/seconds will be displayed so you can make an informed decision about how long the screening process for a HIT can take. If most folks get screened out in 20-30s for a $5 HIT, you may be willing to continue on.

Alternatively, requesters like Pirc should quickly gain a reputation for their annoying screening practices. Either way, the data is simply there to help you make an informed decision on the HIT

We also provide comments (where available) by users. Some of these comments are auto-generated by TV (in cases where users mark a HIT as underpaid, it'll show you their hourly at the time they decided to bail out of the HIT). While not perfect, that along w/ comments will hopefully provide a little insight into the whys they returned the HIT.

Again, none of this data will be fed to the website directly & it will not affect API based ratings at the moment. Its an entirely new system designed for a purpose TV previously has not been equipped to handle. I personally think this system will be a little more resilient to "noise" since much of it is fluid data but some larger conclusions/additions to requester profiles will likely be in the works based on it.

TurkerViewJS 7
Aside from the above changes TVJS has gotten some QoL love:

• The wage tracker now has "sticky nav" attributes and will pop around to the upper right corner of the screen when you're working on a HIT.
• The wage tracker will now update the titlebar tab title to indicate the wage/completion timer so you can still see it when completing another survey.
  • Because so many other scripts also have this functionality you can toggle it off by clicking on TurkerView in the upper nav area to bring up the dashboard and going to the settings (click the gear icon in the upper right of the modal)
• Minor visual changes to certain popovers/tooltips/etc

Feedback, Questions, Suggestions, Etc
This was a quick sprint build over the weekend based off a short reddit convo. I honestly did not have any of this planned or mocked up so there will be gaps in coverage, missing checkbox categories workers want, etc etc.

I'm sure there will also be bugs or missing/incomplete features once it goes through the grind. Please just @me on the forum or DM me.

But in general any bugs/snafus you hit, improvement avenues you see, etc etc as always immensely appreciated.

 

good morning let's make this the best tuesday ever!

 

Title: THOSE BELONGING TO MINORITY GROUPS AND FEMALES ENCOURAGED TO RESPOND: Answer a survey about the atmosphere of cafes | PANDA
Requester: Amelie Burgess [A1AK189WS4GKNZ]
TurkerView: [ $25.85 / hour ]
Description: The project’s aim is to gain an understanding of how culture influences the perceptions of atmospherics in the servicescape. This understanding will be used to examine if and how these perceptions affect behaviors and responses.
Duration: 01:00:00
Available: 1
Reward: $2.50
Qualifications:

• None

[3RB76ORN6TG6830VBUSGH9JM3QQEE6]​

I can't review this one but it took me about 5 minutes.

 

At first glance I was not sure that I was a big fan of this feature, but after considering the entire post it seems like it could be quite useful.
I'm a bit stupid, so it will take me awhile to wrap my head around the change, but either way Thanks for the hard work

 

Minorities and women get all the good surveys, what BS

 

I know I'm sick today and time sure feels like it's staying still but I'm confident it's the 2nd and not the 1st of October

 

C'mon man, I was hoping to get a bite :O

 

So. The student friendship $3 hit. It says the average bonus is between $5-$8. Cool. Says it takes 45 minutes. I do the hit. Takes over an hour. Fine. I earn a grand total of $1.50 bonus. Fine, I'm an idiot. Fill out the survey and submit. It says debugging task complete, if this was a real hit it would now submit to amazon turk. So I just wasted an hour on a hit I can't submit. The only reason I am not having a screaming fit at the moment is my roommate is sleeping and he gets cranky when I yell at the internet in the middle of the night,

 

I just realized I can submit the hit. And that it says the minimum bonus is $3. Life is wonderful.

 

Robot Ethics and Aerial Vehicles Lab [A3C79NCG9MFEH8] Robot Experiment in Emergency Scenarios - $3.00 | PANDA

Unrated

Unrated

Unrated

$15.65 / hour
00:11:30 / completion time

Pros:
A easy game

Cons:
Cannot copy mturk id/completion code
"game" took forever to load
An obnoxious amount of threats of rejection

Qualifications: Total approved HITs >= 100; HIT approval rate (%) >= 90; Location EqualTo US; Single Attempt DoesNotExist ;

To read Robot Ethics and Aerial Vehicles Lab's full profile check out TurkerView!​

 

Did we loose a day/24 hours? My calendar says 10/2. Or is this not the daily thread? Thank you. If we could turn back time ... no.

 

Gee thanks , now I have that terrible song in my head

 

Alex [A193ITC4S0HPC4] Measuring your internet experience - $1.50 | PANDA

Unrated

Unrated

Unrated

$42.86 / hour
00:02:06 / completion time

Pros:

Cons:
Have to install chrome add on

Qualifications: Masters Exists ;

To read Alex's full profile check out TurkerView!​

 

Devin Fidler [A2HIKOY3S9L9O3] Brainstorm Names for a Project Focused on Extreme Innovation - $0.25 | PANDA

Generous

Unrated

Unrated

$13.04 / hour
00:01:09 / completion time

Pros:

Cons:

Qualifications: Masters Exists ;

To read Devin Fidler's full profile check out TurkerView!​

 

Jollibee spaghetti at 1:37 a.m. is the adultest life

 

Junsub Yi [A1R8M99IAULKFJ] Perceptions and Emotions about IoT Device(Smartwatch) - $1.20 | PANDA

Unrated

Unrated

Unrated

$11.11 / hour
00:06:29 / completion time

Pros:
Fair compensation

Cons:
Trapped forever in the bubble mines

Qualifications: Old participants_smartwatch DoesNotExist ; HIT approval rate (%) > 95; Location EqualTo US;

To read Junsub Yi's full profile check out TurkerView!​

 

Roomy got back from Montreal tonight...I got a couple hoodies and fridge magnets

Spoiler: cool eh