01/04 - Totally Snowy Tuesday

Hello everyone. Some of y’all may know and some may not. I’ve been actively working at Turkopticon for over two years trying to work on creating real change to make our conditions a little better. Chris T. has me labeled so please be nice but we welcome constructive criticism anytime. Turkopticon is launching a petition against mass rejections and we welcome everyone to sign and share. A victory for us is a victory for all workers.
https://www.coworker.org/petitions/end-the-harm-of-mass-rejections

 

The F Laboratory [A7GI4MUAC5KI] Short Scenario Survey(~ 4 minutes) - $0.50 | PANDA

Unrated

Unrated

Unrated

$27.27 / hour
00:01:06 / completion time

​

Pros:

Cons:

Catchability: Okay (HIT Last Seen: one minute ago)
Qualifications: Total approved HITs >= 100; HIT approval rate (%) >= 95; CR Research Group #1 >= 100; Location In US; Exc: [-942316087-338805] DoesNotExist ;

To read The F Laboratory's full profile check out TurkerView!​

 

Title: Beliefs Survey(~ 3 minutes) | PANDA
Requester: CSA_Evanston [A1KC5KFTR68Q8Q]
TurkerView: [ $18.61 / hour ]
Description: Survey of public and consumer opinions
Duration: 30 Min
Available: 6
Reward: $0.50
Qualifications:

• Exc: [339546-338957] DoesNotExist
• CR Research Group #1 GreaterThanOrEqualTo 100
• Location In US

[3BG0QG8YO7969N1841ACUU1YH5BCSG]​

 

@Sherry I'm not sure if this is the place for questions, but what provisions are you proposing for those workers who *are* scamming the system, abusing AA times, or otherwise deserve their work rejected?

If "Approval Rate" is the amount of work Approved vs Rejected, how are you proposing that only X amount of HITs will affect a workers rating, while rejections still stand for lack of payment?

Using "all workers agree" is pretty bold.

 

Jane So [A1IRC7CQWB8NEG] consumer research(~ 4 minutes) - $0.50 | PANDA

Unrated

Unrated

Unrated

$9.63 / hour
00:03:07 / completion time

​

Pros:

Cons:
Writing

Catchability: Great (HIT Last Seen: a second ago)
Qualifications: Exc: [-1240403534-338817] DoesNotExist ; Total approved HITs >= 500; HIT approval rate (%) >= 95; CR Research Group #1 >= 100; Location In US;

To read Jane So's full profile check out TurkerView!​

 

Perfect place for the question and what we’re proposing is a limit on the number of rejections from batches to a certain amount without counting all the rejections. Example someone did 300 tasks wrong the requester can still reject the work but only 10 rejections would count against the workers rating. If that person is scamming then the requester could and should block them.
And with using the term “all workers” I can’t see a reason why anyone wouldn’t want a way to be able to work with the reassurance that you wouldn’t kill your approval rating because of one bad decision batch. If I’m wrong in that I’m sorry.

 

I replied to the whole post. Still not familiar with the setup here. My apologies

 

KOFResearch [A1D676J5YMB5EM] Answer a survey about your workplace experiences - $3.50 | PANDA

Unrated

Unrated

Unrated

$28.38 / hour
00:07:24 / completion time

​

Pros:

Cons:

Catchability: Okay (HIT Last Seen: 6 minutes ago)
Qualifications: Employment Status: Employed full time (35+ hours per week) EqualTo 1; Total approved HITs > 500; HIT approval rate (%) > 90; Location EqualTo US;

To read KOFResearch's full profile check out TurkerView!​

 

I personally can get behind this. I like having feedback and knowing that I'm doing something correct, so I honestly rarely do batches for the exact fear you're describing. In a normal workplace, someone doing a large-scale task wrong would be caught a lot quicker than 300 tasks completed, giving the worker the ability to correct their incorrect procedure and complete the rest of their tasks successfully.

 

What metric are you using for "workers rating"?

I may just come off as a cranky old timer, but I've completed enough HITs that I have a strong "buffer" against a mass rejection. That's the safety net. I am more concerned about unwarranted "blocks" or anything that might risk a Masters qualification or get an account unjustly escalated to Amazon for a suspension, verification, or anything of that sort.

 

Jocelyn Belanger [A1FRNYP4JBDAB6] A survey on life habits and politics - max 20 min(~ 20 minutes) - $2.00 | PANDA

Generous

Unrated

Unrated

$22.64 / hour
00:05:18 / completion time

​

Pros:

Cons:
strange questions

Catchability: Great (HIT Last Seen: a second ago)
Qualifications: TP Panel: -290823352 N >= 100; Exc: [339558-338969] DoesNotExist ; CR Research Group #1 >= 100; Location In US;

To read Jocelyn Belanger's full profile check out TurkerView!​

 

Title: Opinions survey(~ 5 minutes) | PANDA
Requester: Carlos Torelli [ARYXH80FTLKS3]
TurkerView: [ $10.39 / hour ]
Description: This is a 5-minutes survey about opinions on brands and varied issues
Duration: 30 Min
Available: 1
Reward: $0.50
Qualifications:

• TP Panel: 1989414341 N GreaterThanOrEqualTo 100
• Exc: [339364-338775] DoesNotExist
• CR Research Group #1 GreaterThanOrEqualTo 100
• Location In US

[3S6FLHYRRIMJ5ITKZUWULF6QSS2755]​

 

Brian Young [A142M5RXHGJ8BG] Informed Consent Form for Browser Extension Test - $1.00 | PANDA

Unrated

Unrated

Unrated

$12.50 / hour
00:04:48 / completion time

​

Pros:

Cons:
Rejected for: Survey did not follow MTurk's terms of service; I will repost the survey once it is fully anonymous.....so yeah, unfair we are being punished for their mistake, if this is not overturned this requester will be blocked.

Catchability: Impossible (HIT Last Seen: 20 hours ago)
Qualifications:

To read Brian Young's full profile check out TurkerView!​

 

That's totally not okay. This is definitely a rejection worth disputing.

 

Title: A survey on life habits and politics - max 20 min(~ 20 minutes) | PANDA
Requester: Jocelyn Belanger [A1FRNYP4JBDAB6]
TurkerView: [ $12.50 / hour ]
Description: A survey for Democrats
Duration: 60 Min
Available: 9
Reward: $2.00
Qualifications:

• TP Panel: -290823352 N GreaterThanOrEqualTo 100
• Exc: [339558-338969] DoesNotExist
• CR Research Group #1 GreaterThanOrEqualTo 100
• Location In US

[3FKD865UYPMUG4CZPJYBX4HIHW0MU9]​

 

Jocelyn Belanger [A1FRNYP4JBDAB6] A survey on life habits and politics - max 20 min(~ 20 minutes) - $2.00 | PANDA

Unrated

Unrated

Unrated

$22.50 / hour
00:05:20 / completion time

​

Pros:

Cons:

Catchability: Great (HIT Last Seen: a second ago)
Qualifications: TP Panel: -290823352 N >= 100; Exc: [339558-338969] DoesNotExist ; CR Research Group #1 >= 100; Location In US;

To read Jocelyn Belanger's full profile check out TurkerView!​

 

At this time we’re using the metric that Amazon gives us on our dashboard.

I would love to discuss what you have in mind that could help us in those situations also because I’m on board to get us the best possible working conditions so we don’t have to be in constant worry that we’re losing our accounts or our approval rating is going to be tanked.

Also we did a survey on Mturk and mass rejections was the top issue that workers wanted a fix for. TO has been having open forums and private discussions with Turkers to try and work on the solution together. This is hopefully just the first step in us workers finally getting a say in how we work.

 

The Approval Rating? I'm not trying to be argumentative I just don't mathematically understand how this would work. (I'm hoping maybe I'm misunderstanding or missing something?)

The current Approval Rating is total amount of Approved vs Rejected HITs.
Saying only X amount of rejections per requester would "count" implies that only X HITs per requester would affect both Approval and Rejection rates (or some new variant of them?) The rest are logged, but unweighted somehow?

 

Title: A short judgement(~ 1 minutes) | PANDA
Requester: Research2020 [A22U02WKALJD90]
TurkerView: [ $10.37 / hour ]
Description: survey
Duration: 30 Min
Available: 26
Reward: $0.20
Qualifications:

• Total approved HITs GreaterThanOrEqualTo 100
• Exc: [10865146-338972] DoesNotExist
• HIT approval rate (%) GreaterThanOrEqualTo 90
• CR Research Group #1 GreaterThanOrEqualTo 100
• Location In US

[3D7N1211PDAIOESKNB4QR49AUSCAMX]​

 

Essentially think of it like limiting the rejection to the group_id. So you're fixing the "weighting" in Amazon's system which is entirely broken (and why I personally am fine w/ TO's position on this more or less).

ATM, if a survey gets rejected it doesn't matter at all, and it weighs nothing. If you go ham on a CI batch and they rejected the entire thing you have no account lol.

IMO it's also bad for requesters. Disproportionately weighing impact this way prevents requesters from just rejecting a huge batch of invalid results even if it's useless to them. Scamming isn't even the problem, some people are just dumb/unsuited to every task. Which they don't know until the requester reviews it, but at that point there is no middle ground because the system is complete shit.

As I type this, weighting by $ value would also be interesting.

And ofc agree, people w/ 1,000,000 HITs largely don't have to personally care about this, but I do think it's an improvement to the overarching system design which is generally good even if I don't care about the "justice" shit.