Hi everyone, Brand new requester here. We have a large number of stores which we’d like to score on attributes that impact their performance (visibility, ingress-egress, etc.). There is some subjectivity involved in this scoring, so we would like to set up a feedback loop to help the worker align with how we would score a store. Because of this learning curve we’d like to keep the number of workers limited and allow them to work on a large number of stores. We would start small, with five HITs in a batch, and give the worker feedback on their scoring. How can we set up our request so that one worker completes all of the HITs in a batch rather than the HITs being split out among multiple workers? Thanks for your help!
You can assign a qualification to the worker and set the hits to require that qual. This page should have info on doing that: https://docs.aws.amazon.com/AWSMechTurk/latest/RequesterUI/AssigningaQualificationTypetoaWorker.html
The initial setup is going to be the difficult part b/c you can't natively ensure a worker can actually get 5 HITs. You also wont really be able to train them, you're better off using a system where you just recruit the workers who happen to intuitively understand your instructions/needs and/or "guess" correctly on qualification HITs. OCMP5 has a setup similar to what you want. What they do is just release "qualification" HITs periodically. A worker has to complete 10 HITs (ALL HITs have a pre-known answer so the worker can be graded) and they're scored. If they get 9/10 correct they get the qual and can do the work. Yes, some people get lucky (or unlucky) and qualify then do crappy work & vice-versa some folks get screened out who would have been really good but its the quickest/dirtiest initial screening system. The big downside here is that you can't "reserve" 10 HITs for each worker, so you essentially have to release exponentially more HITs than you intend to actually qual people for. Where you'd expect to only release 10 HITs per worker you want to get the qual you'll actually have to release far more because some workers will take 3, others 1-2, and then others will be "stuck" having only 5-6 HITs and never get their score. I can't really give you advice on exact #s you have to release because it will all depend on the popularity of your requester account and if workers share the qual with social networks like this, etc etc. Make a plan to keep their score across batches (you'll need to do external HITs for this) that way if you don't get enough workers in the first pass you can just release more HITs until you do. After that you can start with feedback looping your worker base. But honestly that particular problem is one of MTurks weak points, its better suited to projects that can use consensus to determine accuracy there are no great ways of "training" your workforce on MTurk so its going to have to be something you guys build out for your requirements.
Thank you for your advice. Based on your insights and some additional research, I’m contemplating the following approach to enable a worker to address multiple HITs at a time. Can you tell me whether any of this is out of line? First, I thought I might ask whether anyone on this forum is interested in participating. The pilot HIT is in the MT sandbox under the Requester Dena Whitaker. Anyone who checks it out and is interested may respond here with their worker ID. Second, I would group the five stores for the pilot into a batch of HITs. I understand that if I keep the metadata the same for all of the HITS then they will be grouped together for the worker. I will create a custom qualification for these HITs (say, “Store Classifier 1”) and assign one of the workers who has responded in the forum to this classification. It seems like this would limit all five HITs in the batch to that one worker. Third, I would create a second, third, fourth and fifth batch of five HITs apiece. Each of these batches would be slightly differentiated by the description (Store Classification 2, 3, 4, 5) for the HITs within them. Then, like the first batch, I would create a custom qualification for the HITS within each (say, “Store Classifier 2, 3, 4, 5”) and assign one of the workers who has responded in the forum to each of these classifications. I will not reject any work which appears to have been done in good faith. In the sandbox the reward is listed as $2.50, but that’s the “long term” rate. For this first set of batches I’ll set the rate at $3.50 to help offset the time to go through the instructions. For workers who have done a good job and whom I’d like to offer further batches to do, I will give a $.01 bonus. In the message accompanying the bonus I will give feedback on how the worker’s scoring for each store aligned with our scoring. I hope to develop a core of workers who can devote a significant amount of time to this project. At the end of the project I will also give some bonus to those who make significant contributions. Thanks!
I may be incorrect, but I think you can set a limit on the number of HITs a worker can accept in a batch. Say you assign 5 workers with the "storeClassifier' qual, then post a batch of 25 stores with the max accept limit to 5 per worker... @ChrisTurk can correct me if I'm wrong.
They have to have a qual first, I think CT was referring to posting a qual batch prior to that? And there isn't a solution to ensure people take all 5 hits if he just posts the batch open to all that isn't very wasteful. ex. Andreas burning money on every qual test where people only do 1 of the 5 etc.
Oh yeah, that's kind of what I meant. Give 5 workers the qualification somehow...known worker ids, qual test, etc.., then drop a batch for them. Reading back, do you mean there isn't a way to ensure the qualled workers do all the work? Maybe move some of the per HIT pay to a bonus after completing the 5 HITs...a half now, half later type of thing...sorry, just woke up from a nap and my brain is still fuzzy... A3VGBHJ6N2NQYF sandbox link: https://workersandbox.mturk.com/requesters/A3VGBHJ6N2NQYF/projects?ref=w_pl_prvw Anyway @liam_w, I wouldn't mind giving them a try when it's time.
nah, not qualled workers, unqualled workers. If he posts 500 hits without quals, 5 hits each for 100 workers that's just going to be a free for all and a smaller percentage of workers are actually going to get 5.
Ouch. I can’t even do the sandbox hit. The preview looks interesting though. (I have apparently only done 3 sandbox hits)
Hi Ezridax. I don't quite understand your point on not being able to do the sandbox hit. Are you saying that if a worker hasn't met the minimum required hits in the sandbox (as opposed to having done the minimum number of hits in the published hits section) that they can't try out the hit in the sandbox? Thanks,
Yes. That's what I'm saying. I've completed well over the required number on the main platform, but in the sandbox, I've submitted 3 hits.
I am having the same experience as @Ezridax . The sandbox HIT says I am not qualified because I do not have 1000 HITs--I have way more than that number on Mechanical Turk, but only 1 HIT on the sandbox subsite.
Hi. We have redone the HIT in the sandbox with the required number of HITs set at "> 0". Is it possible to do the sandbox HIT now?
The link is https://workersandbox.mturk.com/projects/3VH5C6LHQ3TQ0IPKAGQSPKL8HC8LL0/tasks?ref=w_pl_prvw
