Hard to marry data quality with speed but you're also sort of forcing people to take short cuts when you lower the pay to substandard pay. I've done a decent amount of work for GeoHive in the past but those .10 cent for 10 images were fairly insulting pay for anyone who has been around. Sure, some new worker might eat that $6/7 hourly up but most conscientious workers will avoid stuff that low. You guys used to pay pretty fair. If your budget has put constraints on your ability to pay, perhaps redesign the tasks to make it more equitable? Of course a redesign has its own costs, understandably. There is a question raised in the YT video that I was curious about. Some of the past GEOhive tasks literally could take a few seconds to make a snap judgement. The mapping tasks specifically where it was like "Does map include buildings" and you'd get section where it was literally all forests could be done in a few seconds. I think it is good to be willing to see a worker's screenshare to understand their workflow. To me this begs the question: Are your tasks ever done in Sandbox by designers to see how long the task should take? It would help if there was a naivete to the task but Sandboxing them should at least give an average time of completion (although it sounds like this is already a known known). Bottomline is: what is the trade off? What hourly are you willing to pay for your tasks? I'd hope you'd offer closer to $10/hour but maybe this isn't your desire? Lastly, it is very much rare---thus appreciated, to have a requester reach out to the workers especially with such thoughtfulness. Just wanted to say thanks for that.
Chris hits the nail on the head. I got my qual revoked, but I don't think I went too fast. And there was no mention about my speed in the reject comments. The only thing I know for sure is that I had a few accidental blank submits. I was prepared for those rejects, but I don't know the error rate for any of the HITs I may have gotten wrong and got rejected. None of us will be 100% though, and that's even more true with HITs like these where they are kind of subjective. I may think one building looks like a warehouse and someone else may think differently. But, the communication is always more important that anything. I'm okay with rejects for bad/blank work, even if it's me. But, rejects plus a qual revoke plus HITs that aren't paid enough for how they are designed and what they require leaves a bad taste in my mouth.
I agree with this approach. Put out a qual and test people through the qual, choose the best results as your working base. I would much prefer a qual that required my full attention and in depth reading and understanding vs attempting a hit and hoping that I am performing it right. Especially if it is a batch of hits, this just leads to someone repeating the same usually innocent mistake over and over. I want to be upfront and state that I don't believe I have this qual, but I wanted to join the discussion because this feels like a real breakthrough for us as workers. To be acknowledged and actually approached for a discussion is never seen and it simply feels great to be regarded as beings with something to add to the discussion.
I don't know if I'm being naive, but I always wondered why, in GeoHIVEs and in PFA's HITs, there wasn't a button to go back one image. Does this possibility exist and I never realized it? Is it something that cannot be done? Many times I acknowledged my mistakes when it was too late, and I would have loved to have the chance to go back to the previous image and fix it: I think this would be already a good way to slightly improve the data.
I'm here. I had a lunch date with our director. I am scorning the GH team for not being in the TH forums.
*Hands you a cold beer, pats you on the back* I will make some recommendations, but this boat is no longer mine to steer.
Thank you so much for the hint! I think it didn't work for me in some occasion, so I stopped trying, but I will try again on the next batch!
I do not do GeoHives hits but I came here to say 2 things. First, thank you for trying to make things better both on your side and on ours. I really appreciate the video that you posted. Second,
Haha no worries I understand. I tried to phrase everything as the non-descript "GeoHIVE Team" but I also have a horrible habit of using "you" for darn near every reference to anything ever, so forgive the poor English. I'm an American, its my right to be too lazy to properly use my own language, right? Chrome eventually removed 'backspace' as the back navigation, alt+left arrowkey is the default now I believe but you can remap it to your hearts content.
We always appreciate your communication, Bobby. That's what always made GeoHIVE stand out to me. I hope the team jumps on board again.
I think the issue lies in how the GH team can tell who is merely clicking through and scamming, and who is giving an honest effort, despite their speed. IMO speed is a horrible reason to reject, some people are just quicker than others, we all have our strengths and weaknesses. A lot of us have been at this a long time, it could be someone with 10+ years of button clicking 12hrs of a day under their belt or my 90 yr old grandma or my cat sitting on the keyboard. You never know who is sitting on the other end of that connection and what their ability level is. As others have already pointed out, some sort of "gold" questions mixed in with the actual questions is probably one of the better options. If you could pair that with some automated feedback to workers, even better -- like hey you missed this here, be more careful, with a link/screenshot/something to which thing you screwed up. A tiered qual would allow you to knock someone down a notch and let them know to step it up before they get kicked off completely. I think we all appreciate a chance to learn exactly how we are messing up and fix it. Just reiterating my support for several of you that already mentioned this approach, now that the toddler is napping and I can sit down for a minute.
I understand the fear of speed workers. It would seem that some are doing the work too fast to be doing it well and some are. But there are those that can be that fast and do the job well. These people are an asset to your company. By blocking over speed you will throw out the good as well as the bad. A better measure for quality (as has already been mentioned) are golden maps. Put one or more (depending on the number of maps in the hit) maps in a hit that you already know the answer to. If they worker fails to many of these in a hit, reject the hit. Yes, there are times when the golden map will be the only hit messed up on in a hit, but this will likely be rare and the worker should be prepared for this. Just make sure the amount is reasonable. I would also randomly spot check maps and treat them as golden maps as well. I also like the variable qual idea. I would link the qual with performance on golden maps. Make the qual 1-100 and give all new workers a low qual and all existing workers a medium qual. The more golden maps they get right, the higher there qual will go and the more they get wrong the lower the qual will go. Then limit the number of hit they can complete in a day dependent on their qual number. At a certain number you could eliminate the restriction all-together. This way, there is no risk that an inexperienced worker or poor worker will do a lot of bad work for you.
Hello Bobby. Any possibility of simulated 0.10 hit being posted (like the "Select All Categories that Apply to a Satellite Image" style) so i can record myself doing the hit like you suggested? I'd put in some simulated (free) work so you can get an idea of how I do the hits. Frankly I have always liked the GeoHive hits, but if I need to slow down to do the hits, there is no way I can justify working them. As of late, I have had issues with some images not even loading, and that is clearly not the fault of my internet provider. Sometimes i can get around that by opening the image in another window and wait for it to load while i go through the other images, and then go back and classify the previously unloaded image. But that is extra work and time. There must be a more reliable way to serve those images to us. I had assumed there would be a low reject rate, due to how subjective some of the categories can be. I never thought that speed would be reject factor in all but the most extreme (cheaty) cases. I did about 150 of those hits on the 1st with zero rejects, and near the halfway point of my session was I getting pretty quick as got into a focused zone. So I wonder if there is a way to look at a history of good workers, and give feedback (automagically of course) if quality drops. Maybe even let us see our results after the fact to see how we did. Feedback is important to learning and improving. The way it is setup right now, everything gets sent to a dark box and we never learn if we are doing good work or not except if a hit gets rejected. That can't be good for anyone; either you guys (the business) or us (the mturk workers). I'm sitting on about 50 pending 0.15's, and now i'm worried. But I'm less worried since you decided to reach out to us. Thanks for doing so. -Cnaw
I thought this part was interesting: You Can Have It Good, Fast or Cheap – Pick Two "Everything from collecting the data to making it fit the company’s needs open it up to potential errors. Having data that’s 100% complete and 100% accurate is not only prohibitively expensive, but time consuming and barely nudging the ROI needle. With so much data coming in, decisions have to be made and quickly. That’s why data quality is very much a delicate balancing act – juggling and judging accuracy and completeness." https://blog.kissmetrics.com/data-quality/
