If anyone is having any issues if you could message us with your worker id then we will look into it and see what is going on! @Wilma @lefty
I have a question about some of the validation HITs you have done in the past as we start to prepare our own for mTurk. Do you typically see prior answers and you just say "OK" or enter new value? OR do requesters just enter in again from scratch w/o seeing prior answers and system compares the two. Any insight you could provide would be very much appreciated.
The 2nd scenario is referred to as "majority rules" - you just setup your task to be repeated by X number of workers and take the majority answer as the "correct" value. The specifics of your project determine which way is best, so we'd need more information on what you're doing, what your budget looks like, and what results you're expecting to give you good advice on the subject. Majority tends to be more expensive, but can be accurate with less oversight. Having a "cream of the crop" work force that you can trust to validate the answers of lesser known quantity workers takes more time to set up / recruit for, but you can cut the # of total assignments down (meaning you save money at the expense of time). Well, in general anyway. Specifics of your projects can rule some stuff out. If you're training a wide reaching AI assistant, you can't really limit your worker pool to a few "good" workers since you need varied data. If all you need is to know, without question, the answer is correct then finding 1-2 good workers to plug through everything (similar to just hiring someone off platform) is fine. Etc etc etc down the list of various scenarios
Thanks for the prompt reply and information. It sounds like both ways are used depending on the project. I"m trying to find some examples on mTurk but not finding any at the moment.
Lots of different methods can definitely be successful. As far as finding examples you can look over I'm not sure that's going to be readily available. Most of the validation is done on the back end, which isn't something most companies are going to hand out access to. There are some AWS talks about how A9 does it if you want to give that a look:
