Hi y'all! I am a university researcher who is pretty new to mTurk, and I am trying to improve my HITs as much as possible. I hear back from workers that I give too many attention checks, but I need to run these checks so that I can interpret my data later. I am hoping to make the attention checks less annoying, and was wondering what you guys thought would be better (poll above):
I think it sort of depends on the nature of the study and the type of attention checks you're using. Are they something like "Choose Option 3" or are they asking a specific question about a text?
They're asking a specific question about the task instructions. This is the current text of one attention check for an experiment where we ask people to try to move the mouse to a pink target when there is a misleading "fake" cursor on the screen: "Instructions and Attention Check The green dot is not under your control. It will still move at the same time as you, but it could go somewhere else. Do your best to IGNORE the green dot. Always aim for the pink target. Press x if you should ignore the green dot and aim straight for the pink target. Press y if you should aim away from the pink target to try to get the green dot to hit the target. [option to press x] [option to press y]" Right now, the instructions appear at the beginning of the HIT and during the attention check. I could take the instructions out of the attention check so that there's less reading to do during the HIT itself, and rephrase to something like: "Press x if you should move your mouse to the pink target and ignore the green dot. Press y if you should aim to hit the pink target with the green dot. [option to press x] [option to press y]"
As a worker I expect to get attention checks but they should be occasional not multiple checks per page. I have done surveys where I am certain there where more ACs than actual questions. Also the type can be an issue they should be of the nature that they will not be missed if you were at all paying attention. They should not be tricky or ambiguous so that workers are looking at them trying to guess what you were thinking when you wrote it and afraid to proceed. And my new pet peeve DO NOT put AC's in consent forms most workers scan through them but do not read every word after all they are largely all the same with slight differences if there is something different about that research project (thus the scanning). Lately putting some secret phrase in the middle of a consent form and asking about it later seems to be a popular thing and is very frustrating to workers. Beyond that what are you currently using. Do you feel you need something beyond the standard if your paying attention select option 3t type AC. Or the research shows decisions aren't made.... so select ...
just noticed the example above. It is wordy but I agree with @Slowdive I would leave it in. Is this AC occurring during the activity if so workers might find it less annoying and less distracting if you gave instructions then a brief quiz on the instructions this is fairly common when making sure participants understand the instructions is important.
Ahh... I think that this is a possible attention check format because it's really easy to program/write in. It's really good to learn that this is frustrating and something to avoid. Now that I think about it, these hidden-phrase types of attention checks are not that useful anyway. They ask the worker has to remember some random phrase instead of the actual instruction or whatever thing the requester wanted to make sure is understood. Thanks for your feedback! I will try to cut down the words/make it clearer. This attention check occurs during the activity at a point where the instructions change a little bit. There are new instructions every minute or so for the first 4 minutes, and then things are consistent for the 5 or 6 minutes after that, so there are 4 attention checks in total but there are a bunch in a few minutes. There is a common goal throughout the whole HIT, though, so maybe I can cut down to 2 instructions quizzes -- one at the very beginning and one at the end.
