Seems this was a learning experience for all. I would second the recommendation to use a Qual, either with trusted workers from this batch or with a Qual Test.
Yes this was/is a learning experience. The bad thing is that this is not the only thing I have to do and I spent a ton of time checking through the first batch. Some of you have already contacted me as you noticed that I released a new batch but added a qualification. This was the result of the analysis I did with the first batch. I have reached out to only two people so far based on their results. I have some more I need to reach out to as there are about 4 others that did a super job. Others however I need to create some guided practice for before I can let others (those not in the initial group of six) have the qualification. I just haven't had the time. I did reach out to AWS today to ask for some help on creating a qualification test and my AE said he would have a technical support person reach out to me tomorrow. I hope that will go fast because I did spend three hours today doing research and it was not time I wanted to spend. Enough about that - I will observe that this first experience was great but I am overwhelmed right at this moment trying to keep up and get things better organized. I definitely want to add more of you to this and I think there is more than enough work between this and the next step to go around quite a bit. It is a little complicated as when I was trying to decide how much to pay folks I don't want to be the cheapest miser on the block. I am paying too much to have 3 people do each HIT. I think my pay is fair but it is only going to work if I manage this correctly so I can have good validation processes and have probably less than 2% that need to be checked and or corrected. I had to check about 500 (~25%) hits and about 150 (~7.5%) had errors. More than 125 of the errors were tough to sort out as anything other than maybe not enough focus <- trying to tread lightly here. The rest of the errors were my fault as not having a good enough description of the issue. I am just darn glad I did not do what I almost did and dropped 5,000 hits initially. I appreciate your patience
Some of what I am having to contend with is my fault. So for instance, I am going to make a small change for the future ones that do not have the MD&A in the 10-K but have a reference to where the MD&A is located in the Annual Report (A/R). Before I was asking you to give me the line numbers where the reference was located. That did not work out so well after looking at the results. I actually need more information so at a glance someone can identify where this will be located in the annual report. Last time around the line numbers corresponding to this would have been acceptable The problem with this is when I was verifying it I could not always be certain that particular reference was to the location in the A/R for the MD&A or if that was a bad TOC entry. So when I saw this I had to verify that it was a reference to the A/R (there were some where the person (I hate the word Worker) did give me the TOC entry. To check this I then had to go open the original file so there were quite a few steps involved. Further, the next step of this project is to pull from the A/R the MDA - I need to be able to provide good guidance for the next step. So from now on I am going to ask that you provide enough line numbers so that a person reading it can clearly tell the place to find the information. So in the next round I would like line 359 and line 381. The reason I would want those lines is you can probably see that a reader should be able to clearly understand that they need to look on pages 15-33 of the A/R. I will also note that I want line 381 because it makes it so much easier to check these when I can have to computer pull the first and last line numbers I get from you and dump them into a spreadsheet and then scan them.
