Hummingbird Watch / Introduction

Hi All,

My name's Robert and I'm with a company called Hummingbird Watch. We have a new project starting here shortly and I wanted to introduce myself and our idea. You might be somewhat familiar with some of our HITs already.

Sporadically over the last year we have been uploading HITs to MTurk, asking you to identify whether or not there is smoke in an image. Usually, there was. But that's all about to change.

This summer we will be conducting live fire detection in the Canadian Province of Alberta and the Yukon Territory! It's an exciting trial, and of course, it's not going to be possible without all of you. We want to make sure that we get off on the right foot, and through the month of June, we will be running system optimization trials before our system goes live in July.

Your advice, suggestions and feedback is of high value to us. Especially between now and the beginning of July.

A few immediate and noticeable changes that will be present throughout the summer include:

1. Right now, most of the time, about 90% of our HITs contain smoke. This is going to be reduced significantly. Wildfires are rare, but early detection is important.
2. To accomplish our HITs, you will now be re-directed to an external site and, when smoke is detected, asked to draw a box around it. Smoke will be rare, so you won't be overburdened by drawing boxes, and when you do discover smoke, you will only have to draw one box.
3. The frequency of our HITs is going to increase dramatically. At the present, we post HITs every couple of weeks. Ramping up in June and moving into July, we will have HITs uploaded consistently every 5 minutes between twelve and eighteen hours a day.
4. Unfortunately, this also means we are going to have to pay less. We don't want to, but unfortunately, it's a financial reality that accompanies posting so many HITs. Our HITs right now pay 10 cents to assess 3 images. For the trial this summer, we will likely pay around 1 cent per image. HITs will generally consist of between 2 - 3 images.
5. Accuracy and speed are very important to us. It's important to detect smoke quickly and we are considering placing a time-limit on our HITs of 5 minutes (especially because once the newest set of images are uploaded, the old ones will be obsolete). We aren't sure if this is a good idea yet though. This is a place where your feedback would be really helpful.

Most of the time, there won't be smoke in an image, but we don't want people to become complacent. We also don't want to make people nervous about working on our HITs. We definitely won't be upset if somebody thinks there is smoke in an image and there isn't. What counts though, is accuracy when there is smoke in an image. We're not sure how to manage this.

Also, bonuses. We think that when someone positively IDs a smoke, we would like to reward that. We think maybe 10 cents would be appropriate?

I think we are going to concentrate our forum presence here at Turker Hub (we really like Chris Turk) so if you know anyone that might want to contribute, please direct them here!

If you have any questions or concerns regarding what we are doing, please don't hesitate to reach out.

I'm sure it will be a pleasure to get to know you,

Robert

 

I haven't done any of your hits yet but I'll be on the lookout for them and give them a try. Then I'll be able to hopefully give you good feedback.

I realize that you are on a budget and increasing the number of hits you post considerably means you have to decrease pay. With this in mind I just want to mention that there are some people that won't open another webpage (redirect to an external site) for only a few pennies. We have load times to factor in when figuring out if a hit is worth the pay. I'm sure there will be plenty of people that will be willing to do your hits though.

 

My assumption is he's talking about what mTurk refers to an external HIT, which is still in the iframe its just hosted on their servers instead of the normal mturkcontent/aws servers. I hope so at least, because if not what you said it absolutely correct.

 

@Celtic Lady Thanks a lot. It's definitely a concern of ours. Our plan is to assess this method in June (we need reliable turnaround consistently under five minutes) and if it's taking too long, or our participation levels are too low, then we will re-evaluate and keeps things within the Amazon framework.

 

@ChrisTurk thanks man - we're still developing this portion, so I'll make sure this is how things end up

 

Let's just hope there isn't another Fort McMurray this year.

 

@leafs4_cup I'll say - early detection plays (one of many) an important role in preventing these disasters.

 

@gurlondrums We'll definitely be posting regularly, monitoring the site and providing updates as we reach milestones, we're pumped to be here!

 

Hi @Robert it is great to see you made the trip over to see us at long last! It has been really exciting talking with you this past few months about the project and the upcoming plans. It looks like you have a pretty good plan coming together and I know you have received some of my feedback in email earlier today. As I expressed early this afternoon and others here have expressed as well, load times are one thing that can cause a project to sort of "fall apart" and cause trouble. I highly recommend that you have the very best servers with the hugest amount of memory etc possible for your external server so that we don't DoS you ....LOL.... We love to work on great hits for great requestors and having work like this can be rewarding in and of itself.

I'll talk with you again soon!

@rater9

 

From one requester, and one Robert to another, cool concept!

 

Yea, thanks An... I mean @rater9 . It's been equally exciting and informative getting to know you. Your advice this far has been invaluable. Making sure we are working within the MTurk IFrame was the first order of business during our scrum meeting this morning. Load times definitely make us nervous, but we'll have parts of June to make any required adjustments if we are running into bumpy loading times.

 

@Bobby - thanks man! Back at ya.

 

LOL ... sometimes being a wee tad bit nervous is a good thing! It keeps us all on our toes .... Have a great evening!