Mechanical Turk Integration about cvat HOT 8 OPEN

I hope to have some news in Q2-Q3 on the topic...

from cvat.

Hi,

It is a feature in our mind which we don't have resources to implement till EOY for sure. If you can help it will be awesome. Development environment is described here: #18.

Several notes:

• Need to implement a separate Django application for the feature
• It isn't good to use turkic from VATIC because it isn't supported anymore. I would suggest using an official API (boto3?) - http://boto3.readthedocs.io/en/latest/reference/services/mturk.html
• Need to think how split a task on many HITs (by frame/image?) because HITs should be trivial. Also need to automatically create "verification" tasks to check results. In many case it will be necessary to merge annotation from several HITs.
• GUI to publish, view progress, ... Need to have sketch of it.
• The application should have REST API.

In general it is a big feature which cannot be easily implemented.

from cvat.

General requirements:

• Easy interface to create/review/accept an annotation task for MTurk (separate Django app).
• Probably need to have a separate instance on AWS for such tasks.
• Automatic authorization from MTurk, special group to access only links from MTurk django app
• Email notifications about deadlines to verify annotation tasks, etc...

Requirements (image annotation):

• One image per a segment. Thus it is easy to post such job as HIT.
• Several jobs for the same segment.
• Automatically correlate annotations in different jobs for the same segment. Automatically find "trash" in annotations and discard such HIT.
• Minimum number of jobs for a segment is 3.
• Maximum number of jobs for a segment is 7. After that need to verify or discard HITs due to low quality.
• One user cannot do annotations for the same segment (several similar jobs). Need to improve auth/permissions to access only specific jobs.

from cvat.

Hi, Wanted to know if any progress has been made in this direction :)

from cvat.

@nik13 , we are not working in the direction at the moment.

from cvat.

Nice overview of using a crowd sourcing platform (Toloka): https://www.youtube.com/watch?v=yyybisXHFc0

from cvat.

Has there been any progress on the integration with crowdsourcing platforms?

from cvat.

Very interested in this too!

from cvat.