crowdresearch / daemo-api-client Goto Github PK

I'm thinking about ways to enable traditional data capture workflows. Gold standards are a super common option. What if we created another utility rate_gold function? It takes in a dict input->correct answer as an argument, and then the utility function looks to see if any inputs from the tasks match the input, and if so check if the response matches a gold. It then takes sum(correct) and sends that to the rate() function.

• Specify where to access the Daemo Platform
• Can probably use 2nd person for the platform/interface instructions
• Put in the image of the hill instead: hill sign
• Do I specify price? No mention of price on the tutorial, though the field is required
• Image missing: image:: images/project-options.png
• Comma missing between the tasks in list task_data:

task_data = [
    {
        "image_url": "value"
    }
    {
        "image_url": "value"
    }
]

My rerun is taking about .5s-1s per previously-completed task. What's the engineering reason this is taking so long? I'd imagine these could be done super quickly, since we cached all the old work at the beginning.

10:34:47 daemo.client: adding tasks...
10:34:49 daemo.client: adding previous worker submissions...

I had a bit of a heart attack thinking it was reposting all my tasks again.

there is an initialization problem with peer review when only one response is streamed (either when stream=True or when there only one worker doing only one task)

This would help me do some posthoc data analysis.

Right now it hangs and you have to Ctrl-C. Typically with Python the uncaught exception actually exits the program.