This project demonstrates how to write external graders for CrowdAI.

• Python >= 3.6.5
• Flask >= 1.0.2 (PyPI)
• requests >= 2.19.1 (PyPI)

cp config.example.py config.py
cp grader_list.example.py grader_list.py
vim config.py # set your own s3 credentials
vim grader_list.py # set your own grader api url
flask run

In CrowdAI config/application.yml, set GRADER to the address and port that Flask is listening to.

Edit your chanllenge in CrowdAI, set Grader Identifier to the grader that you want to use (random_grader for default).

To write your own grader, just create a file in graders which inherits CommonGrader class and overwrite the grade method to update the scores, and then register it in grader_list.py. Note that organizer token should be used in grader, according to CrowdAI documentation.

One example named RandomPointsGrader is provided as reference.