Implement robust evaluation pipeline in EvalAI about gsoc-ideas HOT 18 CLOSED

I didn't get what you meant by speed here but the idea is to parallelize the submission processing near the challenge end date so that more people can submit to the challenge.

This is what I meant itself. Thanks.

from gsoc-ideas.

Hi Rishabh,
This Neeraj from IIT Bhubaneswar, I will like to contribute in it please guide me on the way.

from gsoc-ideas.

Hi @RishabhJain2018 I wish to contribute to EvalAI in GSoC 2019. Can you help me how to get started?
Thanks

from gsoc-ideas.

@RishabhJain2018 @Ram81 I am interested in this idea. I would love to work for GSOC 2019 on this issue. I am familiar with Django and have a basic idea about Docker.

from gsoc-ideas.

@RishabhJain2018 @Ram81 @deshraj This is exciting and am interested in the idea! To get familiar with the requirements, can we go ahead and make PRs relevant to this? (As is recommended in the UI ideas for GSoC.)

from gsoc-ideas.

Hey @RishabhJain2018 @deshraj @Ram81 I'm very excited to work on this issue. I'm good in Python and worked on AWS and familiar with dockers. This is my first GSoc and your guidance would help me get started.

from gsoc-ideas.

@RishabhJain2018 @Ram81 @deshraj Sounds interesting! I would like to work on this project 👍

from gsoc-ideas.

Hi, @shiv6146 Looking forward to your GSoC proposal!

from gsoc-ideas.

@deshraj @RishabhJain2018 @Ram81 this idea looks amazing. I am looking forward to work on this issue

from gsoc-ideas.

Awesome! Looking forward to your GSOC Proposal @kurianbenoy :)

from gsoc-ideas.

Hi, @GrayR00t @mrkarna @navneel99 @KhalidRmb @anunay999 Thanks for your interest in the project. Looking forward to your GSoC Proposal.

from gsoc-ideas.

Can you help me how to get started?

@navneel99 Please start by setting up EvalAI on your local machine and then start solving good-first-issue or GSOC-2019 issues.

To get familiar with the requirements, can we go ahead and make PRs relevant to this?

@KhalidRmb Yes.

from gsoc-ideas.

Hi! Had some queries, and it would be very helpful if the mentors can help me navigate them.

• Regarding scaling the workers from 1 to x by the challenge host: I gather it is necessary for docker-based challenges where the host wants to test the submission against diverse environments with different requirements in each worker container.

  • In which case, the host must specify the configuration of each additional worker environment through the UI itself. (of course, the new worker(s) will be integrated with the pre-existing SQS queue for the challenge, and change the leaderboard data accordingly, reflecting the metrics returned by the evaluation in a pre-defined standard format.).

• Coming to non docker-based challenges, the main concern to scale the workers is speed and submission bottlenecks? Because the worker evaluates the submissions sequentially, running workers in parallel (with the same configurations) are faster from the host's perspective. Is that correct or did I miss something?

@RishabhJain2018 @deshraj @Ram81

from gsoc-ideas.

Regarding shifting to Fargate:

• Is only the submission worker to be shifted or the Django container along with it? Because based on incoming traffic to the Django app, Fargate will autoscale the task (which could be based on a single task definition of 2 containers- Django & the Worker).

• If scaling the worker alone, we could define a new task definition only for the worker and use deploy.sh scale through boto3.
  Could you please help clarify the situation here?
  @RishabhJain2018

from gsoc-ideas.

Regarding the task Provide naming for worker containers for different challenges, there already is a mechanism for that in the deploy.sh file here:

https://github.com/Cloud-CV/EvalAI/blob/82dfe27173893fcb3c8ed5b3e09dcd42d365b211/scripts/deployment/deploy.sh#L79

Could you please provide some clarity regarding the task?

from gsoc-ideas.

@RishabhJain2018 @deshraj @Ram81 Could you please take a look at these, and the doubts I've asked on Gitter? The proposal deadline is very near. Thanks!

from gsoc-ideas.

Hi @KhalidRmb,

I gather it is necessary for docker-based challenges where the host wants to test the submission against diverse environments with different requirements in each worker container.

What if a challenge host wants to parallelize the submission processing for the non-docker based challenges?

Coming to non docker-based challenges, the main concern to scale the workers is speed and submission bottlenecks? Because the worker evaluates the submissions sequentially, running workers in parallel (with the same configurations) are faster from the host's perspective. Is that correct or did I miss something?

I didn't get what you meant by speed here but the idea is to parallelize the submission processing near the challenge end date so that more people can submit to the challenge.

Is only the submission worker to be shifted or the Django container along with it?

For now, we're focussing on the worker container only.

If scaling the worker alone, we could define a new task definition only for the worker and use deploy.sh scale through boto3.
Could you please help clarify the situation here?

I'd like to see the complete approach in proposal. Also, I've answered your query.

Regarding the task Provide naming for worker containers for different challenges, there already is a mechanism for that in the deploy.sh file here:

Yes, it is already there. But docker doesn't allow running two containers with the same name on a single machine, so a fix regarding it will be needed in the deliverable.

from gsoc-ideas.

@RishabhJain2018 What level would you say this project is? How good should I be with the stack mentioned? What are the stuff I should really know before I start working on it? I know it's GSoC-related but I was just thinking that I'm gonna get it a try for fun and experience?

from gsoc-ideas.