CI for Schedulers about dask-jobqueue HOT 23 CLOSED

OK, you have admin rights. I guess the difference here is the owner/member of the org distinction? Further reason to push on dask/dask#3223

from dask-jobqueue.

I propose to close this issue by opening more specific ones, e.g. one for PBS and one for SLURM. We already have basic CI with unit tests and for SGE.
Are you OK with this?

from dask-jobqueue.

This repo has an example using SLURM: https://github.com/Snakemake-Profiles/slurm

from dask-jobqueue.

I agree that this is both valuable and somewhat time consuming to set up. If anyone has the time to contribute a CI setup for one of the job schedulers that would be very welcome.

from dask-jobqueue.

I can take a look for SLURM (can you assign this issue to me?)

from dask-jobqueue.

To be honest I rarely use github assignment. It looks like this is difficult to do without having you as part of the organization (which is currently somewhat challenging). We might have to go with informal assignment for the time being.

Regardless, thank you for stepping up!

from dask-jobqueue.

Grid Engine should basically be just copying over what dask-drmaa already has. Maybe that is the best place to start and then we can extend from there. Is Travis CI enabled here?

from dask-jobqueue.

Nope. Nothing yet.

from dask-jobqueue.

Would you be willing to do that? I could copy over the testing code. Just don't think I have permissions to add CI is all.

from dask-jobqueue.

Do you know how I can give you that permission?

from dask-jobqueue.

Think I need to have admin on the repo as it requires the ability to add webhooks. Not sure if there is anything more granular. Honestly don't know if I'd need admin for anything else. Leave it up to you whether to grant it or not.

from dask-jobqueue.

@jhamman @jedwards4b you two also now have rights on this repository

from dask-jobqueue.

Thanks.

Agree that we need to figure out org rights. Will try to make next meeting so we can discuss more.

Enabled Travis CI, but told it to skip until there is a .travis.yml file.

from dask-jobqueue.

Grid Engine should basically be just copying over what dask-drmaa already has. Maybe that is the best place to start and then we can extend from there. Is Travis CI enabled here?

You know, the funny thing is that if we got a testing system for GE then that might be enough motivation to implement an SGECluster here just to get the test coverage. It's rare that one builds source to capture testing, but in this case the ratio of difficulty in writing testing code and source code is so great that it makes sense :)

Testament to @jedwards4b 's refactor of JobQueueCluster I guess :)

from dask-jobqueue.

I have a fair bit of SGE experience, and a cluster that I could test this on apart from travis. I'd be happy to contribute the SGECluster. I'll open a separate issue. (see #3).

from dask-jobqueue.

from dask-jobqueue.

from dask-jobqueue.

Can non member of the org see what's happening on travis-ci for any dask repository?
Shall we fork the repository if we want to test/check the CI build?

from dask-jobqueue.

from dask-jobqueue.

How do you feel about using docker-hub provided images, such as https://hub.docker.com/r/pbspro/pbspro? Do we want to control the entire build cycle or not?
To go more into detail, this one depends on a build image with no Dockerfile provided (https://hub.docker.com/r/pbspro/pbsbase/), how do you feel about that too?

And lastly, I believe we want a minimaly representative test cluster, so with one master and two slaves such as what is done for dask/dask-drmaa, and not a one node master/slave cluster?

from dask-jobqueue.

from dask-jobqueue.

Ran across this dockerized slurm cluster today: https://github.com/giovtorres/slurm-docker-cluster

from dask-jobqueue.

I think this issue can be closed with ##23, #27, #41 and #42.

from dask-jobqueue.