zhubonan / aiida-fireworks-scheduler Goto Github PK

hey, just letting you know that the entry point naming does not follow AiiDA conventions:

The canonical entry point prefix would be fireworks_scheduler

In AiiDA one computer must have a single scheduler. Tools should be added to create duplicated Computer that uses the same SSH transport but with different scheduler, as well as the associated Code.

Implement specialised resources based on the SGE scheduler

• remove the limit in the parallel_env
• allow setting the priority - default to 100 + X
• recording the "user name" of the job and use it to fetch jobs. This way one may support having multiple accounts on the same machine, just as AiiDA does.

If the change is too much I should directly subclass Scheduler instead.

A long-term goal, probably beyond 1.0 is to implement a specialised Transport class to move away from SSH. This will make it easier to work with fenced clusters. However, it will be significant efforts as:

• no python package has yet to provide complete file system abstraction on mongodb
• might have to tweak the aiida-core code
• need to implement specialised Firetask to pull and push the data to the repository before and after each launch

Should add a timeout for graceful termination if no jobs have been found after N loops in both the finite and infinite launch mode.

There seems to be a problem. Should give more verbose report about why the kill was not a success....

Documentation should include how to:

• setup locally
• setup on the remote machine
• example of the configuration files
• readthedocs site
• further developments

Current installation instructions suggest using pip to install, but it doesn't seem to be on pypi currently.

There are a few states that a job can be within fireworks they should all be mapped to JobState correctly.

At the moment it is possible that a job changes the states during the handling, as LaunchPad can only get the jobs one by one. This is a two-step process at the moment. So even if the query selects only the running or queued jobs the returned jobs may end up in completed states.

Allow sometime it can be useful to allow jobs exceeded the defined limit. This is a uncontrollable parameters and will affect the provenance.

Walltime limits should be enforced in the script submitted by the AiiDAJobFirework. This also makes sure the actual firwork launch does no exceeded the scheduler walltime limit and being left in the RUNNING states, which requires manual intervention for fixing.

This should be as simple as getting the fw_id for the calculation and query the mongodb using the lpad interface or using pymongo directly.

The SgeScheduler inheritance was there only for convenience. The FwScheduler should directly inherit from Scheduler. Unnecessary methods for getting commands can be left blank perhaps as we directly overload the get_jobs and submit_from_script and kill methods.

Should reuse LaunchPad instances to avoid creating too many objects and connections.