Files referenced on command line that do not exist are ignored. This should raise an error

Work in this bash auto complete script so that users can use command-line completion with Longbow

_longbow()
{
    local cur prev opts
    COMPREPLY=()
    cur="${COMP_WORDS[COMP_CWORD]}"
    prev="${COMP_WORDS[COMP_CWORD-1]}"
    opts="--about --debug --disconnect --examples --help --hosts --job --jobname --log --recover --resource --replicates --verbose --version"

    if [[ ${cur} == -* ]] ; then
        COMPREPLY=( $(compgen -W "${opts}" -- ${cur}) )
        return 0
    fi
}
complete -F _longbow longbow

Take a look at alternatives to landscape.io, it seems that the service is broken and will be unavailable for quite some time still (already 5 months at date of this post). There does not seem to be much information other than "our servers are over capacity", "therefore we stopped checks" and "we are working on it" so nothing about the future, no dates etc.

Codeclimate seems to do what this project needs, however it seems to pick up numerous more problems than landscape did, some fair but others more annoying and unavoidable (duplicates, especially because we have a per-plugin code structure). Codeclimate appears to also do coverage, so could possible drop coverage.io too and just use cc + travis.

Add in a single job and multijob examples of launching from a script.

The fix issued in issue #5 works but there is a case when a cluster has gpus, by removing the pe flag altogether for jobs running one cpu works for jobs run on serial queues, however if submitting jobs to a gpu configured in a certain way then jobs won't run without the -pe mpi 1. This is a special case, presumably due to gpus being tied to non-serial queues, an override flag should be included.

At the moment there are large blocks of imports to import functions from the corelibrary. This should be cleaned up so that one simply does a single import of longbow and then call the method as longbow.method()

This should be tested to make sure there are no side effects for things such as logging etc before pulling into development branch

Finish developers documentation

To be able to support more applications and do better checking of the command-line for applications, it is crucial to be able to differentiate if a argument should be a file or a parameter.

An example is the new gromacs, they now have all utilities within a gmx wrapper, so mdrun would be invoked by:

gmx mdrun blah blah blah

We can't currently handle these.

All unit tests have passed. Some real world tests will now need to be done to verify features are still working.

The python version check simply issues a warning if the python version is out of scope, this should issue a warning and then fail.

Reported by Hannes Loeffler

There is no exception handling during recovery mode to capture things such as the user interrupt. The following options could be done:

1. The same code that handles this behaviour in the longbowmain method could be implemented in the recovery method. This could allow a deviation in intended behaviour of the interrupt signal between normal operating mode and recovery mode.

2. The code from longbowmain method could simply be moved up a level to the main method and then cover both longowmain and recovery methods. Although there will need to be a way of passing the jobs data structure up the chain as it is not currently in the namespace. This could be done either by declaring an empty dict in the namespace and passing it down to be initialised, or find a way to append it to the exception and pull it out where needed.

This could be as simple as adding python 3.6 to the unit testing tools. However if there are problems then they should be listed here and fixed.

Parameters that are only available within one plug-in whether it is an application or scheduler should begin with the name of the app or scheduler such as "cluster" => "lsf-cluster" so users unfamiliar with schedulers will understand which parameters are generic and which aren't.

Supporting custom installs is pointless given the power of tools such as pip and easyinstall/setuptools etc. The imports needed to support custom installs simply creates more problems than it solves and looks ugly and unpythonic. It is also a pain to write tests that can cover this so leads to an annoyance with coverage.

This should be withdrawn for future versions.

Some users have requested that in addition to the simple examples that appear in the documentation, that we show some examples of how to use some of the more advanced capabilities. This could either be spun out into a separate document or pull out examples into its own section of the documentation.

Adding validation to inputs from all sources and raising errors should they be out of range/wrong format etc will provide an extra layer of protection and make use of Longbow more intuitive.

Investigate whether or not it makes sense to have global files in multi-jobs like in the replica jobs (does this make sense here)

So Longbow is moving away from having the capital L in its code base and directory structure. Longbow version 1.5.0 should change the existing user directory from .Longbow to .longbow automatically and inform the user this has taken place. For first time installers this should simply be skipped.

It was suggested at a recent talk about Longbow that the recovery method could be extended in a way that friendly names could be used for the sessions to recover rather than the dated filenames.

They could perhaps also still use the filenames (because they prevent data loss in conflicting names) but single/parts of multijobs could be extracted from sessions.

In older versions of Longbow there used to be a MODULEOVERRIDE parameter that could be optionally provided in application plugin files to alter the default naming of modules from the name of the plugin file to something else. This could be used by users to set the module name if something peculiar had been used.

This should be restated along with tests, as along the line this functionality has disappeared.

This will need adding to dev docs as well.

There is code and unit test duplication for the longbow() and recovery() methods. The try except blocks in the longbow and recovery methods that handle the user interrupt are identical. This code should be moved up to the launcher() scope where it will feature just once.

Doing this will require refactoring the code slightly so that the jobs structure is in the scope of launcher, it will need passing in to longbow() as part of parameters.

Longbow should allow the launching of arbitrary software, however this should be done by the user having to provide a flag such as --nochecks. This will indicate to Longbow that the user intends to do such things and will then be able to distinguish between mistakes and intentional behaviour.

Longbow should still perform some checks, but the application specific checking should be reduced.

Get rid of the blocking sleep function and separate the staging and polling into two timers so they can be changed independently.

The sleep timer sometimes doesn't behave the way a user would expect.

This branch is no longer required for testing, this was created back in the days when manual testing was done and functionality had to be checked before merging to master. Now CI tools do this at commit and PR time, so this is no longer needed.

The -pe flag in the SGE plugin should not always be called ib, this parameter is under the control of the system administrator and should have a parameter to modify this. This parameter should disappear if number of cores are 1 and possibly mpi is a sensible default for this to start.

The processjobs method will raise an exception if a hosts file is not supplied, this check is already performed in the Longbow application layer so should not be done again. This is prohibitive to developers using the library as they might well wish to simply pass a dictionary of data to be Longbowized.

The zip archive gets downloaded and then extracted properly, however the archive is then not cleaned up.

The -deffnm parameter seems to be picking up or at least pointing to a global file location when used with a global input file -s

The following traceback is generated when trying to launch own compiled gromacs

2017-08-04 13:58:40 - ERROR - longbow - '/work/c01/c01/jtg2/gmx/bin/mdrun_mpi' Traceback (most recent call last): File "/home/jimboid/.local/lib/python2.7/site-packages/longbow/corelibs/entrypoints.py", line 224, in launcher longbow(parameters) File "/home/jimboid/.local/lib/python2.7/site-packages/longbow/corelibs/entrypoints.py", line 266, in longbow jobs = configuration.processconfigs(parameters) File "/home/jimboid/.local/lib/python2.7/site-packages/longbow/corelibs/configuration.py", line 175, in processconfigs _processconfigsfinalinit(jobs) File "/home/jimboid/.local/lib/python2.7/site-packages/longbow/corelibs/configuration.py", line 444, in _processconfigsfinalinit jobs[job]["modules"] = modules[jobs[job]["executable"]] KeyError: '/work/c01/c01/jtg2/gmx/bin/mdrun_mpi'

For the application plugins, document the API hooks that plugin developers can make use of.

Looks like a simple missing white space after the executable.

Some HPC machines seem to create a problem where the environment is not loaded under a non-login shell. This includes environment variables for the scheduler and even the Linux module system. A simple cure is that the user can add "source /etc/profile" into their ~/.bashrc file. Machines where this doesn't happen seem to have a .bashrc setup to reference the various environment files by default.

Simply asking for a login shell does not seem to work either, this appears to be specific to shells requested through a script. I'm not sure if there is some sort of safeguards preventing certain sessions from having a login shell (perhaps the tty issue).

There are several solutions to this:

1. Pass information to the user through documentation, informing them to how to add the relevant line to their .bashrc
2. Append the line "source /etc/profile" to every line that gets passed through to subprocess (not sure if this is bad)
3. Find out if there is a proper way to get a valid login shell when using python subprocess.

It is unclear at this stage which solution is the best one, more work and investigation is required so that we can support as many machines as possible.

If all jobs fail there is a glitch where Longbow hangs.

This bug has been fixed in the PBS plugin but not in any of the others. The fix for this should be the same as in PBS, extend this to all other plugins.

The error message displayed is SSH failure, this is not correct.

Add email flag so that users can specify that complete jobs send emails.

Once Longbow v1.5.0 has been released along with its examples, the unit test for the downloadexamples method will need to have the remainder of the capitals removed. Currently they cannot be taken out due to the tests performing an actual download.

When using the disconnect mode with --disconnect, currently the only way to reconnect is via the --recover option, but this reconnects permanently without then disconnecting again. A new mode should be implemented to reconnect to update the status of the jobs, transfer files and then disconnect again. This should be able to be done as many times as the user wishes up until completion of all jobs.

The current testing suite has lots of functional and partial unit test support. A new set of unit tests should be made available where functional tests are also included but using mock where possible to simulation external connections and files etc.

Remove the sentence "Longbow has been developed with those that use a local batch system in their lab such as Condor in mind. However Longbow can just as easily be ran directly from a desktop machine terminal."

Dis-connectable sessions have made this obsolete, as rightly pointed out by @gshannon1986

Place non private methods into the longbow package init.py. This means that what would have been something like this:

import longbow.corelibs.applications

applications.checkapp()

now becomes

import longbow

longbow.checkapp()

Makes the code more readable in complex software with many imports.

Check user docs to make sure it reflects the more recent code changes.

A point to do with replicates has been pointed out as unclear, this is to do with the creation of the repx directories when global files are used.

There is still some capitalisation getting through in the examples download function.

A number of users have reported this as an annoyance when using import paths.

Look at a more permissive license than the GPLv2 and release the next release and each release after under this.

GPLv2 is not friendly for inclusion into other software under some circumstances, at the very least LGPL should be used.

This is totally unnecessary behaviour, this will be due to the introduction of the new timing code, where as the old code used to do lots of checks that would prevent full utilisation of the core, now the new code allows the checks to be done only at specific time intervals. The effect of this is that the core running longbow is thrashed, whilst the only thing happening is spinning the counter and waiting to do checks.

Perhaps a solution to this is to work out some sleep intervals that will take account of both the status checking time interval and the file staging time interval.

Recovery files tend to build up rapidly over time, currently the user has to periodically clean these out by themselves. A better solution would be either of:

1. Only keep a small number (say the last week)
2. Provide a command-line flag such as --recovery-clean and just wipe out anything in the directory with it.
3. Rearrange how the file is handled within the code. Use the global jobfile parameter simply for initial creation. Then store a handle to the file under each job (no other place to put it). This way can make use of the filename throughout Longbow, no need to create new recovery file on recovery or reconnections and can add code to the cleanup function to kill the recovery file once all jobs are done.

Add --nochecks to the bash autocomplete script.

Gromacs can now be called using the form:

$ gmx mdrun blah blah blah

ARCHER now has this available through the module system so we should support it.

When running several instances of Longbow, where one of them maxes out the queue and the other ends up queuing jobs, never gets to submit its jobs. This also happens to varying degrees when the queue is split over multiple versions, the instance performing queuing never grows its share of the machine.