We should implement a suite of <map_function>_or_recover() functions that either run or recover a map with a given map_id. Will prevent users from needing to wrap all of their persistent map calls in try/except blocks themselves.

We should wrap the FileNotFound in a comprehensible HTMap exception

It should be possible to inspect and modify jobs from a MapResult just like you can via condor_<x> commands. Common operations (condor_rm <clusterid>, condor_q -held -af <stuff>, condor_qedit etc.) should have convenient shortcuts.

Broken by work in #19

There should be a small set of common tasks available via CLI:

• remove a map
• rename a map
• display the status of all maps
• display the status of a map

This will help non-notebook users work with pre-built scripts instead of purely in a REPL.

Introspect the package list and compare to what is currently cached. I think we only need to store one Python install at a time - most people doing scientific work probably do what I do and throw everything in one install.

So that users can write behavior in their mapped functions that switches depending on whether they're running locally or not.

I think this just needs to be enforced at the recovery level. Maps would register themselves in a dictionary when they are submitted or recovered, and clear themselves from the dictionary when removed.

We should print a message containing information about the Python executable to stdout on the execute node.

Unfortunately, it looks like sys.executable can be an empty string under certain conditions, which will make this very tricky.

Via the bindings, of course - looks like htcondor.read_events is the right function.

To extract the inputs without using using iter_**with**_inputs

It fails with AttributeError: module ‘os’ has no attribute ‘uname’

We need to figure out how to get Python to the execute node. Two things to keep in mind:

1. The run.py script that we run on the execute node has one implicit dependency beyond whatever the user's function needs: cloudpickle.
2. Some people don't need this at all - they have full control over their cluster, and can install Python on all of their execute nodes. So the system should probably be "pluggable" to some degree.

See https://docs.google.com/document/d/1rB711Qe7xg6gDH6smavRwjjV6c0J8DWgMVUoZjAGtJ0/edit?usp=sharing

"Innards" page should be moved under there, as well as a new section on how to set up a development environment and how to run the test suite.

@johnkn thinks it won't work on Travis-CI because Condor needs access to directories that won't exist on their build servers.

It should be possible to transfer a map to another user.

Some connection to #47 , which will allow an imported map to be re-run by replacing certain submit variables in a very predictable way.

Filtering is easily implemented as a list comprehension on top of a normal map.

:, /, etc.

One option, which will be available once "map options" are available: mark each job with a special custom flag in its classad, then look for that with a condor_rm at the end of the session.

We should output log messages when stuff happens, using the standard Python logging module.

The only problem is that I'd like to be able to install on Windows for testing, but you can't pip install htcondor on Windows.

See https://stackoverflow.com/questions/6469508/is-it-possible-to-express-a-platform-specific-dependency-in-setup-py-without-bui

We should be able to rename a completed map.

Various resources:

https://ipywidgets.readthedocs.io/en/stable/index.html (the standard)

http://panel.pyviz.org/ (new project that might make things easier?)

This project should probably have a license

Occasionally we try to unpickle an object from a file that HTCondor is still trying to transfer. This tends to throw weird pickling errors.

We should either handle the error gracefully, or use the event tracker to avoid trying to load partially-transferred files.

It should be possible to look at disk/RAM use of map jobs via MapResults.

In particular, paths to the HTMap library itself and to files in the map directory should be interpolated by another submit variable, where possible.

There's significant overhead in running very short jobs, so there should be a way to batch up multiple inputs into a single job. The user just sets the number of inputs per batch, and we handle grouping them up and un-grouping the output.

This should just involve adding one more layer of iterables on the execute and output points.

Document all current settings

It doesn't produce the correct tarball, so it should produce an error early to stop people from doing it.

The empty "tutorial" page beckons...

Should be able to set priority of maps

Add a recipe for grouping inputs based on outputs.

Map just has to be not running, not be completed.

We should implement some kind of DAG workflow support, likely piggy-backing off of HTCondor's DAG support.

One possible API to model this off of: https://keras.io/models/model/

This will fail occasionally with something like PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'C:\\Users\\joshk\\.htmap\\maps\\foo\\cluster_logs\\253.log':

import htmap
htmap.force_clean()

@htmap.htmap
def double(x):
    return 2 * x

result = double.map('foo', range(10))
result.remove()

This:

import htmap

@htmap.htmap
def zero(x):
    return 1 / x

results = zero.map('zero', range(1), force_overwrite = True)
list(results)

produces an output file despite getting a ZeroDivisionError during execute.

Expected behavior: no output file.

Currently, they can be strings or pathlib.Paths, but both are interpreted as local file paths.

Current plan:

MapOptions(
    request_memory = '10MB',
    custom_options = {'WantFlocking': 'true'}
)

We will coerce all of the keys in custom_options to have a leading +, so that they can be entered with

• a leading +
• a leading MY. (case-insensitive)
• no + or MY. (preferred!)

Queries against HTCondor daemons should be cached for a brief period of time to prevent spamming them.

Use case: support things like SQUID on the CHTC pool.

Also, maybe provide an option to not do the zip at all, but just use a Python install available at that path.

The install needs to live as long as the map does, and it needs to be kept stable so that HTCondor can always find it.

The "Completed" job status prevents a map from being renamed - need to check what is happening to a job in that state to see if it's safe to rename the map at that point.

Just what it sounds like.

It's not obvious what a MapResult should do after it's been removed. Right now, things will just fail with vague plain-Python exceptions.

We can't delete the MapResult ourselves. Marking it as as "active" gives it a lot of state that could be hard to manage, and means that lots of methods would need an "am I active?" check at the top that could get ugly.

For example, doing a keyboard interrupt while waiting will leave a dangling tqdm progress bar thread.

It should be possible to pass in submit descriptors ("map options") at the global settings, per-mapped-function, and individual map level.