Sometimes a run_command() may be expensive. Right now Conductor will always re-run a run_command() (right now only run_experiment() task types have their results cached).

Having the ability to alias a task would be useful when we change the name of tasks that have previously archived results.

I've encountered a UnicodeDecodeError when using Conductor ... can't decode byte .. in position .... It seems like codecs.IncrementalEncoder might help solve this problem.

https://docs.python.org/3/library/codecs.html#incremental-encoding-and-decoding

We need to improve the error messages that Conductor shows. We should include as much context as possible to help users understand why the error occurred.

Task identifiers should not need to be fully qualified when the user is trying to refer to tasks defined in the same COND file.

We need to store all experiment data and keep track of it by git hash and the time at which it was generated. We will also want to include machine specifications. cond archive will be used to save and restore these versions.

Environments will be Conductor's way of supporting running experiments on remote machines, in the cloud, in a container, etc. (i.e., any "execution environment" that is not the machine where cond is invoked).

Right now Conductor aborts a task the moment one of its dependencies fails. We should (by default) try to run as much of the task as possible (i.e., run the other dependent tasks in the task's transitive closure that don't rely on outputs of the failed task). We can also add a command line option to restore the existing "fail early" behavior.

Right now Conductor avoids committing tasks to the version index if it fails, but this also has the side effect of not committing dependent tasks that have completed successfully. When dependent tasks are expensive to execute, this can be frustrating.

We've reached the point where our task types are complex enough that we would benefit from having documentation.

Some tasks have lots of dependencies and take a long time to run. It would be nice to have a progress indicator to indicate how many subtasks remain.

Sometimes we will have multiple versions of the same task executed against the same commit. We need a better story around managing these versions.

If we have a complicated setup process, it's helpful to be able to sweep through the configs using syntax similar to run_experiment_group().

We should record the task's output (stdout and stderr) automatically and save it with the rest of the task outputs. We should definitely do this for run_experiment() tasks (maybe run_command() too).

We should integrate with Git to record experiment commit hashes. This integration will also be useful down the road for other features related to a repository's state. The Git integration should degrade gracefully if the user is not using Git in their project.

• Python utility class for interfacing with git (#43)
• Modify version index persistent format to support commit hash tracking
• Configuration option to disable git tracking (#46)
• Graceful fallback when the Conductor project is not tracked using git (#45)
• Record commit hash when running an experiment (#45)
• Experiment version selection using closest commit ancestor (#45)

Right now only run_experiment() tasks support the options={...} and args=[...] options. Having these options available to run_command() would be nice when the command we want to run requires lots of arguments (i.e., the run_command() task would be easier to read).

Conductor now outputs a lot of status information during execution (in part due to #58). Using colors for this output would help better distinguish between the task's output and Conductor's output.

We should add a utility command to Conductor that removes any experiment task directories that are not in the version index (i.e., this would let us garbage collect any failed experiment task directories under cond-out).

Required

• Declaring a task instance
• Adding a "runnable" task (i.e. run something)
• Declaring task dependencies
• Processing an execution plan with task instances and dependencies
• Ability to run a complete execution plan
• cond clean functionality
• cond archive functionality
  • Experiment versioning by git hash and date time (#7)
  • Timestamp based identifiers (handle deduplication too)
• cond restore
• Release scripts
• Tests (#6)
• "Combine" pseudo-task

Nice to have

• CI integration for tests, formatting, and type checking
• Relative task identifiers for deps (#5)
• Consider renaming .condconfig to a non-dotfile? (e.g., cond_config.toml)

Moved to next version

• Task aliases in cond_config.toml

It's useful to be able to validate that the tasks in a COND file were defined correctly, especially since we use Python to programmatically create tasks. The check should not actually run any of the tasks defined in the file.

This is useful because the best practice right now is to commit the COND file changes first and then run. But if there is an error, we end up making many small commits while fixing up the file.

We get the following warning when releasing using pep517.build:

pep517.build is deprecated. Consider switching to https://pypi.org/project/build/

When running an experiment using --again, it would be useful to know how long the task took to run the last time.

If a task has the same name as their parent directory, it would be convenient if we could refer to it on the command line (when running cond run) without its trailing name (i.e., just the task identifier path).

For example task world defined in hello/world/COND could also be referred to as //hello/world as well instead of its fully qualified name, //hello/world:world. This would only apply to the CLI, not to dependencies in COND files.

If a task fails, we should print that it failed immediately. Right now we wait until other tasks finish running before printing that a failure occurred.

Users should be able to declare dependent tasks and Conductor should be able to load and validate them

• Add default arguments to task declarations
• Validate task dependencies
• Add mechanism to load all dependencies

Now that we have git integration, it would be useful to be able run a task again for the current commit. The motivation is that using --again force reruns the task and its dependencies. But if a previous --again run had partially succeeded, it's not straightforward to retry the run without rerunning successfully completed dependent tasks.

It would help to be able to declare arguments that should be passed to the executable in a run_experiment() and run_command() as part of the task definition in the COND file. This is because certain experiments may require many arguments and it is tedious to either (i) create a separate script for each instance, or (ii) tack them on to the run argument (making it really long).

For example

run_experiment(
  name="benchmark",
  run="./run_benchmark.sh",
  args={
    memory: "2G",
    threads: "3",
  },
)

would translate to running ./run_benchmark.sh --memory=2G --threads=3.

We need to handle task aborting gracefully (e.g., when the user presses Ctrl-C while a task is running).

If you have a long list of dependent tasks, it is useful to be able to partition them automatically (so they can run on different machines).

Occasionally we will need to run tasks that must run as root (e.g., because they access system counters). It would be nice to have a way to handle these situations gracefully.

Conductor should be able to parse the dependency graph to create an execution plan and detect any errors in the graph (e.g., dependency cycles). Conductor should then be able to execute the plan.

• Assemble dependencies into a graph
• Validate graph and generate an execution plan
• Execute the execution plan
  • Set up input/output paths properly
  • Detect re-execution of cached results

Conductor should have a (at least minimal) website. We can host the documentation there.

When writing custom Python scripts (e.g., to process experiment data), it will be helpful to have helper functions that help with using the COND_DEPS and COND_OUT environment variables.

The way we currently implement task types is not ideal (a bit too hacky), especially with our need to separately compile the Conductor "standard library" task types. The code can probably be refactored and improved.

Often one needs to generate a parameter sweep, or several run_experiment() instances where certain parameters need to be varied. We should help users with this and provide a nice way to specify a "configuration" where the configuration can then also be stored with the experiment task output.

A follow up to #69. There are situations where we will not want to re-run for the current commit. We just want to make sure we have a version no older than some given commit.

Specifying a timeout for a task can be useful if there is a risk of the task running for longer than expected.

This project should be well tested. We need to set up tests for our Python code.

• Set up initial testing infrastructure
• Add a reasonable number of tests for the core parts of Conductor
• Set up a GitHub Action workflow that runs the tests

Sometimes we may want to restore results multiple times (to include new results). It would be nice to ignore any cached results that have an identical version to existing cached results.

Add a convenience file that maps to COND_OUT/some_file if the environment variable is set, falling back to sys.stdout otherwise. This helps keep experiment scripts runnable without being invoked by Conductor.

Conductor currently prints blank lines between progress messages. This works fine when tasks are executed sequentially. But when we launch tasks in parallel, the spacing between messages is less useful (and can be confusing).

Some tasks may be parallelizable (in the sense that they are "safe" (e.g., not performance sensitive) to run concurrently with other tasks). It would be useful if the user could specify this when defining their tasks. Then Conductor should launch parallelizable tasks concurrently on behalf of the user.

• Refactor task execution to enable parallel execution (#47)
• Implement parallel execution (#48)

Experiment results are stored in directories with a timestamp for uniqueness. However this makes it a bit cumbersome to find the latest results for a particular task. We should provide a way for users to easily get this information (e.g., a command that provides the path to the task's outputs).

Running cmake from run_command() seems to result in an error:

CMake Error: Could not find CMAKE_ROOT !!!
CMake has most likely not been installed correctly.
Modules directory not found in

CMake Error: Error executing cmake::LoadCache(). Aborting.

Maybe this is due to PATH problems or some other environment misconfiguration.