meta-optimization / l2l Goto Github PK

Most software packages have their tests on the same level as e.g. doc.
This allows out of code testing.

• Define a set of basic functional requirements

• Adapt already existing optimizee examples to a test unit which explores the functional requirements

• Add support for all optimizers

After the automated tests are up we can add CI to L2L

In the optimizer the parameter naming is not standardized due to the namedtuple parameters and the class arguments. There should be an easier way to create parameters.

We would like to streamline how parameters and fitness results are sent from the optimizee to the optimizers. Links to issue #5

There is a lot of boiler code when creating the optimizer, e.g. adding all parameters to the trajectory. We should check if everything is needed to add and if it is possible not to add certain parameters.

There are lots of unused branches. They should be removed.

Create a view on the trajcetory for cleaning up purposes.

• Create a visual representation for the object to see where the data is stored
• Remove redundancies
• improve data access

The framework produces a lot of files, on supercomputers with I node issues this might cause issues.

Solution zip the files per generation?

The abstract optimizee class should also have optimizee_parameters like the optimizer has.

https://github.com/Meta-optimization/L2L/blob/master/bin/l2l-template.py does not contain the call to experiment.end_experiment(optimizer). Please add for completeness.

EnKF can be added as an optimizer to the optimizer section. Code is ready and can be transferred.

As the titel says:

The folder 'results' has been created in the parent folder of the L2L repo.
I only ran the tests.

This is a collection issue for the logging refactoring task.

1. Standardize log file location: make it configurable on installation
2. A single log per run
3. At end of generation add all optimizee logs to a single large logfile
   a. Keep the individual logfiles
4. Create easy to parse log line format
5. Check all loglines for informative contect
6. Add 'ids' to logline to add additional information

Verify where the logging information is being stored and make it consistent through the whole framework.

coverall.io is connected to the github repo but not invoked.

Currently each individual has their own trajectory file where the Individual is the parameter set that they will execute. This should be changed so that only one trajectory exists.

• Change the way the trajectory is pickled
• Change the way the trajectory and the individual are loaded by the optimizee
• Update the trajectory with the results and HOF/ other data to be preserved after each generation

Create an interface for Unicore to easily submit jobs to HPCs supported by Unicore.

Genetic Algorithm optimizer should output the generation and individual number of the best individual after the total simulation ends. I.e. we need to save individual and generation number of the best individual in an appropriate structure.

Logger facilities should be checked and made homogeneous. For example a yaml file is used to configure the logger, but most of the cases the logger file is set from the run script without any additional files.

The commandline option --process-dependency-links was dropped in pip v19. What's the advised alternative?

I see a install.sh in the root, but that appears to be doing something unrelated. Or do I see that wrong?

The documentation should address the new changes with the experiment class.

It is not possible to debug the optimizee while the program is executed. Only a single instance alone can be debugged (using main) without the l2l process. That means the outer loop cannot be applied to see the optimization.

attributeError: module 'platform' has no attribute 'linux_distribution'
Specifically this is caused because the matplot lib needed for L2L is an old one. (<3.1)

https://stackoverflow.com/questions/63636697/error-module-platform-has-no-attribute-linux-distribution

Why is L2L dependend on such an old matplotlib?
Why is L2L plotting?

Find out how logfiles are produced in L2L.
Check if important information is logged.

find out important messages and start thinking about the regex to parse

I spend the last two days debugging the deployment of Jube.

It ended up being caused by the exec line not being correct. Given that this is an extremely essential step and we can expect that a lot of user will come from Ubuntu. This should be solved.

I would also suggest to add a unit test in the framework testing this behavior.
Currently the tests fail silently if this happens. Having a dedicated test running a small exec with the supplied config would fail with a clear suggestion how to fix the problem

Finalize the Jube PR.

I encountered the challenge that when an optimizee fails due to 'reasons'. The outer loop hangs on the existing of a file that would be created by the optimizee.
This is not optimal behaviour

Line 220ish in JUBE_runner.py

` # Wait for ready files to be written

    while not self.is_done(ready_files):

        time.sleep(5)`

All runner scripts are using the old format and should be adapted to the l2l-template.py

Some optimizers are lacking a proper explanation i.e. how the optimization algorithm works. A short outline and citation to read further on should be added. EvolutionaryStrategies is a good example.

The current versions of linux do not come with pip but pip3.
This means that out of the box newer linux users can not use the install script

Some intelligence needs to be added to the install.sh to catch this behavior.

In the optimizee check that the fitness is sent to the optimizer correctly and in standardized fashion.

For example in the runner/main scripts similar code is copied from one runner to the other. Are there ways to reduce this overhead? Links also to issue #4

We're running into an issue with L2L master (as of time of posting). We've written an optimizee, but an error is generated when we try to run the optimization:

Traceback (most recent call last):
  File "model.py", line 180, in <module>
    main()
  File "model.py", line 173, in main
    env.run(optimizee.simulate)
  File "/home/brent/.local/lib/python3.8/site-packages/Learning_to_Learn-1.0.0b0-py3.8.egg/l2l/utils/environment.py", line 45, in run
  File "/home/brent/.local/lib/python3.8/site-packages/Learning_to_Learn-1.0.0b0-py3.8.egg/l2l/utils/JUBE_runner.py", line 53, in __init__
KeyError: 'paths_obj'

Not 100% sure if it's the source for the error, but we set the paths thus:

    root_dir_path = "."
    paths = Paths(name, dict(run_no='test'), root_dir_path=root_dir_path)
    traj.f_add_parameter_to_group("JUBE_params", "paths", paths)

Basically copied from one of the examples. Any idea what's going wrong? There is an optimizee.bin created in the path you might expect, so it seems the paths_obj is succesfully used somewhere.

Our code: https://github.com/brenthuisman/l2l-arbor
We use L2L @ master and https://github.com/Meta-optimization/JUBE @ master.

An optional dataset structure where it is possible to register optimizer and optimizee. The optimizer can send the index of the dataset to the optimizee to read then the data from.

Often the fitness function is delivered as a dictionary, depending on the optimizer a list may be required. In the process of converting between lists and dictionaries errors may occur. We should check if the conversions are needed and if they are done correctly.

Since matplotlib is not in the requirements anymore we should remove any existing pieces of code of matplotlib. It can affects the run of some examples. If the removal breaks something, we need to fix it in an other way.

Provide also examples. Step by step tutorial (e.g. how to write the optimizee and integrate it) A collab or jupyter notebook would be interesting.

When starting the script a config file, path.conf is required which sets the path to the results. This should be a string, instead a config file. Additionally the path should be checked a directory should be added if the folder is missing.

• JUSUF cloud: create instances, volumes and network settings
• Issue (02.11.2020): ssh forwarding not working properly -- can't access ES-nodes
  -- temporary solution: allocated floating IPs for all instances
• Deployment / Installation
  • CentOS: update
  • Install java, add repositories, add certificates
  • Install ELK stack
• Wednesday, Nov 4th:
  • Issues with JUSUF-Cloud networks --> admins in contact with RedHat
  • Configuration of ELK (when issues are resolved)
  • Install/Deploy Filebeat and Metribeat on edge machines and test monitoring
• Thursday, Nov 5th:
  • Deployed, installed and configured single-node ELK stack
  • Kibana: http://134.94.88.100:5601/
  • Documentation on collab2
• Friday, Nov 6th:
  • installed filebeat, configured it, tested locally
  • logging of custom logfiles possible
  • logging onthe supercomputer not possible yet