igitugraz / l2l Goto Github PK

So while doing the benchmark evaluations today, with grid search optimizer for 10D
cases(you can change function_id = 1 in bin/ltl-fun-gs.py) this error was thrown:

Traceback (most recent call last):
  File "/usr/lib/python3.4/runpy.py", line 170, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/lib/python3.4/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/home/sinisa/.local/lib/python3.4/site-packages/scoop/bootstrap/__main__.py", line 302, in <module>
    b.main()
  File "/home/sinisa/.local/lib/python3.4/site-packages/scoop/bootstrap/__main__.py", line 92, in main
    self.run()
  File "/home/sinisa/.local/lib/python3.4/site-packages/scoop/bootstrap/__main__.py", line 290, in run
    futures_startup()
  File "/home/sinisa/.local/lib/python3.4/site-packages/scoop/bootstrap/__main__.py", line 271, in futures_startup
    run_name="__main__"
  File "/home/sinisa/.local/lib/python3.4/site-packages/scoop/futures.py", line 64, in _startup
    result = _controller.switch(rootFuture, *args, **kargs)
  File "/home/sinisa/.local/lib/python3.4/site-packages/scoop/_control.py", line 253, in runController
    raise future.exceptionValue
  File "/home/sinisa/.local/lib/python3.4/site-packages/scoop/_control.py", line 127, in runFuture
    future.resultValue = future.callable(*future.args, **future.kargs)
  File "/usr/lib/python3.4/runpy.py", line 240, in run_path
    pkg_name=pkg_name, script_name=fname)
  File "/usr/lib/python3.4/runpy.py", line 96, in _run_module_code
    mod_name, mod_spec, pkg_name, script_name)
  File "/usr/lib/python3.4/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "bin/ltl-fun-gs.py", line 99, in <module>
    main()
  File "bin/ltl-fun-gs.py", line 73, in main
    parameters=parameters)
  File "/home/sinisa/Uni/Project_CI/LTL/ltl/optimizers/gridsearch/optimizer.py", line 97, in __init__
    curr_param_list = np.meshgrid(*([curr_param_list] * param_length), indexing='ij')
  File "/home/sinisa/.local/lib/python3.4/site-packages/numpy/lib/function_base.py", line 4532, in meshgrid
    mult_fact = np.ones(shape, dtype=int)
  File "/home/sinisa/.local/lib/python3.4/site-packages/numpy/core/numeric.py", line 192, in ones
    a = empty(shape, dtype, order)
MemoryError

For 2D cases it seems to be working fine though

Performance would probably be best individual in that iteration/generation.

Should it be a weighted sum (as it is currently) or a lexicographic ordering?

Instead of using the style-checker run-style-check.sh I would like to recommend to use pep8 speaks instead. One can configure it in the same fashion as the checker script and it can run e.g. only on the code of the actual pull request. Right now the CI always breaks because of all pep8 issues from all files. Is this a volitional behavior?

Pypet has the following issues that are critical to fix

1. A memory leak that leads to quadratically increasing memory as the number of generations increase (This is a real reason for the closed issue #16)
2. A steadily increasing runtime as the number of generations increase

Currently, it seems like only CE has default parameters.

Since there are papers [1][2] that claim it's the bees knees.

[1] Salimans, Tim, et al. "Evolution Strategies as a Scalable Alternative to Reinforcement Learning." arXiv preprint arXiv:1703.03864 (2017).
[2] Morse, Gregory, and Kenneth O. Stanley. "Simple Evolutionary Optimization Can Rival Stochastic Gradient Descent in Neural Networks." Proceedings of the 2016 on Genetic and Evolutionary Computation Conference. ACM, 2016.

When adding parameter groups in the CE and GS optimizers, they complain that the ParameterDict is not callable.
e.g.
traj.parameters.f_add_parameter_group('grid_spec')
TypeError: 'ParameterDict' object is not callable

We want the step size to be a hyper-parameter of the optimizee rather than an optimizer hyper-parameter, since each optimizee has different requirements, and may even have different required step sizes for each dimension.

One of the solutions suggested by @guillaumeBellec would be to allow the current bounding function to also scale individual so that all optimizers only have to generate individuals between 0 and 1. The optimizee knows the required scaling along with the bounding and returns an appropriately scaled and bounded individual.

We have to think about how this would affect the optimizers we have, since it would change the error landscape they see. Specifically, how would the cross-entropy optimizer have to be modified to be able to do this? (i.e. how would it compensate for the scaling?).

@maharjun @bohnstingl @franzscherr Any thoughts about this?

Other possible ways to achieve optimizer independent step-size are also welcome.

Having the value of root_dir in the bin/*.py file and having to edit/un-edit it before each checkin might not be ideal.

One solution, suggested by @dsalaj, would be to have a separate file, that is ignored by git, containing the path string, and the bin/*.py files just read this.

SCOOP BUG: SCOOP basically does not read out of the pipes to which the remote stdout is attached. This means that if the remote process prints more than 64 kb of data to stdout, that subprocess will hang and your simulations will never complete. Experienced this while trying to run the Ison Memory Simulation experiment and by observing the SCOOP code.

It seems like bin/ltl-fun-gs.py is using the old FuntionOptimizee interface.

And report the results from multiple starting points or fixed starting points.

E.g. strings, or just a list of possible values.

I have three questions regarding the benchmark evaluation:

The commit ID is kind of misleading, since one has to commit at least once to set the path for local execution. The problem is that the commit is only locally and cannot be pushed to the master directly.
How can we handle this?

The parameters for the individuals used to get the best performance are not tracked.
This might be also an interesting information?

The field "actual optima" seems to be None at every execution.
Am I doing something wrong or is this not implemented yet?

Thanks for your help!

The fact that none of the optimizers have any capabilities for seeding is problematic in the least. I think the following two features will be required for all optimizers that require random number generation,

1. There must be a seed parameter that defaults to None (In which case, the generator is unseeded)
2. They must NOT use the default numpy generator, this is because, when creating an optimizee with already existing code, in many cases, the optimizee will be using numpy's default generator and it will be difficult to change it there. This will cause conflicts with the optimizer if the numpy generator is used by both.

Should be based on the method in published literature

Including RMSProp

To be consistent with the rest of the optimizers

How do we construct such functions? (references)

• Plot average performance vs iteration for all optimizers (#44)
• Each optimizer should be run for multiple starting points and average fitness reported
• Comparison should be done for functions that have known optima (e.g. Gaussian combinations)
• Need to benchmark with extremely high dimensional functions (> 100-1000 D)

Which is not known in most cases. It should be defined as the delta change in fitness where the optimization is stopped.

Seeing as how pypet leaking memory isn't enough, we also have SCOOP filling in for it. I have currently plugged the leak on my own fork and will put up the relevant pull request with the changes shortly. Until that pull request is merged into scoop master, we will go with my fork.

The paper [1] proposes alternatives to the CE method. Maybe they are also interesting for LTL purposes?

[1] http://citeseerx.ist.psu.edu/viewdoc/download;jsessionid=16227005B9C7ACE7E5471E6073081A77?doi=10.1.1.330.6278&rep=rep1&type=pdf

The formula for the rosenbrock function as given in the mentioned reference and other pages includes a factor of 100 in one of the terms that has been erroneously omitted in the current implementation.

I'm using the simulated annealing optimizer, with the following parameters:

parameters = SimulatedAnnealingParameters(n_parallel_runs=1, noisy_step=.1, temp_decay=0.95, n_iteration=150, stop_criterion=np.Inf, seed=np.random.randint(1e5), cooling_schedule=AvailableCoolingSchedules.DEFAULT)

    optimizer = SimulatedAnnealingOptimizer(traj, optimizee_create_individual=optimizee.create_individual, optimizee_fitness_weights=(1,), parameters=parameters, optimizee_bounding_func=optimizee.bounding_func)

I'm returning from my optimizee a fitness between 0 and 1 (1 being the best), but still, the optimizer is trying to minimize that value.

Is that normal?

From the appendix of:
Onbaşoğlu, Esin, and Linet Özdamar. “Parallel Simulated Annealing Algorithms in Global Optimization.” Journal of Global Optimization 19, no. 1 (January 1, 2001): 27–50. https://dx.doi.org/10.1023/A:1008350810199.

The plan is to generate a markdown snippet that contains the following information: name of Function, parameters, name of optimizer, parameters, number of iterations, optima found, actual optima (if known), time of run (approx), current git commit id (no uncommited changes), date!

The markdown snippet is generated using a Jinja2 template.
Git commit id is included using GitPython

The class has format:

class Recorder:
  def start():
    # Check no uncommitted changes (throw error if not)

  def end(...args...):
    # Generate markdown file with the above fields in the current directory
    # Optionally, the markdown file contains the cost for each iteration (and possibly plot)

Usage:

recorder = Recorder()
recorder.start() # Start before everything else
# Do your experiment
recorder.end() # Last line of code

The user uploads the generated markdown file wherever -- GitHub wiki, their own repository etc.

It appears that even after the latest patch scoop is still losing futures. This things appears to be meant as some sort of insurance against a worker failing midway, however, due to issues it doesn't seem to work correctly. I have some idea of what's wrong and will get around to fixing it. However in the meanwhile, I would suggest the following workaround

go to the scoop code (to find out the directory, type import scoop; scoop.__file__ in an IPython console) to the file _types.py to line 389 and comment out self.askForPreviousFutures(). SCOOP will stop losing futures. Although the robustness to a lost worker will be gone, this should be fine considering we don't really lose workers ever.

The create_individual function or the spec therefrom should be used instead of having to change this for each optimizee. This breaks interface consistency.

The following lines are creating multiple directories at the same level as the parent

Paths should be generated using os.path.join(parent, child). For example,

xml_path = os.path.join(self.path, "jube_xml")
if not os.path.exists(xml_path):
    os.makedirs(xml_path)

The real issue is presented further down in the source code. This seems to be the same as the initial setting

The main issue is that, by manually generating the paths, we lose compatibility between OS-s and risk misplacing files.

• Results also generated in a form appropriate for the Github wiki
• To address commit id issue raised in #38 the following is proposed:
  • Each run of a benchmark creates file in the existing tree e.g. results/benchmark-<id>/<filename>.md.
  • The person running the benchmark should check in the created benchmark files and push it to master.
  • Then, the .md generated in the benchmark should be pasted onto the github wiki in the benchmarks page
• The benchmark md file should not contain non human paraseable information (e.g. the mean and noisy covariance currently being recorded)
• The parameters for the individuals used to get the best performance should be tracked (From #38)
• The actual optima should say "unknown" if unknown.

Ideas from David's work.

See work from Kroese and Rubenstein

The postprocess function is called recursively and the stack is only unwinded after finishing all computations. This results in heavy memory consumption when using local variables within the postprocess function.

I cannot seem to think of a way to implement smoothing for gaussian mixture models that makes any sense. This is because in the current implementation, there is absolutely no logic to decide which component gets linearly combined with which. This leads to pretty arbitrary smoothing I would think. Is there any way to do this in a manner that makes sense?

Hi! In this portion of the optimizer, what I understand is that you first check if there are repeated individuals, and then you want to mutate those repeated

But in line 176, I think the iteration should start with i+1 instead of i. Otherwise , since your second loop starts with the same individial as the first one, you would be mutating all individuals 80% probability. Furthermore, I believe that, in line 175, offspring should iterate on [:-1].

Implementation available in scikit-learn

Especially in noisy environments it could be useful that the parameters chosen by the optimizer are evaluated multiple times to better estimate the mean and the variance of the optimizees fitness.

as reported by the optimizee and not the fitness multiplied by the fitness weights (which should only be used internally by the optimizer).

Any tips on how can this be solved? And where? Since LTL-SPORE is based on vanilla pattern matching experiment from spore-nest-module, would it be easier to adapt the optimizer? On the other hand optimizers should be decoupled from optimizees.

Console output:

$ python3 -m scoop bin/ltl-pm-gd.py 
[2017-06-01 13:05:02,949] launcher  INFO    SCOOP 0.7 1.1 on linux using Python 3.5.2 (default, Nov 17 2016, 17:05:23) [GCC 5.4.0 20160609], API: 1013
[2017-06-01 13:05:02,949] launcher  INFO    Deploying 4 worker(s) over 1 host(s).
[2017-06-01 13:05:02,949] launcher  INFO    Worker distribution: 
[2017-06-01 13:05:02,950] launcher  INFO       127.0.0.1:	3 + origin
All output can be found in file  /home/dsalaj/Documents/CI_project/LTL-SPORE/results/LTL-PatternMatching-GD/run-no-test/results/output.log
Change the values in logging.yaml to control log level and destination
e.g. change the handler to console for the loggers you're interesting in to get output to stdout
MainProcess STDOUT INFO     Established redirection of `stdout`.
MainProcess snn_utils.comm.zmq INFO     Subscribing to tcp://127.0.0.1:8001
Traceback (most recent call last):
  File "./zmq_proxy_node_py3.py", line 7, in <module>
    ZMQProxyNode().main()
  File "/home/dsalaj/Documents/CI_project/LTL-SPORE/spore/snn_utils/comm/music/node/__init__.py", line 75, in main
    self._setup(self._music_setup)
  File "/home/dsalaj/Documents/CI_project/LTL-SPORE/spore/zmq_proxy_node.py", line 57, in _setup
    self._server = Publisher(music_zmq_proxy_config['communication']['port'])
  File "/home/dsalaj/Documents/CI_project/LTL-SPORE/spore/snn_utils/comm/zmq/__init__.py", line 29, in __init__
    self._sock.bind(address)
  File "zmq/backend/cython/socket.pyx", line 495, in zmq.backend.cython.socket.Socket.bind (zmq/backend/cython/socket.c:5653)
  File "zmq/backend/cython/checkrc.pxd", line 25, in zmq.backend.cython.checkrc._check_rc (zmq/backend/cython/socket.c:10014)
zmq.error.ZMQError: Address already in use
[ubuntu-x230:16282] *** An error occurred in MPI_Group_free
[ubuntu-x230:16282] *** on communicator MPI_COMM_WORLD
[ubuntu-x230:16282] *** MPI_ERR_GROUP: invalid group
[ubuntu-x230:16282] *** MPI_ERRORS_ARE_FATAL: your MPI job will now abort
Traceback (most recent call last):
  File "./zmq_proxy_node_py3.py", line 7, in <module>
    ZMQProxyNode().main()
  File "/home/dsalaj/Documents/CI_project/LTL-SPORE/spore/snn_utils/comm/music/node/__init__.py", line 75, in main
    self._setup(self._music_setup)
  File "/home/dsalaj/Documents/CI_project/LTL-SPORE/spore/zmq_proxy_node.py", line 57, in _setup
    self._server = Publisher(music_zmq_proxy_config['communication']['port'])
  File "/home/dsalaj/Documents/CI_project/LTL-SPORE/spore/snn_utils/comm/zmq/__init__.py", line 29, in __init__
    self._sock.bind(address)
  File "zmq/backend/cython/socket.pyx", line 495, in zmq.backend.cython.socket.Socket.bind (zmq/backend/cython/socket.c:5653)
  File "zmq/backend/cython/checkrc.pxd", line 25, in zmq.backend.cython.checkrc._check_rc (zmq/backend/cython/socket.c:10014)
zmq.error.ZMQError: Address already in use
Traceback (most recent call last):
  File "./zmq_proxy_node_py3.py", line 7, in <module>
    ZMQProxyNode().main()
  File "/home/dsalaj/Documents/CI_project/LTL-SPORE/spore/snn_utils/comm/music/node/__init__.py", line 75, in main
    self._setup(self._music_setup)
  File "/home/dsalaj/Documents/CI_project/LTL-SPORE/spore/zmq_proxy_node.py", line 57, in _setup
    self._server = Publisher(music_zmq_proxy_config['communication']['port'])
  File "/home/dsalaj/Documents/CI_project/LTL-SPORE/spore/snn_utils/comm/zmq/__init__.py", line 29, in __init__
    self._sock.bind(address)
  File "zmq/backend/cython/socket.pyx", line 495, in zmq.backend.cython.socket.Socket.bind (zmq/backend/cython/socket.c:5653)
  File "zmq/backend/cython/checkrc.pxd", line 25, in zmq.backend.cython.checkrc._check_rc (zmq/backend/cython/socket.c:10014)
zmq.error.ZMQError: Address already in use
--------------------------------------------------------------------------
mpirun has exited due to process rank 2 with PID 16282 on
node ubuntu-x230 exiting improperly. There are two reasons this could occur:

1. this process did not call "init" before exiting, but others in
the job did. This can cause a job to hang indefinitely while it waits
for all processes to call "init". By rule, if one process calls "init",
then ALL processes must call "init" prior to termination.

2. this process called "init", but exited without calling "finalize".
By rule, all processes that call "init" MUST call "finalize" prior to
exiting or it will be considered an "abnormal termination"

This may have caused other processes in the application to be
terminated by signals sent by mpirun (as reported here).
--------------------------------------------------------------------------
[ubuntu-x230:16291] *** An error occurred in MPI_Group_free
[ubuntu-x230:16291] *** on communicator MPI_COMM_WORLD
[ubuntu-x230:16291] *** MPI_ERR_GROUP: invalid group
[ubuntu-x230:16291] *** MPI_ERRORS_ARE_FATAL: your MPI job will now abort
[ubuntu-x230:16287] *** An error occurred in MPI_Group_free
[ubuntu-x230:16287] *** on communicator MPI_COMM_WORLD
[ubuntu-x230:16287] *** MPI_ERR_GROUP: invalid group
[ubuntu-x230:16287] *** MPI_ERRORS_ARE_FATAL: your MPI job will now abort
--------------------------------------------------------------------------
mpirun has exited due to process rank 2 with PID 16287 on
node ubuntu-x230 exiting improperly. There are two reasons this could occur:

1. this process did not call "init" before exiting, but others in
the job did. This can cause a job to hang indefinitely while it waits
for all processes to call "init". By rule, if one process calls "init",
then ALL processes must call "init" prior to termination.

2. this process called "init", but exited without calling "finalize".
By rule, all processes that call "init" MUST call "finalize" prior to
exiting or it will be considered an "abnormal termination"

This may have caused other processes in the application to be
terminated by signals sent by mpirun (as reported here).
--------------------------------------------------------------------------
--------------------------------------------------------------------------
mpirun has exited due to process rank 2 with PID 16291 on
node ubuntu-x230 exiting improperly. There are two reasons this could occur:

1. this process did not call "init" before exiting, but others in
the job did. This can cause a job to hang indefinitely while it waits
for all processes to call "init". By rule, if one process calls "init",
then ALL processes must call "init" prior to termination.

2. this process called "init", but exited without calling "finalize".
By rule, all processes that call "init" MUST call "finalize" prior to
exiting or it will be considered an "abnormal termination"

This may have caused other processes in the application to be
terminated by signals sent by mpirun (as reported here).
--------------------------------------------------------------------------

A dummy Optimizer that does a grid search over a given range of parameters.