orionw / configtune Goto Github PK

Are these debugging lines that should be removed?

https://github.com/orionw/tuningDEAP/blob/276fadbc05f51a1b4240ec58623bc4b6aa0c1dbd/tuningdeap/main.py#L105-L106

From

"<param_name>": {
    "<para_type>": [<min>, <max>, <step>]
}

to

"param_name": {
    "type": "<para_type>"
    "min": <min>,
    "max": <max>,
    "step": <step>
}

"step" could be called "precision". Are there better names?

Could possibly enforce a time-out no matter where we are in the tuning process and write the results to file.

Not sure if wanted or needed, but it is an option.

It'd be nice to be able to feed old results into the start of tuningDeap so that you don't have to restart every time.

We could do this by taking in a csv file similar to the outputted results files and using that to initialize the initial population.

Similar to TuningDeap functionality.

https://github.com/orionw/tuningDEAP/blob/276fadbc05f51a1b4240ec58623bc4b6aa0c1dbd/tuningdeap/main.py#L37-L47

The output director could be made an argument to the constructor. This allows it to be set dynamically more easily. We could also get rid of the output flag and let the presence of output_dir determine whether to write the output, i.e. if output_dir is None don't write anything.

With 2 generations of size 2, I am seeing that chromosomes are duplicated. For example:

The results from generation 1 are:
   hidden_layer_size  n_hidden_layers  use_batch_norm  use_skip  dropout     score
0                 27                5               1         0      0.2  0.755746
1                 16                4               1         0      0.2  0.755746
On generation 2
The results from generation 2 are:
   hidden_layer_size  n_hidden_layers  use_batch_norm  use_skip  dropout     score
0                 16                4               1         0      0.2  0.755746
1                 27                5               1         0      0.2  0.755746

Based on my output, it did not call the evaluate function twice for the same chromosome, so it did not waste time. However, I did not get the 2x2 chromosomes and evaluate calls I was expecting.

Looking at https://scikit-optimize.github.io/, it looks like we could use skopt as well. We could wrap skopt and give users the option to choose between bayesian and genetic algorithms.

This issue is to investigate the potential use cases and feasibility as well as prototyping an initial class that could do Bayesian optimization, if possible.

Current TuningDeap expects a function that returns a tuple and TuningBayes expects a scalar. It would be nice to standardize this.

https://github.com/orionw/tuningDEAP/blob/276fadbc05f51a1b4240ec58623bc4b6aa0c1dbd/tuningdeap/main.py#L40

I think it would be good to set whether we minimize or maximize in the constructor, so it is in the same place as the fitness function. This allows for dynamic setting of min/max.

The tests output

INFO:tuningdeap.main:There was an exception evaluating: eval_function() missing 1 required positional argument: 'n_values'

Is this a problem?

Would help with reproducibility, if someone wanted it.

The name tuningdeap is no longer relevant since we have bayesian methods.

Maybe configtuner or configtuning? Or tuningml?

We'll need to update setup.py and pypi once we decide.

In just the simple test code in the DNA repo, I am getting that the population decays from 2 to 1 in from the 1st to the 2nd generation.

Also, the the chromosome in generation 2 matches one of the chromosomes in generation 1 exactly, but with a different score. Futhermore, the score of the 1 chromosome in gen 2 equals the score from the non-matching chromosome in gen 1. This seems like a bug in the code.

Here are the output files:

,hidden_layer_size,n_hidden_layers,use_batch_norm,use_skip,dropout,score
0,21,3,0,0,0.1,0.5951957171535395
1,3,7,0,1,0.30000000000000004,0.31058696414310893

,hidden_layer_size,n_hidden_layers,use_batch_norm,use_skip,dropout,score
0,21,3,0,0,0.1,0.31058696414310893

It seems that the population can sometimes dwindle from one generation to the next. In the tests I get:

INFO:tuningdeap.main:The population has a size of 8 after rejecting the invalid: should be 10

What is causing this? It does not seem like desired (default) behavior.