sheffieldml / gpyopt Goto Github PK

View Code? Open in Web Editor NEW

928.0 928.0 260.0 124.05 MB

Gaussian Process Optimization using GPy

License: BSD 3-Clause "New" or "Revised" License

Python 23.64% Jupyter Notebook 76.36%

gpyopt's People

Contributors

Stargazers

Watchers

Forkers

maxentile beckdaniel lenovor benjamesbabala commonlibs alibaheri yinyumeng thesilencelies joshloyal mikecroucher iaroslav-ai florianwilhelm calm85 c0g saschwan jprk omorhefere moonseopkim rasmusbergpalm zhusj jdaaph mathdr f-a stringhamc midak ivan-wang erikwijmans mossuru777 paulrolland1307 fx-kirin yxchng nrontsis amaotone timovwb daventero statml gpfins nawalgao whg1g15 mdlavin amueller lrozo markusprim sverzijl yangqiu labodyn komorihi alexkuhnle loneknightz jkaardal alansaul vishalbelsare siddsach xrarach tinutjoy europj stanley-jc anirband mabodo fdtomasi seanhsieh alaya-in-matrix leleonora craigfe fmnyikosa liujiandu befelix grseb9s jjakimoto takuma-ya tom-hydrogen jenny-nlc natandemaman farquet hitarthk jintaiyang epistra mostafa86 fredericsauv iamharvey vmarkovtsev urjitbhatia auserj tony32769 liuweiping2020 benzei xflee silverspirit akshayats parsonszeng pac-bayesian edlanglois mariabaldeon marpulli ajgpitch shubhampachori12110095 rafaol knut0815 o7s8r6 deepaklabh

gpyopt's Issues

IndexError: bug or problem?

Hi there,

I can't seem to figure out the cause of the following problem (stacktrace included below).

My problem is the optimization of an objective function with six hyperparameters, of which two are discrete and the rest continuous.

Could you help me out where I made a mistake, or where the bug is?

Code:

def objective(args):
    arg = args.reshape(-1,)
    # ...
    return np.array([[mse]])

domain = [
# ...
]

initial_parameters = np.array([[1000, 0.5, 0.3, 1.25, 1e-8, 10], 
                               [1000, 0.5, 0.3, 1.25, 1e-8, 10], 
                               [1000, 0.5, 0.3, 1.25, 1e-8, 10]])

initial_mse = np.array([[0.15945], 
                        [0.10871], 
                        [0.053224]])

bo = GPyOpt.methods.BayesianOptimization(f=objective, domain=domain, 
X=initial_parameters, Y=initial_mse)

Stacktrace:

IndexError                                Traceback (most recent call last)
<ipython-input-17-ca9b66ea1c32> in <module>()
---> 31 bo = GPyOpt.methods.BayesianOptimization(f=objective, domain=domain, X=initial_parameters, Y=initial_mse)
     32 
     33 

/Users/Reinier/anaconda/lib/python3.5/site-packages/GPyOpt/methods/bayesian_optimization.py in __init__(self, f, domain, constrains, cost_withGradients, model_type, X, Y, initial_design_numdata, initial_design_type, acquisition_type, normalize_Y, exact_feval, acquisition_optimizer_type, model_update_interval, evaluator_type, batch_size, num_cores, verbosity, verbosity_model, bounds, **kwargs)
    234 
    235         # --- Initialize everything
--> 236         self.run_optimization(max_iter=0,verbosity=self.verbosity)
    237 
    238     def _model_chooser(self):

/Users/Reinier/anaconda/lib/python3.5/site-packages/GPyOpt/methods/bayesian_optimization.py in run_optimization(self, max_iter, max_time, eps, verbosity, save_models_parameters, report_file, evaluations_file, models_file, **kwargs)
    450                 self.acquisition_optimizer.optimizer ='CMA'
    451             print('WARNING: "acqu_optimize_method" will be deprecated in the next version!')
--> 452         super(BayesianOptimization, self).run_optimization(max_iter = max_iter, max_time = max_time,  eps = eps, verbosity=verbosity, save_models_parameters = save_models_parameters, report_file = report_file, evaluations_file= evaluations_file, models_file=models_file)
    453 
    454 

/Users/Reinier/anaconda/lib/python3.5/site-packages/GPyOpt/core/bo.py in run_optimization(self, max_iter, max_time, eps, verbosity, save_models_parameters, report_file, evaluations_file, models_file)
    104 
    105             # --- Update and optimize acquisition and compute the exploration level in the next evaluation
--> 106             self.suggested_sample = self._compute_next_evaluations()
    107 
    108             if not ((self.num_acquisitions < self.max_iter) and (self._distance_last_evaluations() > self.eps)):

/Users/Reinier/anaconda/lib/python3.5/site-packages/GPyOpt/core/bo.py in _compute_next_evaluations(self)
    182         Computes the location of the new evaluation (optimizes the acquisition in the standard case).
    183         """
--> 184         return self.evaluator.compute_batch()
    185 
    186     def _update_model(self):

/Users/Reinier/anaconda/lib/python3.5/site-packages/GPyOpt/core/evaluators/sequential.py in compute_batch(self)
     19         Selects the new location to evaluate the objective.
     20         """
---> 21         return self.acquisition.optimize()
     22 
     23 

/Users/Reinier/anaconda/lib/python3.5/site-packages/GPyOpt/acquisitions/base.py in optimize(self)
     57             out = self.optimizer.optimize(f=self.acquisition_function)[0]
     58         else:
---> 59             out = self.optimizer.optimize(f=self.acquisition_function, f_df=self.acquisition_function_withGradients)[0]
     60         return out
     61 

/Users/Reinier/anaconda/lib/python3.5/site-packages/GPyOpt/optimization/acquisition_optimizer.py in optimize(self, f, df, f_df)
    287 
    288         for i in range(num_discrete):
--> 289             self.mixed_optimizer.fix_dimensions(dims=self.discrete_dims, values=self.discrete_values[i,:])
    290             partial_x_min[i,:] , partial_f_min[i,:] = self.mixed_optimizer.optimize(f, df, f_df)
    291 

/Users/Reinier/anaconda/lib/python3.5/site-packages/GPyOpt/optimization/acquisition_optimizer.py in fix_dimensions(self, dims, values)
    118 
    119         # -- take only the fixed components of the random samples
--> 120         self.samples = self.samples[:,np.array(self.free_dims)] # take only the component of active dims
    121         self.subspace = self.space.get_subspace(self.free_dims)
    122         self.optimizer = select_optimizer(self.optimizer_name)(Design_space(self.subspace), **self.kwargs)

IndexError: index 4 is out of bounds for axis 1 with size 4

UnboundLocalError: local variable 'x_min' referenced before assignment

Run the following code, an exception is raised.
This exception happens when all values in discrete domains are sampled.


All locations of the design space have been sampled.
---------------------------------------------------------------------------
UnboundLocalError                         Traceback (most recent call last)
<ipython-input-6-8e768ff06e92> in <module>()
----> 1 myProblem.run_optimization(max_iter)

/usr/local/lib/python3.5/site-packages/GPyOpt-1.0.2-py3.5.egg/GPyOpt/methods/bayesian_optimization.py in run_optimization(self, max_iter, max_time, eps, verbosity, save_models_parameters, report_file, evaluations_file, models_file, **kwargs)
    450                 self.acquisition_optimizer.optimizer ='CMA'
    451             print('WARNING: "acqu_optimize_method" will be deprecated in the next version!')
--> 452         super(BayesianOptimization, self).run_optimization(max_iter = max_iter, max_time = max_time,  eps = eps, verbosity=verbosity, save_models_parameters = save_models_parameters, report_file = report_file, evaluations_file= evaluations_file, models_file=models_file)
    453 
    454 

/usr/local/lib/python3.5/site-packages/GPyOpt-1.0.2-py3.5.egg/GPyOpt/core/bo.py in run_optimization(self, max_iter, max_time, eps, verbosity, save_models_parameters, report_file, evaluations_file, models_file)
    104 
    105             # --- Update and optimize acquisition and compute the exploration level in the next evaluation
--> 106             self.suggested_sample = self._compute_next_evaluations()
    107 
    108             if not ((self.num_acquisitions < self.max_iter) and (self._distance_last_evaluations() > self.eps)):

/usr/local/lib/python3.5/site-packages/GPyOpt-1.0.2-py3.5.egg/GPyOpt/core/bo.py in _compute_next_evaluations(self)
    182         Computes the location of the new evaluation (optimizes the acquisition in the standard case).
    183         """
--> 184         return self.evaluator.compute_batch()
    185 
    186     def _update_model(self):

/usr/local/lib/python3.5/site-packages/GPyOpt-1.0.2-py3.5.egg/GPyOpt/core/evaluators/sequential.py in compute_batch(self)
     19         Selects the new location to evaluate the objective.
     20         """
---> 21         return self.acquisition.optimize()
     22 
     23 

/usr/local/lib/python3.5/site-packages/GPyOpt-1.0.2-py3.5.egg/GPyOpt/acquisitions/base.py in optimize(self)
     57             out = self.optimizer.optimize(f=self.acquisition_function)[0]
     58         else:
---> 59             out = self.optimizer.optimize(f=self.acquisition_function, f_df=self.acquisition_function_withGradients)[0]
     60         return out
     61 

/usr/local/lib/python3.5/site-packages/GPyOpt-1.0.2-py3.5.egg/GPyOpt/optimization/acquisition_optimizer.py in optimize(self, f, df, f_df)
    238             #break
    239 
--> 240         self.pulled_arms = np.vstack((self.pulled_arms, x_min))
    241 
    242 

UnboundLocalError: local variable 'x_min' referenced before assignment

import matplotlib
matplotlib.use('Agg')
import GPy
import GPyOpt
from numpy.random import seed

def myf(x):
    return (2*x)**2

bounds = [{'name': 'var_1', 'type': 'discrete', 'domain': (-5, -4, -3, -2, -1, 0, 1, 2, 3, 4, 5)}]

max_iter = 10

myProblem = GPyOpt.methods.BayesianOptimization(myf, bounds, exact_feval=True)

myProblem.run_optimization(max_iter)

Use of GPyOpt for Bayesian Uncertainty Quantification of differential equations

To compare GPyOpt with other contributions in uncertainty quantification, do you have examples or tutorials for the following models: Lorenz63 (dim=3), Lorenz96(dim=9) and JAK-STAT as used by Campbell, Chkrebtii, Calderhead and Girolami in https://arxiv.org/abs/1306.2365 and Hennig and Kersting? Many of these authors will participate to the forthcoming GP Summer School.

These examples could be rolled into the prototype templates of GPyOpt. The strength of GPyOPT is the unifying use of python whereas the papers mentioned above are all on different software.

Similarity kernel parameter selection

Assume I want to use some similarity kernel with parameter (e.g. exponent of sq. L2 distance divided by sigma). Do I need to specify explicitly the parameter or is the parameter inferred during the optimization process by GPyOpt, e.g. selecting one which gives best cross validation value for error of mean function on available observations?

Optimization of own 2d functions: assertion error

Hi all,
I want to use GPyOpt to optimize a 2d function that I defined myself.
Here is what I get. Thanks in advance.

def f_sim(x):
    # takes its minimum value at [0,0]
    return(x[1]**2+x[0]**4)
bounds = [(-2, 2), (-1, 1)]

# Creates three identical objects that we will later use to compare the optimization strategies 
myBopt2D = GPyOpt.methods.BayesianOptimization(f_sim,
                                              bounds=bounds,
                                              acquisition='LCB',   # Lower confidence bound method 
                                              acquisition_par = 2, # Set parameter psi=2 
                                              normalize = True)    # Normalize the acquisition funtction

---------------------------------------------------------------------------
AssertionError                            Traceback (most recent call last)
<ipython-input-100-699f9af9accc> in <module>()
      4                                               acquisition='LCB',   # Lower confidence bound method
      5                                               acquisition_par = 2, # Set parameter psi=2
----> 6                                               normalize = True)    # Normalize the acquisition funtction

/usr/local/lib/python2.7/dist-packages/GPyOpt/methods/bayesian_optimization.pyc in __init__(self, f, bounds, kernel, X, Y, numdata_inital_design, type_initial_design, model_optimize_interval, acquisition, acquisition_par, model_optimize_restarts, sparseGP, num_inducing, normalize, exact_feval, verbosity)
     76         else:
     77             self.kernel = kernel
---> 78         self._init_model()
     79         self.first_time_optimization = True  # control over the optimization of the GP
     80 

/usr/local/lib/python2.7/dist-packages/GPyOpt/methods/bayesian_optimization.pyc in _init_model(self)
    116                 self.model = GPy.models.SparseGPRegression(self.X, self.Y, kernel=self.kernel, num_inducing=self.num_inducing)
    117         else:
--> 118             self.model = GPy.models.GPRegression(self.X,self.Y,kernel=self.kernel)
    119 
    120         if self.exact_feval == True:

/usr/local/lib/python2.7/dist-packages/GPy/core/parameterization/parameterized.pyc in __call__(self, *args, **kw)
     17         self._in_init_ = True
     18         #import ipdb;ipdb.set_trace()
---> 19         self = super(ParametersChangedMeta, self).__call__(*args, **kw)
     20         logger.debug("finished init")
     21         self._in_init_ = False

/usr/local/lib/python2.7/dist-packages/GPy/models/gp_regression.pyc in __init__(self, X, Y, kernel, Y_metadata, normalizer, noise_var, mean_function)
     34         likelihood = likelihoods.Gaussian(variance=noise_var)
     35 
---> 36         super(GPRegression, self).__init__(X, Y, kernel, likelihood, name='GP regression', Y_metadata=Y_metadata, normalizer=normalizer, mean_function=mean_function)
     37 

/usr/local/lib/python2.7/dist-packages/GPy/core/gp.pyc in __init__(self, X, Y, kernel, likelihood, mean_function, inference_method, name, Y_metadata, normalizer)
     47         self.num_data, self.input_dim = self.X.shape
     48 
---> 49         assert Y.ndim == 2
     50         logger.info("initializing Y")
     51 

AssertionError:

ImportError: No module named 'core'

I tried to run the first example from the manual:
http://nbviewer.ipython.org/github/SheffieldML/GPyOpt/blob/master/manual/GPyOpt_reference_manual.ipynb

The line:

import GPyOpt

fails with the following error:

ImportError: No module named 'core'

Happens on Python 3.4 / Windows 8 x64.

I think the problem is Python 3.x, but I cannot use another version because of other packages... any way to fix this?

_update_model with one data point

In line 194 of GPyOpt/core/bo.py
self.model.updateModel(self.X,(self.Y-self.Y.mean())/(self.Y.std()),self.suggested_sample,self.Y_new)

You might want to check for self.Y having >1 data points to work with. I was just doing a single random run (i.e. initial_design_numdata=1) before trying to call run_optimization, and it crashes without any error message on that line because the standard deviation is 0.

Error in parallel computation

As long as I setup batch_size>1, the model cannot even be built. I am looking for argmin(Y) of a XGBoost Model.

What is the problem?

Works:

batch_size = 1
num_cores = 4

Does NOT Work:

batch_size = 3
num_cores = 1

Model building:

opt = GPyOpt.methods.BayesianOptimization(f = model.f,  
                                          domain = domain_norm,                  
                                          normalize_Y = True,
                                          model_type= 'GP',
                                          evaluator_type = 'local_penalization',
                                          batch_size = batch_size,
                                          num_cores = num_cores,
                                          acquisition_type = 'EI',              
                                          acquisition_jitter = 0.25)

When batch_size>1, model building hangs there forever, when I force to break it down, the error shows up:

Error in parallel computation. Fall back to single process!

Dimension bug in acquisition_optimizer

In GPyOpt_parallel_optimization.ipynb, I changed the var_1 from continuous to discrete and got the following error.

It seems there is an dimension bug somewhere.

I am using numpy==1.11.0, and latest GPyOpt release.

This is my setup (the only thing that I've changed):
domain = [{'name': 'var_1', 'type': 'discrete', 'domain': (-3,-2,0,1,2,3,4,5,6,7,8,10)}, {'name': 'var_2', 'type': 'continuous', 'domain': bounds[1]}]
This is the error message:
GPyOpt/optimization/acquisition_optimizer.py, line 120, in fix_dimensions self.samples = self.samples[:,np.array(self.free_dims)] # take only the component of active dims IndexError: index 1 is out of bounds for axis 1 with size 1

Mixing batch_size and num_cores

When creating an instance of BayesianOptimization with a batch size or number of cores bigger than 1, both values seems to be mixed up:
Especially the batch_size will be used as the number of core. The problem comes from the file methods/bayesian_optimization.py l.174:
self.objective = SingleObjective(self.f, self.batch_size, self.num_cores,self.objective_name

This does not corresponds to the init function of SingleObjective (core/task/objective.py l.36):
def __init__(self, func, num_cores = 1, objective_name = 'no_name', batch_type = 'synchronous', unfold_args=False, space = None)

Is it a bug? I'm new to GPyOpt so there might be something I'm missing...

What approach is used for Bayesian Optimization in GPyOpt?

Did you build this code based on some specific paper / papers? If so, which ones?

Bayesian updating of hyperparameters

Dear Developers,
We were able to customise our kernel and BO.model, and used it with run_optimisation. The BO procedure works well, but we noticed that if we update our model with each acquisition, this also updates the hyperparameters of the GP process (maximum likelihood) and often results in overfitting. Increasing model update interval was not a good idea either. We cannot find how to a) impose upper/lower numerical bounds on the hyperparameter search and b) refit the model with each acquisitions but update the model hyperparameters more slowly. Ideally, we would use Bayesian updating of hyperparameters (with priors on the hyperparameters), is this available? This is currently the biggest issue for us.
Many thanks for your assistance, Milica.

Gaussian Process defaults to constant mean and variance throughout the domain

This refers to the devel branch. Couldn't test this on the master.

I have a 2D optimization problem. Optimize x,y in [0,2000000] over a complicated functions. After 25 runs GPyOpt stops as having found the "optimum". Now because I optimized this function in other ways I know that not to be true. It seems like the problem is that the optimizer stops too early (25 runs it's just too few) and when I plot the acquisition function I get this very weird and completely wrong posterior:

This is in spite of the fact that the function is not flat at all. The 25 runs have these inputs:

[[  554022.18082771   837330.24086706]
 [  603955.26174629   913575.22736361]
 [  871682.61154303  1208867.22113813]
 [  333424.86148015  1610445.10644461]
 [ 1820857.74113731   729840.13906509]
 [ 1581724.07500661  1087447.33561535]
 [ 1320236.72084362   399413.66943186]
 [ 1252644.39261987   978566.35132267]
 [   41922.88862248   875342.96660996]
 [  959290.22350186  1563699.95650388]
 [   37187.69129177  1120012.7272474 ]
 [ 1436628.17206779   410814.1332408 ]
 [ 1732309.40510174  1813431.75106976]
 [  529015.44987526   692654.54154614]
 [  730317.15687086   471378.82506464]
 [  700758.77014555    32881.55202727]
 [  610413.30341607   717372.31286463]
 [  136686.49824964  1833799.95991568]
 [  461725.79515407   230566.15417022]
 [ 1524996.01465518   248755.38929407]
 [  147287.30588703    99918.38886542]
 [  390810.58866436  1606621.28275969]
 [ 1507735.40298871   803206.42371126]
 [   62073.3998766   1442550.39300023]
 [  344237.59736354   955076.55786266]
 [  344237.59736354   955076.55786266]]

And produce a large range of outputs.

[[ -5444604.30579861]
 [ -4846435.10319802]
 [ -3813369.21313364]
 [-10871314.33351388]
 [ -4977461.22761602]
 [ -4027234.58931002]
 [ -5891597.80309109]
 [ -4270727.32179308]
 [ -5694584.23650084]
 [ -3711169.85177035]
 [ -5606411.28383687]
 [ -5925348.52915016]
 [ -3583504.2144228 ]
 [ -6236937.05403522]
 [ -8076707.63590015]
 [ -5633996.63497615]
 [ -5932892.03629752]
 [ -7612143.30723312]
 [ -6366437.91667295]
 [ -3475903.3506581 ]
 [ -6581225.39228972]
 [ -6512401.53862414]
 [ -4722818.19982356]
 [ -6185242.92427387]
 [ -9579614.18637638]
 [-10343506.13677566]]

You should be able to replicate the same wrong posterior by inputting those numbers and calling:

    optimization = GPyOpt.methods.BayesianOptimization(experiment,
                                                       bounds,
                                                       X=initial_x,
                                                       Y=initial_y
                                                       )

Changing normalization, acquisition functions or models have no effect.

No module named task.cost and a question about bayesian_optimization.py(in methods)

Hi!
I have two questions and need help.

I have already successfully installed and used GPyOpt-master. Now I wanted to switch to
GPyOpt-devel. However, I am facing with the following error, although I have installed GPyOpt-devel:

ImportError: No module named task.cost

The second question is about docstring in bayesian_optimization.py (in methods of GPyOpt)

"""Main class to initialize a Bayesian Optimization method.
:param f: function to optimize. It should take 2-dimentional numpy arrays as input and return 2-dimentional outputs (one evaluation per row)."""

I have a numpy array as my input and also a corresponding function evaluation (also in numpy). However, I am confused to define f in this way. Could you please help how can I define my f.
In the tutorial notebooks there are some analytical functions for illustration purpose and not real data as my case.

Thanks in Advance for your help!

devel branch: some errors and their apparent fixes

bayesinan_optimization.py:
import deepgp seems to be unused and causes import error on my machine. So far did not experience any problems from commenting it.
ModularBayesianOptimization class initialization does not set initial_iter variable, this causes error in GPyOpt_constrained_optimization.ipynb. With these fixed GPyOpt_constrained_optimization.ipynb seemed to work fine.

Imaging updated model with each acquisition point

Dear Developers,
When we run_optimisation, we cannot seem to image how the model evolves with each acquisition, we only see the end result. We found a notebook where the BO is run in a loop with max_iter=1; this of course illustrates the evolution of the model, but is the loop numerically equivalent to doing continuous BO (i.e. are the hyperparameters re-initialised with max_iter=1)? Ideally we would run BO continuously for max_iter=N steps, how would we plot at each step?
Many thanks, Milica.

Handling underflow in kernels

Hi all,

I've been trying to use your package for hyperparameter optimization. It, like scikit-learn's Gaussian Processes, often underflows when the square exponential kernel is used. If model_type = 'GP' and acquisition_type is either expected improvement or maximum probability of improvement, I get the error:

  File "/a/python3.5/site-packages/GPyOpt/acquisitions/MPI.py", line 38, in _compute_acq
    _, Phi, _ = get_quantiles(self.jitter, fmin, m, s)    
  File "/a/python3.5/site-packages/GPyOpt/util/general.py", line 138, in get_quantiles
    phi = np.exp(-0.5 * u**2) / np.sqrt(2*np.pi)

The relevant code snippet is:

opt_kwargs = {
          'f': self._objective_func,
          'domain': self._space,
          'normalize_Y': False,
          'batch_size': 1,
          'model_type': 'GP',
          'acquisition_type': 'EI',
          'num_cores': 1,
          'exact_feval': False,
          'initial_design_numdata': self.NUM_RANDOM_INIT_POINTS,
}

And printing X, Y, and space gives

[[0.03278289121021576, 0.9223270601253915, 0.041078090660775515, 1, 1, 1, 1, 2, 1, 0, 0], [0.0459930266775439, 0.5466045681486984, 0.006978856119194612, 0, 0, 0, 2, 2, 1, 2, 1], [0.04378915391665336, 0.578530090404731, 0.09136408190257667, 0, 1, 0, 0, 0, 1, 0, 0], [0.01803270069606775, 0.6932997271162962, 0.08527496299747057, 0, 1, 1, 0, 2, 0, 2, 1], [0.027558962781239666, 0.8592069274365637, 0.034190797138337045, 0, 1, 1, 2, 1, 0, 1, 0], [0.015420387491286701, 0.5956928159271585, 0.04633636991599571, 0, 1, 0, 0, 1, 1, 2, 0], [0.00925519288002172, 0.604910074085572, 0.08911441982609125, 0, 1, 1, 1, 1, 0, 1, 1], [0.04860391821550771, 0.6562647739938442, 0.004181031065854369, 1, 1, 1, 0, 0, 1, 2, 1], [0.012232522924794003, 0.8889700787725456, 0.06011932984882709, 0, 0, 1, 2, 0, 1, 2, 0], [0.007140128397424967, 0.7054756177189522, 0.022160119641586238, 1, 0, 1, 2, 0, 1, 2, 0], [0.003325095442111388, 0.7258961266957953, 0.09103359760065749, 1, 1, 0, 2, 0, 0, 0, 1], [0.021410765938286835, 0.7598611438932474, 0.07642207817613791, 1, 0, 1, 0, 0, 0, 0, 1]] 
[[-0.09225001256533977], [-0.6108390574677939], [-0.49642830217129075], [-0.14971756743156198], [-0.07259310916767191], [-0.6389030419685992], [-0.1849272466827503], [-0.8354279891304349], [-0.07268078200483091], [-0.07268078200483091], [-0.46476992862887007], [-0.07259310916767191]] 
[{'name': 'penalty', 'type': 'continuous', 'domain': (5e-08, 0.05)}, {'name': 'rate_decay', 'type': 'continuous', 'domain': (0.5, 0.99)}, {'name': 'learning_rate', 'type': 'continuous', 'domain': (0.001, 0.1)}, {'name': 'contraction_type', 'type': 'discrete', 'domain': (0, 1)}, {'name': 'weight_init', 'type': 'discrete', 'domain': (0, 1)}, {'name': 'optimizer', 'type': 'discrete', 'domain': (0, 1)}, {'name': 'log2_layer_seq', 'type': 'discrete', 'domain': (0, 1, 2)}, {'name': 'log2_init_fmap', 'type': 'discrete', 'domain': (0, 1, 2)}, {'name': 'log2_fmap_mult', 'type': 'discrete', 'domain': (0, 1)}, {'name': 'log2_batch_size', 'type': 'discrete', 'domain': (0, 1, 2)}, {'name': 'weight_reg', 'type': 'discrete', 'domain': (0, 1)}]

Scaling all values between 0 and 1 doesn't help because of the variance normalization term in the kernel. Perhaps kernel calculations would be more stable in the log space? Anyway, it's not clear how to address this.

Thank you for your time,
Sean

Treat each iteration of BO as a time-step in GPyOpt

Hi!
I am trying to treat each iteration of BO as a certain time-step. In other words, would it be possible to enforce the BO to evaluate the objective function at constrained time-step? It seems dealing the
time-step as an additional input is a sensible choice, but how can I proceed in GPyOpt after considering the additional input in my kernel? Is there any need to change the formulation of EI?
Thanks for help!

GP_MCMC and EI_MCMC and integrating hyperparameters

Dear Javier!
I want to solve an optimization problem and "concurrently" optimize the hyperparameters of covariance function. In other words, we are interested to initialize the hyperparametes at very first step by some values, then improve them at each iteration. (there is some reason for this simultaneous goal).
My question is does the GP_MCMC and EI_MCMC is fit for this goal? I see a notebook is available entitling "GPy_Opt_integrating_model_hyperparameters" however, I am not sure if that is similar to my case (sorry I am very new to this statistical machine learning concept).

Thanks for your help!

Strange behavior of plot_acquisition()

Hi!
Sometime GPyOpt suggests the sample which is not the maximum of acquisition function. It does not make scene to me. Furthermore, in the following figure, the uncertainty around 0.5 is almost zero, why at that point we see the acquisition is maximized?

OR this figure:

begginer_optimization_1d.py demo appears not to be working with latest GPy and GPyOpt

When I try to run this demo with the latest github version of GPyOpt (45788ec) and GPy (6a5c2ea1) using the following code

%pylab inline
import GPyOpt
BO_demo_1d = GPyOpt.demos.begginer_optimization_1d()

I get the traceback given below. Seems that GPyOpt or GPy and out of sync somehow. Any suggestions on how to get this to work would be much appreciated as this looks like a very nice package.

Populating the interactive namespace from numpy and matplotlib
-----
----- Running demo. It may take a few seconds.
-----
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-1-23845951051f> in <module>()
      1 get_ipython().magic(u'pylab inline')
      2 import GPyOpt
----> 3 BO_demo_1d = GPyOpt.demos.begginer_optimization_1d()

/Users/jameskermode/lib/python/GPyOpt-0.1.4-py2.7.egg/GPyOpt/demos/begginer_optimization_1d.pyc in begginer_optimization_1d(plots)
     47     # Run the optimization
     48     BO_demo_1d.run_optimization(max_iter,                                   # evaluation budget
---> 49                                     eps=10e-6)                              # stop criterion
     50 
     51 

/Users/jameskermode/lib/python/GPyOpt-0.1.4-py2.7.egg/GPyOpt/core/bo.pyc in run_optimization(self, max_iter, n_inbatch, acqu_optimize_method, acqu_optimize_restarts, batch_method, eps, n_procs, true_gradients, verbose)
    118             # --- Update model
    119             try:
--> 120                 self._update_model()
    121             except np.linalg.linalg.LinAlgError:
    122                 break

/Users/jameskermode/lib/python/GPyOpt-0.1.4-py2.7.egg/GPyOpt/core/bo.pyc in _update_model(self)
    212 
    213         # ------- Optimize acquisition function
--> 214         self.suggested_sample = self._optimize_acquisition()
    215 
    216 

/Users/jameskermode/lib/python/GPyOpt-0.1.4-py2.7.egg/GPyOpt/core/bo.pyc in _optimize_acquisition(self)
    187         # ------ Selection of the batch method (if any, predictive used when n_inbathc=1)
    188         if self.batch_method == 'predictive':
--> 189             X_batch = predictive_batch_optimization(acqu_name, acquisition_par, acquisition, d_acquisition, bounds, acqu_optimize_restarts, acqu_optimize_method, model, n_inbatch)
    190         elif self.batch_method == 'lp':
    191             X_batch = lp_batch_optimization(self.acquisition_func, bounds, acqu_optimize_restarts, acqu_optimize_method, model, n_inbatch)

/Users/jameskermode/lib/python/GPyOpt-0.1.4-py2.7.egg/GPyOpt/core/optimization.pyc in predictive_batch_optimization(acqu_name, acquisition_par, acquisition, d_acquisition, bounds, acqu_optimize_restarts, acqu_optimize_method, model, n_inbatch)
     35 
     36     # Optimization of the first element in the batch
---> 37     X_new = optimize_acquisition(acquisition, d_acquisition, bounds, acqu_optimize_restarts, acqu_optimize_method, model, X_batch=None, L=None, Min=None)
     38     X_batch = reshape(X_new,input_dim)
     39     k=1

/Users/jameskermode/lib/python/GPyOpt-0.1.4-py2.7.egg/GPyOpt/core/optimization.pyc in optimize_acquisition(acquisition, d_acquisition, bounds, acqu_optimize_restarts, acqu_optimize_method, model, X_batch, L, Min)
    140         res =  fast_acquisition_optimization(acquisition, d_acquisition, bounds,acqu_optimize_restarts, model, 'brute', X_batch, L, Min)
    141     elif acqu_optimize_method=='fast_random':
--> 142         res =  fast_acquisition_optimization(acquisition, d_acquisition, bounds,acqu_optimize_restarts, model, 'random', X_batch, L, Min)
    143     elif acqu_optimize_method=='DIRECT':
    144         res = wrapper_DIRECT(acquisition,bounds)

/Users/jameskermode/lib/python/GPyOpt-0.1.4-py2.7.egg/GPyOpt/core/optimization.pyc in fast_acquisition_optimization(acquisition, d_acquisition, bounds, acqu_optimize_restarts, model, method_type, X_batch, L, Min)
    155     else:
    156         samples = multigrid(bounds, acqu_optimize_restarts)
--> 157     pred_samples = acquisition(samples)
    158     x0 =  samples[np.argmin(pred_samples)]
    159     best_x,_ = wrapper_lbfgsb(acquisition,d_acquisition,x0 = np.array(x0),bounds=bounds)

/Users/jameskermode/lib/python/GPyOpt-0.1.4-py2.7.egg/GPyOpt/core/acquisition.pyc in acquisition_function(self, x)
     36         Expected Improvement
     37         """
---> 38         m, s, fmin = get_moments(self.model, x)
     39         phi, Phi, _ = get_quantiles(self.acquisition_par, fmin, m, s)
     40         f_acqu = (fmin - m + self.acquisition_par) * Phi + s * phi

/Users/jameskermode/lib/python/GPyOpt-0.1.4-py2.7.egg/GPyOpt/util/general.pyc in get_moments(model, x)
     62     input_dim = model.X.shape[1]
     63     x = reshape(x,input_dim)
---> 64     fmin = min(model.predict(model.X)[0])
     65     m, v = model.predict(x)
     66     s = np.sqrt(np.clip(v, 0, np.inf))

/Users/jameskermode/lib/python/GPy-0.8.4-py2.7-macosx-10.10-x86_64.egg/GPy/core/gp.pyc in predict(self, Xnew, full_cov, Y_metadata, kern)
    263         """
    264         #predict the latent function values
--> 265         mu, var = self._raw_predict(Xnew, full_cov=full_cov, kern=kern)
    266         if self.normalizer is not None:
    267             mu, var = self.normalizer.inverse_mean(mu), self.normalizer.inverse_variance(var)

/Users/jameskermode/lib/python/GPy-0.8.4-py2.7-macosx-10.10-x86_64.egg/GPy/core/gp.pyc in _raw_predict(self, Xnew, full_cov, kern)
    213 
    214         Kx = kern.K(self._predictive_variable, Xnew)
--> 215         mu = np.dot(Kx.T, self.posterior.woodbury_vector)
    216         if len(mu.shape)==1:
    217             mu = mu.reshape(-1,1)

ValueError: shapes (4,3) and (4,1) not aligned: 3 (dim 1) != 4 (dim 0)

Periodic Boundary Conditions in BO.model

Dear Developers,
The model functions we seek to optimise with BO are known to have periodic boundary conditions (PBC), and introducing this feature into the model has previously saved us excessive sampling at the boundaries. We are now working with a simple RBF kernel in our BO.model, should we switch to the periodic-exponential to introduce PBC?
Many thanks, Milica.

Noiseless MPI assigns probability 1. to lowest sampled point.

I think there's a bug in the MPI and EI acquisition functions:

The green line is GPyOpt's MPI and EI acquisition function (with jitter=0.01 and 0.5 respectively) and the blue is when I write down the equations myself.

Since the model is noiseless the probability of sampling a value lower by a non-zero margin (jitter) at the current lowest point should be 0.

The problem is that jitter is added rather than subtracted here https://github.com/SheffieldML/GPyOpt/blob/master/GPyOpt/util/general.py#L137.
So instead of calculating p(f(x) + jitter < f(x+)) you're calculating p(f(x) - jitter < f(x+)).

The unfortunate effect is that a simple function such as 2*x^2 can't be optimized:

After the two initial samples (which are made without use of the acquisition function), the optimization stops because the MPI acquisition function is max at a point that is already sampled.

Best, Rasmus.

Python 3 compatibility

Although the end of lifetime of Python 2.7 is 2020, it is important to be compatible with Python 3 in order to allow projects depending on GPyOpt to upgrade to Python 3. With the help of from __future__ import print_function, absolute_import, division as well as the six package it is possible with minimal effort to make the package python 2/3 compatible.

Broken link - Tutorial Notebook

Hi,

The link for the Tutorial Notebook in the README.md file is broken (the same link exists in manual/Readme.md and here).

Maybe it should be changed to this one?

Cheers!

Nikitas

Variable names within function

I'd like to be able to use GPyOpt to model accuracy as a function of hyper-parameters for a machine learning algorithm run as a separate program through subprocess. To do so, I need access to the variables names in addition to their suggested values so that I can provide them unambiguously to the program. I can't figure out a good way to do this because my function that takes the suggested variable settings, runs the program, and returns the result cannot seem to see any outside variables, even when specified as global. Do you have any advice for how I might be able to make this work? Thanks!

Batch design with LOCAL PENELAZATION and non-myopic Bayesian Optimization

Hi,

I have a general question. Dose the results of "local penalization" in bach design is "one ofter the other"? I mean does the "sequence" mater? What is the difference between "local penalization" and the algorithm called GLASSES in non-myopic Bayesian Optimization Paper? It seems in non-myopic Bayesian Optimization the sequences is matter. Is there any implementation for that?

Thanks in advance!

ImportError: cannot import name grid

I am facing "ImportError: cannot import name grid" when trying to install the library on ubuntu.

Maximization rather than Minimization with GPyOpt

I want to solve a maximization problem with GPyOpt, but I do not know what changes are needed for this goal? If we have an analytic function like f converting f to (-1)*f will work, however, when we have only some observations of f, how we should deal with this problem?

Improve the output of GPyOpt

To improve and generalize the way of the output is generated in GPyOpt.

Design an abstract class for representing the output of BO process.
Unify the code for generating output files for different interfaces.

initial_design_numdata ignored when X, Y are set

Hi there,

First off: great job devs. This is a really nice package.

Second, if I initialize a GPyOpt.methods.BayesianOptimization object with some initial data points using args X and Y, but I also set initial_design_numdata to a positive value, it seems like the optimizer should still sample the latter number of points randomly and fit the values from X and Y, rather than simply ignoring initial_design_numdata. Consider the following:

initial_design_numdata currently defaults to None. This seems like a value that might specify "no initial points if X and Y are set, otherwise use some constant value." If the behaviour is to always ignore initial_design_numdata in the face of X and Y, it might as well default to that constant value (5) straight away.
Given that it is highly likely that the objective function takes a long time to calculate, it's a real possibility that initial sampling could have been cut short then restarted. As it stands, there's no way to resume random sampling if there are too few samples to build a half-decent initial model from. Concretely, I have sampled 2 points randomly which happen to be close together and I get underflow errors if I try initializing the model with just those two points.

Thanks very much for your time,
Sean

Context Dependency Optimization and GPyOpt

Hi,
In some optimization problem, some variable is not an optimization variable,
but something that is fixed by the environment.In the literature, this is
refereed to as context. One way to handle context is to define an
additional input on kernel. Then, we fix the context and optimize the acquisition over the
remaining optimization variable. Does the GPyOpt can deal with the contexts?
If not, can someone please give some hints to implement this idea?
I want to implement something like this:

bounds = [(bounds of opt_variable),(bounds of context)]
k_parameters = GPy.kern.RBF(1, 1, 1, active_dims=[0])
k_context = GPy.kern.RBF(1, 1, 1, active_dims=[1])
kern = k_optimization_variable * k_context
GPyOpt.methods.BayesianOptimization(...,active_dims = [0])

In other words, how can I fix the context and optimize the acquisition only over optimization variable?

Thanks for your help in advance!

No module named task.cost

Hi everyone!
I still have the problem to use the new version of GPyOpt with the following error:

import GPyOpt
Traceback (most recent call last):
File "", line 1, in
File "build/bdist.macosx-10.5-x86_64/egg/GPyOpt/init.py", line 4, in

File "build/bdist.macosx-10.5-x86_64/egg/GPyOpt/core/init.py", line 4, in

File "build/bdist.macosx-10.5-x86_64/egg/GPyOpt/core/bo.py", line 7, in
ImportError: No module named task.cost

Any idea to resolve this problem?
THANKS

save_models_parameters for GPModel_MCMC

In line 64 of GPyOpt/core/bo.py,
if not isinstance(self.model, GPyOpt.models.GPModel) or isinstance(self.model, GPyOpt.models.GPModel_MCMC):

Based on the subsequent error message, I'm guessing it should be,
if not (isinstance(self.model, GPyOpt.models.GPModel) or isinstance(self.model, GPyOpt.models.GPModel_MCMC)):

setting initial_design_numdata = 0 in BayesianOptimization method result in exception

According to the docstr for initial_design_numdata
:initial_design_numdata: number of initial points that are collected jointly before start running the optimization.
So if I set initial design numdata to 0, my assumption for bayesian optimization is that it will not sample datas before optimization. However, it triggers an exception.

Currently, initial_design_numdata defaults to "None", but it still collects 5 samples. I am not sure if this is expected.

def myfunc(x):
    print(x)
    return np.sin(x) * np.exp(-x)

bounds = [
    {'name': 'x', 'type': 'discrete', 'domain': np.arange(-100, 100, 0.01)},
]

myProblem = GPyOpt.methods.BayesianOptimization(
    myfunc, 
    bounds, 
    exact_feval = True,
    num_cores = 8,
    normalize_Y = False,
    maximize = True,
    initial_design_numdata = 0
)

---------------------------------------------------------------------------
error                                     Traceback (most recent call last)
<ipython-input-7-dc6bf1db909d> in <module>()
      6     normalize_Y = False,
      7     maximize = True,
----> 8     initial_design_numdata = 0
      9 )

/usr/local/lib/python3.5/site-packages/GPyOpt-1.0.2-py3.5.egg/GPyOpt/methods/bayesian_optimization.py in __init__(self, f, domain, constrains, cost_withGradients, model_type, X, Y, initial_design_numdata, initial_design_type, acquisition_type, normalize_Y, exact_feval, acquisition_optimizer_type, model_update_interval, evaluator_type, batch_size, num_cores, verbosity, verbosity_model, bounds, maximize, **kwargs)
    236 
    237         # --- Initialize everything
--> 238         self.run_optimization(max_iter=0,verbosity=self.verbosity)
    239 
    240     def _model_chooser(self):

/usr/local/lib/python3.5/site-packages/GPyOpt-1.0.2-py3.5.egg/GPyOpt/methods/bayesian_optimization.py in run_optimization(self, max_iter, max_time, eps, verbosity, save_models_parameters, report_file, evaluations_file, models_file, **kwargs)
    451                 self.acquisition_optimizer.optimizer ='CMA'
    452             print('WARNING: "acqu_optimize_method" will be deprecated in the next version!')
--> 453         super(BayesianOptimization, self).run_optimization(max_iter = max_iter, max_time = max_time,  eps = eps, verbosity=verbosity, save_models_parameters = save_models_parameters, report_file = report_file, evaluations_file= evaluations_file, models_file=models_file)
    454 
    455     def _sign(self,f):

/usr/local/lib/python3.5/site-packages/GPyOpt-1.0.2-py3.5.egg/GPyOpt/core/bo.py in run_optimization(self, max_iter, max_time, eps, verbosity, save_models_parameters, report_file, evaluations_file, models_file)
    100             try:
    101                 self._update_pulled_arms() # only used in bandit problems
--> 102                 self._update_model()
    103             except np.linalg.linalg.LinAlgError:
    104                 break

/usr/local/lib/python3.5/site-packages/GPyOpt-1.0.2-py3.5.egg/GPyOpt/core/bo.py in _update_model(self)
    193                 self.model.updateModel(self.X,(self.Y-self.Y.mean())/(self.Y.std()),self.suggested_sample,self.Y_new)
    194             else:
--> 195                 self.model.updateModel(self.X, self.Y,self.suggested_sample,self.Y_new)
    196 
    197         self._save_model_parameter_values()

/usr/local/lib/python3.5/site-packages/GPyOpt-1.0.2-py3.5.egg/GPyOpt/models/gpmodel.py in updateModel(self, X_all, Y_all, X_new, Y_new)
     79         if self.normalize_Y:
     80             Y_all = (Y_all - Y_all.mean())/(Y_all.std())
---> 81         if self.model is None: self._create_model(X_all, Y_all)
     82         else:
     83             self.model.set_XY(X_all, Y_all)

/usr/local/lib/python3.5/site-packages/GPyOpt-1.0.2-py3.5.egg/GPyOpt/models/gpmodel.py in _create_model(self, X, Y)
     62 
     63         if not self.sparse:
---> 64             self.model = GPy.models.GPRegression(X, Y, kernel=kern, noise_var=noise_var)
     65         else:
     66             self.model = GPy.models.SparseGPRegression(X, Y, kernel=kern, num_inducing=self.num_inducing)

/usr/local/lib/python3.5/site-packages/paramz/parameterized.py in __call__(self, *args, **kw)
     52         self._model_initialized_ = initialize
     53         if initialize:
---> 54             self.initialize_parameter()
     55         else:
     56             import warnings

/usr/local/lib/python3.5/site-packages/paramz/core/parameter_core.py in initialize_parameter(self)
    328         self._highest_parent_._connect_fixes()
    329         self._highest_parent_._connect_parameters() #logger.debug("calling parameters changed")
--> 330         self.parameters_changed()
    331 
    332     @property

/usr/local/lib/python3.5/site-packages/GPy/core/gp.py in parameters_changed(self)
    190             this method yourself, there may be unexpected consequences.
    191         """
--> 192         self.posterior, self._log_marginal_likelihood, self.grad_dict = self.inference_method.inference(self.kern, self.X, self.likelihood, self.Y_normalized, self.mean_function, self.Y_metadata)
    193         self.likelihood.update_gradients(self.grad_dict['dL_dthetaL'])
    194         self.kern.update_gradients_full(self.grad_dict['dL_dK'], self.X)

/usr/local/lib/python3.5/site-packages/GPy/inference/latent_function_inference/exact_gaussian_inference.py in inference(self, kern, X, likelihood, Y, mean_function, Y_metadata, K, precision, Z_tilde)
     38 
     39         if K is None:
---> 40             K = kern.K(X)
     41 
     42         Ky = K.copy()

/usr/local/lib/python3.5/site-packages/GPy/kern/src/kernel_slice_operations.py in wrap(self, X, X2, *a, **kw)
     77     def wrap(self, X, X2 = None, *a, **kw):
     78         with _Slice_wrap(self, X, X2) as s:
---> 79             ret = f(self, s.X, s.X2, *a, **kw)
     80         return ret
     81     return wrap

<decorator-gen-144> in K(self, X, X2)

/usr/local/lib/python3.5/site-packages/paramz/caching.py in g(obj, *args, **kw)
    280             else:
    281                 cacher = cache[self.f] = Cacher(self.f, self.limit, self.ignore_args, self.force_kwargs, cacher_enabled=cache.caching_enabled)
--> 282             return cacher(*args, **kw)
    283         g.__name__ = f.__name__
    284         g.__doc__ = f.__doc__

/usr/local/lib/python3.5/site-packages/paramz/caching.py in __call__(self, *args, **kw)
    176             # If we need to compute, we compute the operation, but fail gracefully, if the operation has an error:
    177             try:
--> 178                 new_output = self.operation(*args, **kw)
    179             except:
    180                 self.reset()

/usr/local/lib/python3.5/site-packages/GPy/kern/src/stationary.py in K(self, X, X2)
     95         K(X, X2) = K_of_r((X-X2)**2)
     96         """
---> 97         r = self._scaled_dist(X, X2)
     98         return self.K_of_r(r)
     99 

<decorator-gen-147> in _scaled_dist(self, X, X2)

/usr/local/lib/python3.5/site-packages/paramz/caching.py in g(obj, *args, **kw)
    280             else:
    281                 cacher = cache[self.f] = Cacher(self.f, self.limit, self.ignore_args, self.force_kwargs, cacher_enabled=cache.caching_enabled)
--> 282             return cacher(*args, **kw)
    283         g.__name__ = f.__name__
    284         g.__doc__ = f.__doc__

/usr/local/lib/python3.5/site-packages/paramz/caching.py in __call__(self, *args, **kw)
    176             # If we need to compute, we compute the operation, but fail gracefully, if the operation has an error:
    177             try:
--> 178                 new_output = self.operation(*args, **kw)
    179             except:
    180                 self.reset()

/usr/local/lib/python3.5/site-packages/GPy/kern/src/stationary.py in _scaled_dist(self, X, X2)
    149             return self._unscaled_dist(X/self.lengthscale, X2)
    150         else:
--> 151             return self._unscaled_dist(X, X2)/self.lengthscale
    152 
    153     def Kdiag(self, X):

/usr/local/lib/python3.5/site-packages/GPy/kern/src/stationary.py in _unscaled_dist(self, X, X2)
    119         if X2 is None:
    120             Xsq = np.sum(np.square(X),1)
--> 121             r2 = -2.*tdot(X) + (Xsq[:,None] + Xsq[None,:])
    122             util.diag.view(r2)[:,]= 0. # force diagnoal to be zero: sometime numerically a little negative
    123             r2 = np.clip(r2, 0, np.inf)

/usr/local/lib/python3.5/site-packages/GPy/util/linalg.py in tdot(*args, **kwargs)
    322 
    323 def tdot(*args, **kwargs):
--> 324     return tdot_blas(*args, **kwargs)
    325 
    326 def DSYR_blas(A, x, alpha=1.):

/usr/local/lib/python3.5/site-packages/GPy/util/linalg.py in tdot_blas(mat, out)
    316     mat = np.asfortranarray(mat)
    317     out = blas.dsyrk(alpha=1.0, a=mat, beta=0.0, c=out, overwrite_c=1,
--> 318                      trans=0, lower=0)
    319 
    320     symmetrify(out, upper=True)

error: failed in converting 2nd keyword `c' of _fblas.dsyrk to C/Fortran array

Workaround for discrete optimization?

Imagine that I build objective function such that internally it discretizes the real input numbers to integer ones (e.g. 1.6 -> 2, 0.3 -> 0 etc) and computes the objective on these discrete values. This way while GPyOpt sees variables as real numbers effectively it does discrete optimization, as I can always discretize found solution while preserving objective value. On the one hand side, it is guaranteed (at least in theory) that in the limit the result of optimization is globally optimal, thus I would expect that result of implicit discrete optimization will also be global. On the other hand side, it is usually assumed that the objective should be Lipschitz continuous (in theory), which it is not for the outlined case, but it might work just fine in practice. Is this the way people adapt Bayesian Optimization to the discrete case or is there something more to it?

Link to sources in the static site is wrong

Hi! The link to the git sources in:

http://sheffieldml.github.io/GPyOpt/

Actually links to the GPy sources

Why this linear problem generates such strange predictions?

Hello.

Testing out the devel branch on the simplest problem I could find (the function sums up the two input variables):

bounds = [{'name': 'var_1', 'type': 'continuous', 'domain': (-1,1)},
          {'name': 'var_2', 'type': 'continuous', 'domain': (-1, 1)}
          ]

def myf(x):
    x = GPyOpt.util.general.reshape(x, 2)
    if x.shape[1] != 2:
        return 'wrong input dimension'
    else:
        x1 = x[:, 0] #column of var1
        x2 = x[:, 1] #column of var2
        return x1+x2

max_iter = 50
import GPyOpt
import numpy as np
myProblem = GPyOpt.methods.BayesianOptimization(myf,bounds)
myProblem.run_optimization(max_iter)

When I then call
myProblem.plot_acquisition("test.png")
I get a very weird prediction. The posterior mean is fine, the posterior's standard deviation is way out of proportion and basically random.

Am I doing something wrong?

Matern kernel

I understand that "kernel" will become deprecated in the new version. How can one specify the Gpy Matern ARD 5/2 kernel with the newest API?

GPyOpt.methods.bayesian_optimization.BayesianOptimization(f, domain=None, constrains=None, cost_withGradients=None, model_type='GP', X=None, Y=None, initial_design_numdata=None, initial_design_type='random', acquisition_type='EI', normalize_Y=True, exact_feval=False, acquisition_optimizer_type='lbfgs', model_update_interval=1, evaluator_type='sequential', batch_size=1, num_cores=1, verbosity=True, verbosity_model=False, bounds=None, **kwargs)

Exploiting gradient information in BO

Dear Developers,
At the moment we are using BO by inputting single data points, but we are keen to exploit also the gradient information which we obtain simultaneously from the data point calculation. Can you point us in the right direction on how to add this information to our model and speed up the BO?
Many thanks, Milica.

Acquisition optimizer error

Hi,

I have been running into this error with both the DIRECT and CMA optimizers:

    133     if isinstance(s, np.ndarray):
    134         s[s<1e-10] = 1e-10
--> 135     elif s< 1e-10:
    136         s = 1e-10
    137     u = (fmin-m+acquisition_par)/s

Any idea how this is caused? I can't seem to figure out what s is and why it is a list.

Does setting verbose or verbosity in the run_optimization has any effect?

I just started playing around with GPyOpt and I noticed that there are stepwise output printed during the optimization run?

`
opt = GPyOpt.methods.BayesianOptimization(f=gpyRFC, domain=domain, model_type='GP_MCMC', acquisition_type='EI_MCMC', acquisition_weight=0.1)

opt.run_optimization(max_iter=150, verbose=True, verbosity=True)`

Which files should I explore to fix this? I can then submit a PR.

Thanks

how to parallelize nontrivial functions without "error in parallel computation"?

GPyOpt BayesianOptimization doesn't seem to work with complex objective functions that don't automatically output arrays. Does the wrapper not support delayed evaluation? For instance

import xgboost as xgb

def train(args):
   args = np.atleast_2d(args)
   n = args.shape[0]
   parameters = [{'gamma': args[i,0], 'lambda': args[i,1], 
    'max_depth': args[i,2].astype(int)} for i in range(n)]
   def xgbtrain(dtrain, dtest, arg):
        evals_result = {}
        watchlist = [(dtest, 'eval')]
        model  = xgb.train(arg, dtrain, evals = watchlist)
        return evals_result['eval']['rmse'][-1]
   loss = map(lambda arg: xgbtrain(train, dtest, arg), args)
   return np.array(loss).reshape(n,1)

when I run this with

bounds = [{'name': 'gamma', 'type': 'continuous', 'domain': (0, 10)},
              {'name': 'lambda', 'type': 'continuous', 'domain': (1,25)},
              {'name': 'max_depth', 'type': 'continuous', 'domain': (3, 5)}]

bopt = GPyOpt.methods.BayesianOptimization(f=train,
                               domain = bounds,
                                batch_size = 3,
                                num_cores = 3,
                               evaluator_type='local_penalization')

bopt.run_optimization(max_iter = 10)

It returns the error "error in parallel computation: fall back to single process". Is there a heuristic way to write training evaluation wrapper objectives that the module can handle?

Discrete and categorical data?

Can GPyOpt currently deal with categorical and discrete inputs? I couldn't find any tutorial on this aspect

problems when following firstexamples.md

Thanks for making this library available to the public !

While trying to getting started with it, I tried to follow firstexamples.md and I noticed:

from GPyOpt import BayesianOptimization does not work but from GPyOpt.methods import BayesianOptimization does
my python interpreter does not like the backslash in def f(x): return (6*x-2)**2\np.sin(12*x-4). Should it be * instead of \ ?

(I installed GPyOpy and Gpy using pip on OSX, GPyOpt.version is 1.0.2 and GPy.version is 1.4.0 -- am I using a too old version or should firstexamples.md be updated ?)

The way to change from minimize to maximize

Nice to meet you.
I belong to a university in Japan.(e,g. I'm a Japanese and a university student)

These days I use GPyOpt to research Bayesian Optimization.
To learn about GPyOpt, I read this .
http://nbviewer.jupyter.org/github/SheffieldML/GPyOpt/blob/devel/manual/GPyOpt_reference_manual.ipynb
There is a example how to solve "minimize" problem.
But there is not a example how to solve "maximize" problem.
Could you teach me how to change problem from "minimize" to "maximize" by using GPyOpt's function or tool?

I'm sorry I can't explain my problem well for my poor vocabulary.
And I little have knowledge GPyOpt and BaysianOptimization,so I'll be glad to teach answer simply.
Thank you in advance for your help.

Optimizer: Trying to use CMA results in crash

When porting from the old-style code, the Opt_CMA class seems to have lost its optimize() method -- in fact, the code is there, but it is a part of __init__(). As a result, when trying the CMA acquisition optimizer, the default Optimizer::optimize() is used which returns a pair of None's ...

I have it fixed and running (actually the test will finish some time tommorrow) together with some other fixes to optimize.py. Shall I create a pull request?

Customising Bayesian Optimisation class

Dear Developers,
Following the notebook tutorials, we started using the BayesianOptimisation class, but we soon found it restricted to particular option choices. In order to go custom, we switched to the core.BO class, which appears to be at the heart of modularBO. Is this the right way to go?
Best regards, Milica.

devel branch: how is optimization of acquisition function over discrete variables done?

From your previous messages, I understand that in order to deal with discrete parameters you restrict search in the acquisition function only over feasible (discrete) values. As gradient descent is at least not obvious to apply in this case, I assume you use a combination of gradient descent and some smart enumeration of discrete parameters. I would greatly appreciate if you would drop a short message on how specifically you optimize, so that I can understand what I should expect in terms of scalability.

Error when choose other acquisition_optimizer_type

In [6]:

myProblem = GPyOpt.methods.BayesianOptimization(
    CO, 
    bounds, 
    exact_feval=True,
    maximize = True,
    normalize_Y = True,
    num_cores = 7,
    batch_size = 7,
    acquisition_optimizer_type='DIRECT',
    evaluator_type = 'random'
)
Cannot find DIRECT library, please install it to use this option.
3.2091067970977254 28.0 125.0
2.3088033218144273 31.0 313.0
4.767196848416326 30.0 106.0
3.699853608735686 18.0 300.0
0.11330175343781992 19.0 115.0
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-6-e8c2660be0d5> in <module>()
      8     batch_size = 7,
      9     acquisition_optimizer_type='DIRECT',
---> 10     evaluator_type = 'random'
     11 )

/usr/local/lib/python3.5/site-packages/GPyOpt-1.0.2-py3.5.egg/GPyOpt/methods/bayesian_optimization.py in __init__(self, f, domain, constrains, cost_withGradients, model_type, X, Y, initial_design_numdata, initial_design_type, acquisition_type, normalize_Y, exact_feval, acquisition_optimizer_type, model_update_interval, evaluator_type, batch_size, num_cores, verbosity, verbosity_model, bounds, maximize, **kwargs)
    241 
    242         # --- Initialize everything
--> 243         self.run_optimization(max_iter=0,verbosity=self.verbosity)
    244 
    245     def _model_chooser(self):

/usr/local/lib/python3.5/site-packages/GPyOpt-1.0.2-py3.5.egg/GPyOpt/methods/bayesian_optimization.py in run_optimization(self, max_iter, max_time, eps, verbosity, save_models_parameters, report_file, evaluations_file, models_file, **kwargs)
    455                 self.acquisition_optimizer.optimizer ='CMA'
    456             print('WARNING: "acqu_optimize_method" will be deprecated in the next version!')
--> 457         super(BayesianOptimization, self).run_optimization(max_iter = max_iter, max_time = max_time,  eps = eps, verbosity=verbosity, save_models_parameters = save_models_parameters, report_file = report_file, evaluations_file= evaluations_file, models_file=models_file)
    458 
    459     def _sign(self,f):

/usr/local/lib/python3.5/site-packages/GPyOpt-1.0.2-py3.5.egg/GPyOpt/core/bo.py in run_optimization(self, max_iter, max_time, eps, verbosity, save_models_parameters, report_file, evaluations_file, models_file)
    106 
    107             # --- Update and optimize acquisition and compute the exploration level in the next evaluation
--> 108             self.suggested_sample = self._compute_next_evaluations()
    109 
    110             if not ((self.num_acquisitions < self.max_iter) and (self._distance_last_evaluations() > self.eps)):

/usr/local/lib/python3.5/site-packages/GPyOpt-1.0.2-py3.5.egg/GPyOpt/core/bo.py in _compute_next_evaluations(self)
    184         Computes the location of the new evaluation (optimizes the acquisition in the standard case).
    185         """
--> 186         return self.evaluator.compute_batch()
    187 
    188     def _update_model(self):

/usr/local/lib/python3.5/site-packages/GPyOpt-1.0.2-py3.5.egg/GPyOpt/core/evaluators/batch_random.py in compute_batch(self)
     25         Adds to the first location batch_size-1 randomly selected elements.
     26         """
---> 27         return np.vstack((self.acquisition.optimize(),initial_design('random',self.acquisition.space,self.batch_size-1)))

/usr/local/lib/python3.5/site-packages/GPyOpt-1.0.2-py3.5.egg/GPyOpt/acquisitions/base.py in optimize(self)
     57             out = self.optimizer.optimize(f=self.acquisition_function)[0]
     58         else:
---> 59             out = self.optimizer.optimize(f=self.acquisition_function, f_df=self.acquisition_function_withGradients)[0]
     60         return out
     61 

/usr/local/lib/python3.5/site-packages/GPyOpt-1.0.2-py3.5.egg/GPyOpt/optimization/acquisition_optimizer.py in optimize(self, f, df, f_df)
    281         for i in range(num_discrete):
    282             self.mixed_optimizer.fix_dimensions(dims=self.discrete_dims, values=self.discrete_values[i,:])
--> 283             partial_x_min[i,:] , partial_f_min[i,:] = self.mixed_optimizer.optimize(f, df, f_df)
    284 
    285         return np.atleast_2d(partial_x_min[np.argmin(partial_f_min)]), np.atleast_2d(min(partial_f_min))

/usr/local/lib/python3.5/site-packages/GPyOpt-1.0.2-py3.5.egg/GPyOpt/optimization/acquisition_optimizer.py in optimize(self, f, df, f_df)
    175             if self.search:
    176                 if self.f_df == None: fp_dfp = None  # -- In case no gradients are available
--> 177                 x_min, f_min = self.optimizer.optimize(x0, f =fp, df=None, f_df=fp_dfp)
    178                 return self._expand_vector(x_min), f_min
    179             else:

TypeError: 'NoneType' object is not utterable

for "DIRECT", the actual exception is this:

Traceback (most recent call last):
  File "/usr/local/lib/python3.5/site-packages/GPyOpt-1.0.2-py3.5.egg/GPyOpt/optimization/optimizer.py", line 96, in optimize
    x,_,_ = solve(DIRECT_f_wrapper(f),lB,uB, maxT=self.maxiter)
  File "/usr/local/lib/python3.5/site-packages/DIRECT/__init__.py", line 206, in solve
    raise Exception(ERROR_MESSAGES[abs(ierror)-1])
IndexError: tuple index out of range

Similar error is observed when "CMA" is chosen.


Traceback (most recent call last):
  File "/usr/local/lib/python3.5/site-packages/GPyOpt-1.0.2-py3.5.egg/GPyOpt/optimization/optimizer.py", line 123, in __init__
    x = cma.fmin(CMA_f_wrapper(f), (uB + lB) * 0.5, 0.6, options={"bounds":[lB, uB], "verbose":-1})[0]
NameError: name 'f' is not defined
Traceback (most recent call last):

The exception is bit misleading, because the error is caused by 'f' not defined.