smtorg / smt Goto Github PK

View Code? Open in Web Editor NEW

625.0 625.0 190.0 123.14 MB

Surrogate Modeling Toolbox

Home Page: https://smt.readthedocs.io/en/stable

License: BSD 3-Clause "New" or "Revised" License

Python 5.71% C++ 0.12% Jupyter Notebook 94.14% Cython 0.03%

derivative machine-learning mixture-of-experts multi-fidelity predictive-modeling sampling surrogate-models

smt's People

Contributors

Stargazers

Watchers

Forkers

hwangjt robfalck justinsgray jomorlier relf thiagoritto thearn shivajimedida nuaalht m-meliani wangshuo17 richard27yang dushhyanthrajaram sichenghe iywo giegloop driesverstraete alexdiazbolivar dracula-ybp shb84 sharadgupta27 cyrilpic saberdm yuhaibo2017 thesadman donaldratnerjones gyani123 npuljc felix187 shaobinli dahjeong haojia632 friedenhe bugleilei jbussemaker ljuillen volpatto rmtorre mcresest afcarl irfan-gh boomtechnology kodexp xyuan fzahle dongshengjia vgurev sirius-aerostar siesi repriem minaskar chrislupp amosjoseph flo-code jimmy-inl zgtjdxlhz ertandemiral rruusu tjmackman xinbai123 gpilania emilerouxsmb xclmj ha-jun-sung xmlconf git163 vindrouet yutaozhou holm-xie k86228 scalet98 tmxkhj rishirelan zhihaoliu-git vishalbelsare zeta1999 mixed-optimization-benchmark leeshutao witekklim natonera esch3r xiehuohuo anfelopera syam-s geofiber udemirezen ldap joshuauk1026 paul-saves bouhlelma killingvector raul-rufato lamkina zeliangsu conorncleeton sh2ocn robingrapin luizftiberio wimvanrees zhang-yufeng

smt's Issues

How high is high?

Hi all,

This is not an issue exactly.

The docs (and the paper) mention that unlike Kriging, KPLS(K) can deal with high-dimensional models.

However, I find it hard to make KPLS work with anything beyond 30 dimensions (even for trivial models).

Did anyone manage to make KPLS work in such cases?

When using the Full Factorial sampling method for the training of the WingWeight SMT function , it seems that the prediction of the values doesn't run smoothly for less than 600 training points. Consequently, for higher dimension problems that minimum requirement might rise. When it comes to the Sphere function, for any number of dimensions higher than 3 the FullFactorial sampling results in faulty function values.

Installation fails if numpy is not yet installed

If you install smt 0.3.4 from pypi on a clean environment, it fails because setup.py wants to use numpy, which is not available, giving a ModuleNotFoundError.

appveyor Python 3.4

Should add Python 3.4 in appveyor.yml file. It is temporarily removed due to the error from

conda update conda

Save and reload trained models

How can I save and reload trained models?
I tried to pickle them, but it does not work.
I would like to train once and then use the trained model in another python file to predict data points.
Thank you very much, your package is incredible!

Scipy 1.5+ breaking EGO tutorial notebook

The EGO tutorial notebook is not behaving as expected. Tested July 6, 2020.

Installed SMT in Docker container running on Ubuntu 18.04 with Python 3.8.3.

Here are other packages installed by pip during building. Since it is relevant to this problem, I will highlight Scipy==1.5.1:

Package                       Version
----------------------------- ---------
alabaster                     0.7.12
attrs                         19.3.0
Babel                         2.8.0
backcall                      0.2.0
bleach                        3.1.5
certifi                       2020.6.20
chardet                       3.0.4
cycler                        0.10.0
Cython                        0.29.20
decorator                     4.4.2
defusedxml                    0.6.0
docutils                      0.16
entrypoints                   0.3
idna                          2.10
imagesize                     1.2.0
ipykernel                     5.3.1
ipython                       7.16.1
ipython-genutils              0.2.0
ipywidgets                    7.5.1
jedi                          0.17.1
Jinja2                        2.11.2
joblib                        0.16.0
jsonschema                    3.2.0
jupyter                       1.0.0
jupyter-client                6.1.5
jupyter-console               6.1.0
jupyter-core                  4.6.3
kiwisolver                    1.2.0
MarkupSafe                    1.1.1
matplotlib                    3.2.2
mistune                       0.8.4
nbconvert                     5.6.1
nbformat                      5.0.7
notebook                      6.0.3
numexpr                       2.7.1
numpy                         1.19.0
numpydoc                      1.1.0
packaging                     20.4
pandas                        1.0.5
pandocfilters                 1.4.2
parso                         0.7.0
pexpect                       4.8.0
pickleshare                   0.7.5
pip                           20.1.1
prometheus-client             0.8.0
prompt-toolkit                3.0.5
ptyprocess                    0.6.0
pyDOE2                        1.3.0
Pygments                      2.6.1
pyparsing                     2.4.7
pyrsistent                    0.16.0
python-dateutil               2.8.1
pytz                          2020.1
pyzmq                         19.0.1
qtconsole                     4.7.5
QtPy                          1.9.0
requests                      2.24.0
scikit-learn                  0.23.1
scipy                         1.5.1
seaborn                       0.10.1
Send2Trash                    1.5.0
setuptools                    47.1.1
six                           1.15.0
smt                           0.5.3
snowballstemmer               2.0.0
Sphinx                        3.1.2
sphinxcontrib-applehelp       1.0.2
sphinxcontrib-devhelp         1.0.2
sphinxcontrib-htmlhelp        1.0.3
sphinxcontrib-jsmath          1.0.1
sphinxcontrib-qthelp          1.0.3
sphinxcontrib-serializinghtml 1.1.4
tables                        3.6.1
terminado                     0.8.3
testpath                      0.4.4
threadpoolctl                 2.1.0
tornado                       6.0.4
tqdm                          4.47.0
traitlets                     4.3.3
urllib3                       1.25.9
wcwidth                       0.2.5
webencodings                  0.5.1
wheel                         0.34.2
widgetsnbextension            3.5.1

Installation of SMT was performed after installing Cython, numpy and other packages, and was apparently successful.

When running the cells in the EGO Tutorial notebook, I received the following error and trace for any calls that involve Scipy's minimize function, which is itself wrapped by the optimize method:

ValueError Traceback (most recent call last)
in
15 ego = EGO(n_iter=n_iter, criterion=criterion, xdoe=xdoe, xlimits=xlimits)
16
---> 17 x_opt, y_opt, ind_best, x_data, y_data, x_doe, y_doe = ego.optimize(fun=rosenbrock)
18
19 print('Xopt for Rosenbrock ', x_opt,y_opt, ' obtained using EGO criterion = ', criterion )

/usr/local/lib/python3.8/site-packages/smt/applications/ego.py in optimize(self, fun)
155 # Virtual enrichement loop
156 for p in range(n_parallel):
--> 157 x_et_k, success = self._find_points(x_data, y_data)
158 if not success:
159 self.log(

/usr/local/lib/python3.8/site-packages/smt/applications/ego.py in _find_points(self, x_data, y_data)
251 for ii in range(n_start):
252 opt_all.append(
--> 253 minimize(
254 self.obj_k,
255 x_start[ii, :],

/usr/local/lib/python3.8/site-packages/scipy/optimize/_minimize.py in minimize(fun, x0, args, method, jac, hess, hessp, bounds, constraints, tol, callback, options)
623 return _minimize_cobyla(fun, x0, args, constraints, **options)
624 elif meth == 'slsqp':
--> 625 return _minimize_slsqp(fun, x0, args, jac, bounds,
626 constraints, callback=callback, **options)
627 elif meth == 'trust-constr':

/usr/local/lib/python3.8/site-packages/scipy/optimize/slsqp.py in _minimize_slsqp(func, x0, args, jac, bounds, constraints, maxiter, ftol, iprint, disp, eps, callback, finite_diff_rel_step, **unknown_options)
366
367 # ScalarFunction provides function and gradient evaluation
--> 368 sf = _prepare_scalar_function(func, x, jac=jac, args=args, epsilon=eps,
369 finite_diff_rel_step=finite_diff_rel_step,
370 bounds=new_bounds)

/usr/local/lib/python3.8/site-packages/scipy/optimize/optimize.py in _prepare_scalar_function(fun, x0, jac, args, bounds, epsilon, finite_diff_rel_step, hess)
259 # ScalarFunction caches. Reuse of fun(x) during grad
260 # calculation reduces overall function evaluations.
--> 261 sf = ScalarFunction(fun, x0, args, grad, hess,
262 finite_diff_rel_step, bounds, epsilon=epsilon)
263

/usr/local/lib/python3.8/site-packages/scipy/optimize/_differentiable_functions.py in init(self, fun, x0, args, grad, hess, finite_diff_rel_step, finite_diff_bounds, epsilon)
93
94 self._update_grad_impl = update_grad
---> 95 self._update_grad()
96
97 # Hessian Evaluation

/usr/local/lib/python3.8/site-packages/scipy/optimize/_differentiable_functions.py in _update_grad(self)
169 def _update_grad(self):
170 if not self.g_updated:
--> 171 self._update_grad_impl()
172 self.g_updated = True
173

/usr/local/lib/python3.8/site-packages/scipy/optimize/_differentiable_functions.py in update_grad()
89 self._update_fun()
90 self.ngev += 1
---> 91 self.g = approx_derivative(fun_wrapped, self.x, f0=self.f,
92 **finite_diff_options)
93

/usr/local/lib/python3.8/site-packages/scipy/optimize/_numdiff.py in approx_derivative(fun, x0, method, rel_step, abs_step, f0, bounds, sparsity, as_linear_operator, args, kwargs)
386 f0 = np.atleast_1d(f0)
387 if f0.ndim > 1:
--> 388 raise ValueError("f0 passed has more than 1 dimension.")
389
390 if np.any((x0 < lb) | (x0 > ub)):

ValueError: f0 passed has more than 1 dimension.

Downgrading Scipy==1.4.1 (the last release prior to the release of the EGO notebook in April) led to expected behavior for the section that uses fun = x sin x. I could make this section work with Scipy==1.5.1 by adding .flatten() to all of the return calls in the function definitions for EI, SBO, and UCB. Note that SBO does not return the global minimum; is that expected behavior?

For the EGO section using Rosenbrock function, I could not find a way to flatten the output from the function in a way to make it work with Scipy 1.5.1. The section works with Scipy==1.4.1 if I reduce the number of iterations to 50 (from default of 100 in the notebook). Using 100 leads to the following runtime error being repeated (this could be a separate issue, but recording here for reference, can break out to separate issue if appropriate):
/usr/local/lib/python3.8/site-packages/smt/applications/ego.py:197: RuntimeWarning: divide by zero encountered in true_divide args0 = (f_min - pred) / sig

Followed by exceptions like this:
exception : 65-th leading minor of the array is not positive definite

Let me know if I can provide any additional data. Thank you.

GEKPLS Extra Points?

What are extra points in the context of GEKPLS? The documentation simply says "Number of extra points per training point" What is an extra point and when should I enable them?

Thanks,

Kevin

no license file

Please add a BSD license file, this is required for packaging.

Multiple input features cannot have the same value

Hi,

I am trying to implement a KPLS model with my dataset, but it raises this error:
"Multiple input features cannot have the same value".

How can I solve it?

Thank you very much for your support

EDIT: I solved by dropping equal instances, by using dataframe.drop_duplicates()
However now, it seems it is not converging and gives me thousands of these exceptions:

....
exception : 125-th leading minor of the array is not positive definite
exception : 125-th leading minor of the array is not positive definite
exception : 136-th leading minor of the array is not positive definite
exception : array must not contain infs or NaNs
exception : array must not contain infs or NaNs
exception : array must not contain infs or NaNs
.....

How can I solve this problem?
Thank you very much :)

Issue (dll numpy) Python 3.6 on windows

Kriging based surrogates are not pickable

Current Kriging based surrogates are not pickable because of options related to regression or correlation functions which are actual functions which results in the following error message

cPickle.PicklingError: Can't pickle : attribute lookup __builtin__.function failed

Actually the declaration of those options (poly, corr) is a bit hacky as such option allows to specify a callable while acceptable values are documented to be an enumerate.
This implementation has the benefit to allow the user to specify is own functions for correlation and regression but it prevents from pickling.
I would like to propose a pull request to modify the code to have the current Krigings implementation pickable (hence make these options an enumerate). I am not sure at the moment on what is the best way to keep the "user-defined correlation or regression functions" capability. Is it used? Do we have to keep it? Plugin mechanism? Any thoughts?

GEKPLS 1D?

Does GEKPLS work for 1D? I am trying this basic 1D example. am am getting the following error:

  File "kriging_utils.py", line 269, in ge_compute_pls
    _X[2, 1] += delta_x * (xlimits[1, 1] - xlimits[1, 0])
IndexError: index 1 is out of bounds for axis 1 with size 1

with the following code

import numpy as np
from smt.surrogate_models import GEKPLS

def f(x):
    return x*np.sin(x)
def f_deriv(x):
    return x * np.cos(x) + np.sin(x)

limits = np.zeros((1,2))
limits[0,0] = 0
limits[0,1] = 8
t = GEKPLS(theta0=[1], xlimits=limits, print_prediction=False, extra_points=0, print_global=False)
trainingx = np.atleast_2d(np.array([0.,4,8.])).T
print(trainingx)
trainingy = f(trainingx)
print(trainingy)
t.set_training_values(trainingx, trainingy)
trainingderiv = f_deriv(trainingx)
print(trainingderiv)

t.set_training_derivatives(trainingx, trainingderiv, 0)
t.train()

any ideas?

Thanks,

Error using rmtb

I have run the tutorial case for CRM successfully (based on inputs given in the forum) and am now trying on some synthetic data . I have a sample set of data values and have tried to use rmtb ( using tutorial script for CRM wing). It looks to be working fine except for the error in the last step. I am attaching the codes I have used and the screen shot of error. Any suggestion would be of great help.
Thanks
Narahari

six module requirement

six should be added to install_requires in setup.py and README

How to use SMT for discrete data

Not an issue but was wondering how to use the toolbox when there is no function but a table of data ? For example if I run Computational Fluid Dynamics code for a standard case (generic wing) at a (1) set of angle of attack , (2) Mach Numbers, (3) Sweep angles and get Lift/Drag ratio. I am interested in constructing a surrogate for the data in order to find optimal combination of angle of attack, sweep angle and Mach number. Of course, the problem itself is old many people have worked on it, but I am interested in the process . How should I start ?
Thanks in advance

EGO uses variance instead of standard deviation

Unless I am completely mistaken, the EGO application (in smt/applications/ego.py) uses variances (sigma squared), as returned by KRG.predict_variances(), where it should really use standard deviations (sigma). This affects both the EI and UCB criteria, as well as the visualization of the example in doc/_src_docs/applications/ego.rst.

Both the EI and UCB criteria produce better results and the error estimates and the expected improvement look much more like a Kriging model should when this is fixed by adding the necessary np.sqrt() calls around predict_variances().

no 0.1 tag?

Hi,
Seems v0.1 has been announced on the gdr-mascotnum mailing list however I don't see any tag.
j

Parallelizing EGO

While using your implementation of the EGO algorithm, I noticed that at the start it suggests n_doe number of points for the objective function (in one array). Each subsequent iteration suggests one point. Is there an option to continue suggesting more than one point? That would help me greatly in parallelizing the calls to my objective function.

Thank you for your time.

RTMS explanation

Hello guys! First of all, really cool library.

I have been working on Surrogates.jl, a Julia package that mostly has the same goal.
I am trying to implement from this paper here: "A fast-prediction surrogate model for large datasets" which I believe it's related to this library.
However, I am not understanding how to build the matrix $F$, for each i-esim row I should evaluate the i-esim training point x at some splines, right? The number of columns is the degree of the splines? I am bit confused. Could you shed some light?

RMTB does not build properly on OSX >= 10.13

SM renaming?

Hi. I am a bit surprised by the recent renaming of SM in Methods (#85) :

First, why not just SurrogateModel which seems to be the natural move regarding the purpose of SMT?
Second, in this case, it should have been Method not Methods (same remark goes for Problem renamed in Problems).

Error in test_ego.py

Function test_branin_2D() in test_ego.py has one of the minima wrong. [3.14, 12.275] should be [3.14, 2.275].

Problem running CRM wing case

I have been exploring the use of SMT package as a part of student curriculum here in our university, hence I have downloaded the package from github. I have tried out some of the examples without any issue but have run into a specific problem related to RMTB and RMTC methods. Either in the One_D_step or CRM_wing case I am getting a module not found error (screen shot attached). Any pointers to identify the error would be of great help.

The compute_rms_error function returns the normalized RMS error

QP predict_values not correct for multiple points

Hi,

It seems the QP model has a bug when predicting multiple points in one call:

import numpy as np

from smt.surrogate_models import QP, KRG
from smt.examples.rans_crm_wing.rans_crm_wing import (
    get_rans_crm_wing,
    plot_rans_crm_wing,
)

xt, yt, xlimits = get_rans_crm_wing()

interp = QP()
interp.set_training_values(xt, yt)
interp.train()
v0 = np.zeros((4, 2))
for ix, i in enumerate([10, 11, 12, 13]):
    v0[ix, :] = interp.predict_values(np.atleast_2d(xt[i, :]))
v1 = interp.predict_values(np.atleast_2d(xt[10:14, :]))
print('single nt', v0)
print('multiple nt', v1)

print('diff', v0 - v1)

which produces the following outputs:

single nt [[0.01004526 0.19395995]
 [0.01246052 0.34478044]
 [0.01835094 0.49012219]
 [0.02771651 0.6299852 ]]
multiple nt [[0.01004526 0.01246052]
 [0.01835094 0.02771651]
 [0.19395995 0.34478044]
 [0.49012219 0.6299852 ]]
diff [[-6.93889390e-18  1.81499430e-01]
 [-5.89041754e-03  3.17063930e-01]
 [-1.75609013e-01  1.45341748e-01]
 [-4.62405678e-01  0.00000000e+00]]

The issue popped up when using the VFM model, where the LF model is called with the full HF dataset in one call. works for KRG.

Exception: Leading minor of the array is not positive definite

I tried to train a Kriging model with 2 input dimensions data read from a csv file. The code is as follows

#!/usr/bin/env python
# coding: utf-8
import numpy as np
import matplotlib.pyplot as plt
import pandas as pd
from smt.surrogate_models import KRG

# Read CSV Data
CSVData = pd.read_csv('X1X2Y.csv')
data = CSVData.values
xt = data[:,0:2]
yt = data[:,2]

# Remove unique elements
u,indices = np.unique(xt,axis=0,return_index=True)
xt = xt[indices,:]
yt = yt[indices]

# Train the model
sm = KRG(theta0=[1e-2,1e-2])
sm.set_training_values(xt, yt)
sm.train()

During the training, I am getting the following error

_________________________________________________________________________
   
                                  Kriging
___________________________________________________________________________
   
 Problem size
   
      # training points.        : 815
   
___________________________________________________________________________
   
 Training
   
   Training ...
exception :  69-th leading minor of the array is not positive definite
exception :  79-th leading minor of the array is not positive definite
exception :  78-th leading minor of the array is not positive definite
exception :  69-th leading minor of the array is not positive definite
exception :  69-th leading minor of the array is not positive definite
exception :  79-th leading minor of the array is not positive definite
exception :  78-th leading minor of the array is not positive definite

What could be the reason for this error ? How can I fix this ? I am hereby attaching the code and the data I used.

X1X2Kriging.zip

How to control verbosity?

I My query is , is there any provision provided in the library to not print the intermediate info.
If yes? Can you please tell me.
If no, Can you suggest me where should I change in the code.

Optimization framework+adding CNN surrogate base model

Hello,

I would appreciate if you can let me know about any open source optimization code, that can use your surrogate model outputs for optimization process.
In addition, I would like to know if that is possible to add convectional neural network (CNN) to your already developed surrogate models.

Thanks,
Anahita

Remove clustered sampling

Torsion_vibration and water_flow are monotone in every coordinate

Just a comment, not an issue:
the test cases torsion_vibration (15d) and water_flow (8d) both seem to be monotone,
increasing or decreasing, in every coordinate, so the optima are on the bounding boxes;
see Coord-sketch.py and .log under https://gist.github.com/denis-bz .
Does anyone know of more realistic test cases ?

ImportError

Hi, probably it is a stupid error, but I am quite new to python. So I apologize in advance.
I tried to run

from smt.surrogate_models import RBF

but I received this error:
ImportError: cannot import name 'RBF' from 'smt.surrogate_models' (w:\17_users\trainees\battolla\smt\smt\surrogate_models_init_.py)

How can I do to solve it? Thank you very much.

Kriging variance prediction not working for multiple points

I'm trying to predict the variance of a Kriging-based surrogate model for multiple points at a time, but it seems that the function is only implemented for one point at a time.

Reproduce using (note that pull requests #158 and #159 first need to be accepted):

# Python 3.6
# SMT 0.3.3
import numpy as np
from smt.surrogate_models.krg import KRG

x = np.random.random((10, 3))
y = np.random.random((10, 2))

kriging = KRG(theta0=[1e-2]*3)
kriging.set_training_values(x, y)
kriging.train()

var = kriging.predict_variances(x)

This results in:

Traceback (most recent call last):
  File "reproduce.py", line 11, in <module>
    var = kriging.predict_variances(x)
  File "....\site-packages\smt\surrogate_models\surrogate_model.py", line 370, in predict_variances
    s2 = self._predict_variances(x)
  File "....\site-packages\smt\surrogate_models\krg_based.py", line 389, in _predict_variances
    1.0 - (rt ** 2.0).sum(axis=0) + (u ** 2.0).sum(axis=0)
ValueError: operands could not be broadcast together with shapes (2,) (10,)

Replacing the last line with the following makes the prediction work:

var = np.zeros(y.shape)
for i in range(x.shape[0]):
    var[i, :] = kriging.predict_variances(x[[i], :])

It would be nice if this could be corrected.

How to compute gradients?

Hi,
I have read your documentation and gradient-enhanced KPLS models seem really promising. However I did not see any function in your documentation that enables to compute training derivatives to feed into the model.
In a general problem, how can I do that? Using finite differencing? Do you have any external code, source that you can suggest me to compute gradients?
Thank you very much.

Citation format

I'm using your kriging implemention and LHS method to construct a surrogate model of injection molding simulations (similar to this one). Are you ok with this citation format

M. A. Bouhlel, et al., Surrogate modeling toolbox, 2018. URL:https://github.com/SMTorg/SMT.

or do I need to add more information? Thanks!

Mac tests

Add Mac tests

can smt store the model?

I want store a model that be trained, because every time train the same model by the same data is very inefficient.

1-dimensional surrogate with GEKPLS

Hi, I'm not able to train a GE-KPLS surrogate when the input is 1-dimensional. There seems to be an implicit requirement on the surrogate being at least 2-dimensional. See:

import numpy as np
from smt.surrogate_models import GEKPLS
from smt.problems import NdimRosenbrock
from smt.sampling_methods import LHS

# Construction of the DOE
nsamp = 10
fun = NdimRosenbrock(ndim=1)
sampling = LHS(xlimits=fun.xlimits, criterion="m")
xt = sampling(nsamp)
yt = fun(xt)
yd = np.zeros((nsamp,1))
yd[:,0] = fun(xt, kx=0).flatten()

# Build the GEKPLS model
sm = GEKPLS(
    theta0=[1e-2], xlimits=fun.xlimits, extra_points=1, print_prediction=False
)
sm.set_training_values(xt, yt)
sm.set_training_derivatives(xt, yd[:,0], 0)
sm.train()

which gave the error

Traceback (most recent call last):
  File "1D_test.py", line 23, in <module>
    sm.train()
  File "site-packages/smt/surrogate_models/surrogate_model.py", line 248, in train
    self._train()
  File "site-packages/smt/surrogate_models/krg_based.py", line 132, in _train
    self._new_train()
  File "site-packages/smt/surrogate_models/krg_based.py", line 93, in _new_train
    X, y = self._compute_pls(X.copy(), y.copy())
  File "site-packages/smt/surrogate_models/gekpls.py", line 56, in _compute_pls
    self.options["extra_points"],
  File "site-packages/smt/utils/kriging_utils.py", line 261, in ge_compute_pls
    _X[2, 1] += delta_x * (xlimits[1, 1] - xlimits[1, 0])
IndexError: index 1 is out of bounds for axis 1 with size 1

The logic in ge_compute_pls would suggest that 2D is the lowest that is supported. Is that a mathematical requirement or an artifact of the implementation? Thanks,

Neil

Regression tests

Add regressions for the missed functions:
VFM, MOE etc

Derivatives prediction by kriging is not working

Issue

In kriging surrogates, gradient prediction provided by predict_derivatives() might be wrong when the option is set as poly='linear'.

Test

I took the 2D surrogate example from GEKPLS example and checked the derivative values in the following three ways. A test point at which the derivatives are computed is one of the training points, so that I can have reference values there.

Training derivative at a test point (reference value)
Predicted derivative by sm.predict_derivatives()
Finite-differenced derivative approximation of sm.predict_values()

I believe 2 and 3 should always match. 1 and 2/3 may not match for digits, but at least they should give similar values.
In my test, KRG and GEKPLS worked as expected with the default option poly='constant', however, they did not when poly='linear'. For the latter case, 1 and 3 were consistent, while 2 (derivative prediction) gave inconsistent values.

Script

import numpy as np
from mpl_toolkits.mplot3d import Axes3D
import matplotlib.pyplot as plt

from smt.surrogate_models import GEKPLS, KRG, KPLS, KPLSK
from smt.problems import Sphere
from smt.sampling_methods import LHS

def main():
    # Construction of the DOE
    fun = Sphere(ndim=2)
    sampling = LHS(xlimits=fun.xlimits, criterion="m")
    xt = sampling(20)
    yt = fun(xt)
    # Compute the training derivatives
    for i in range(2):
        yd = fun(xt, kx=i)
        yt = np.concatenate((yt, yd), axis=1)

    # check KRG models
    print('--------------------------')
    print('KRG with poly = constant' )
    sm_krg_c = KRG(theta0=[1e-2], poly='constant', print_global=False)
    sm_krg_c.set_training_values(xt, yt[:, 0])
    sm_krg_c.train()
    check_derivatives(sm_krg_c, xt, yt)

    print('--------------------------')
    print('KRG with poly = linear' )
    sm_krg_l = KRG(theta0=[1e-2], poly='linear', print_global=False)
    sm_krg_l.set_training_values(xt, yt[:, 0])
    sm_krg_l.train()
    check_derivatives(sm_krg_l, xt, yt)

    # check GEKPLS models
    print('--------------------------')
    print('GEKPLS with poly = constant' )
    sm_gekpls_c = GEKPLS(theta0=[1e-2], poly='constant', xlimits=fun.xlimits, extra_points=1, print_global=False)
    sm_gekpls_c.set_training_values(xt, yt[:, 0])
    for i in range(2):
        sm_gekpls_c.set_training_derivatives(xt, yt[:, 1 + i].reshape((yt.shape[0], 1)), i)
    sm_gekpls_c.train()
    check_derivatives(sm_gekpls_c, xt, yt)

    print('--------------------------')
    print('GEKPLS with poly = linear' )
    sm_gekpls_l = GEKPLS(theta0=[1e-2], poly='linear', xlimits=fun.xlimits, extra_points=1, print_global=False)
    sm_gekpls_l.set_training_values(xt, yt[:, 0])
    for i in range(2):
        sm_gekpls_l.set_training_derivatives(xt, yt[:, 1 + i].reshape((yt.shape[0], 1)), i)
    sm_gekpls_l.train()
    check_derivatives(sm_gekpls_l, xt, yt)

def check_derivatives(sm, xt, yt, i=10):
    # Compares three derivatives at i-th traning point
    # 1. Training derivative: "exact" value
    # 2. Predicted derivative: obtaied by sm.predict_derivatives()
    # 3. Finite-differenced derivative: forward-differencing sm.predict_values()

    # testing point
    x_test = xt[10].reshape((1,2))

    # 2. derivatives prediction by surrogate
    dydx_predict = np.zeros(2)
    dydx_predict[0] = sm.predict_derivatives(x_test, kx=0)[0,0]
    dydx_predict[1] = sm.predict_derivatives(x_test, kx=1)[0,0]

    # 3. derivative by finite-differencing sm.predict_values()
    dydx_fd = np.zeros(2)
    h = 1.e-3  # step size
    x_step1 = x_test + np.array([[h, 0.]])
    x_step2 = x_test + np.array([[0., h]])
    dydx_fd[0] = (sm.predict_values(x_step1) - sm.predict_values(x_test)) / h
    dydx_fd[1] = (sm.predict_values(x_step2) - sm.predict_values(x_test)) / h

    # compare results
    print('Test point: x=', x_test)
    print('  Training derivatives    =', yt[i, 1:])
    print('  Predicted derivatives   =', dydx_predict)
    print('  Finite-diff derivatives =', dydx_fd)
    
if __name__ == '__main__':
    main()

Output

--------------------------
KRG with poly = constant
Test point: x= [[0.37942888 4.64215952]]
  Training derivatives    = [0.75885776 9.28431903]
  Predicted derivatives   = [0.75886037 9.28431727]
  Finite-diff derivatives = [0.75833791 9.28770331]

--------------------------
KRG with poly = linear
Test point: x= [[0.37942888 4.64215952]]
  Training derivatives    = [0.75885776 9.28431903]
  Predicted derivatives   = [3.49403164 6.46394124]
  Finite-diff derivatives = [0.76132158 9.28927155]

--------------------------
GEKPLS with poly = constant
Test point: x= [[0.37942888 4.64215952]]
  Training derivatives    = [0.75885776 9.28431903]
  Predicted derivatives   = [0.7590008 9.2832744]
  Finite-diff derivatives = [0.75934032 9.28445791]

--------------------------
GEKPLS with poly = linear
Test point: x= [[0.37942888 4.64215952]]
  Training derivatives    = [0.75885776 9.28431903]
  Predicted derivatives   = [7.29028109 8.71618885]
  Finite-diff derivatives = [0.75827747 9.28190784]

Notes:

Due to the randomness in LHS, the output value above will not be reproducible. You should be able to see the discrepancy in Predicted derivatives after running the script at most several times.
Although not included in the test script, I had the same issue with KPLS, KPLSK. I did not see this issue in 1D input cases.

Thank you,
Shugo

Help me emstimate surrogates model

Dear you.
Now I have the three input parameters. The each parameter I have the 20 values look like that.

nodes1 = np.array([0.05675, 0.05934, 0.05633, 0.0557 , 0.05702, 0.06401, 0.06322, 0.06571, 0.06099, 0.05832, 0.06196, 0.06463, 0.05507, 0.06351, 0.06287, 0.06122, 0.05407, 0.05985, 0.05774,0.06015])

nodes2 = np.array([0.9486, 0.9095, 0.9856, 0.9318, 1.0477, 1.0489,1.0663, 0.9184, 0.9646, 1.0345, 1.0168, 1.0565, 0.9727, 0.9907, 0.9277, 0.9548, 1.0933, 1.0751,1.0026, 1.0231])

nodes3 = np.array([51.813, 54.279, 52.659, 51.197 , 46.629, 49.791, 48.581, 54.799, 46.413, 47.078, 52.367, 48.204, 50.389, 45.402, 47.893, 50.796 , 49.332, 53.323, 53.713, 45.757])

each pair of value I have a output : example: (node1_1; node2_1; node3_1) have a output ( 0.0204232), so I have the 20 output values.

output=np.array([0.0204232,0.0205054,0.0204971,0.0204463,0.0206686,0.0206678,0.0206883,0.0204627,0.020426,0.0206532,0.0206322,0.020677,0.0204431,0.0205319,0.0204508,0.0204115,0.020721,0.0206988,0.0206179,0.0206418])

how to apply the kriging method find the surrogates model for input and output?
please help me. thanks you so much

I am not able To extract the variances of new points

I am not able To extract the variances of new points.
Here is my code
import numpy as np
import matplotlib.pyplot as plt
from mpl_toolkits.mplot3d import Axes3D
from smt.surrogate_models import RBF
sm = RBF(print_training=False)
sm.options['print_prediction'] = False

def black(x,y):
return (1-x)**2+(y-x)**2

Intialising with some data

xt = np.random.randn(100,2)*10
z=[]
for i in range(100):
temp=(black(xt[i][0],xt[i][1]))
z.append(temp)
Z=(np.array(z))
sm = RBF(d0=5)
sm.set_training_values(xt,Z)
sm.train()

Done Modeling Now we will start plotting

fig = plt.figure()
ax = Axes3D(fig)
surf = ax.plot_trisurf(xt[:,0], xt[:,1], Z, cmap=cm.jet, linewidth=0.1)
fig.colorbar(surf, shrink=0.5, aspect=5)
ax.set_xlabel('X Label')
ax.set_ylabel('Y Label')
ax.set_zlabel('Joining Points')
#plt.savefig('teste.pdf')
plt.show()

Now we will plot the actual curve

MOdeling Function

fig = plt.figure()
ax = fig.add_subplot(111, projection='3d')
yt = np.linspace(-10,10,50)
xt = np.linspace(-10,10,50)
X, Y = np.meshgrid(xt, yt)
final=[]
for i in range(len(xt)):
for j in range(len(xt)):
final.append([xt[i],yt[j]])
zs = sm.predict_values(np.array(final))
var = sm.predict_variances()
Z=np.zeros((50,50))
for i in range(len(final)):
Z[i//50][i%50]=zs[i]
ax.plot_surface(X, Y, Z)
ax.set_xlabel('X Label')
ax.set_ylabel('Y Label')
ax.set_zlabel('Modeling')
plt.show()

The Error it is showing :

NotImplementedError Traceback (most recent call last)
in
11 final.append([xt[i],yt[j]])
12 zs = sm.predict_values(np.array(final))
---> 13 var = sm.predict_variances(xt)
14 Z=np.zeros((50,50))
15 for i in range(len(final)):

~\Anaconda3\lib\site-packages\smt\surrogate_models\surrogate_model.py in predict_variances(self, x)
333 Variances.
334 """
--> 335 check_support(self, 'variances')
336 check_nx(self.nx, x)
337 n = x.shape[0]

~\Anaconda3\lib\site-packages\smt\utils\checks.py in check_support(sm, name, fail)
21 if not sm.supports[name] or fail:
22 class_name = sm.class.name
---> 23 raise NotImplementedError('{} does not support {}'.format(class_name, name))
24
25 def check_nx(nx, x):

NotImplementedError: RBF does not support variances

i'm trying to run RBF example

RBF example has some errors
but other example is ok

Errors in full factorial sampling when using weights

Using the weights argument for the FullFactorial sampling class throws up a few issues (under Anaconda3-2020.02-Windows-x86_64, with smt-0.4.3-cp37-cp37m-win_amd64.whl from PyPI):

If used with an array as input for weights:

Traceback (most recent call last):
  File ".\rbf.py", line 31, in <module>
    ff = smt.sampling_methods.FullFactorial(xlimits=lims, weights=np.array([ff_nx, ff_ny]))
  File "C:\Anaconda3\lib\site-packages\smt\sampling_methods\sampling_method.py", line 38, in __init__
    self.options.update(kwargs)
  File "C:\Anaconda3\lib\site-packages\smt\utils\options_dictionary.py", line 115, in update
    self[name] = dict_[name]
  File "C:\Anaconda3\lib\site-packages\smt\utils\options_dictionary.py", line 71, in __setitem__
    self._assert_valid(name, value)
  File "C:\Anaconda3\lib\site-packages\smt\utils\options_dictionary.py", line 85, in _assert_valid
    assert value in values or isinstance(
ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()

I've patched my own version to fix this by changing the options declare statement in the class initialisation

        self.options.declare("weights", values=[None], types=[list, np.ndarray])

        self.options.declare("weights", values=None, types=[list, np.ndarray])

If used with a list as input for weights:

Traceback (most recent call last):
  File ".\rbf.py", line 32, in <module>
    ff = smt.sampling_methods.FullFactorial(xlimits=lims, weights=[ff_nx, ff_ny])
  File "C:\Anaconda3\lib\site-packages\smt\sampling_methods\sampling_method.py", line 38, in __init__
    self.options.update(kwargs)
  File "C:\Anaconda3\lib\site-packages\smt\utils\options_dictionary.py", line 115, in update
    self[name] = dict_[name]
  File "C:\Anaconda3\lib\site-packages\smt\utils\options_dictionary.py", line 71, in __setitem__
    self._assert_valid(name, value)
  File "C:\Anaconda3\lib\site-packages\smt\utils\options_dictionary.py", line 86, in _assert_valid
    value, types
TypeError: isinstance() arg 2 must be a type or tuple of types

Again, I've patched my own version by changing the options declare statement in the class initialisation

        self.options.declare("weights", values=[None], types=[list, np.ndarray])

        self.options.declare("weights", values=None, types=(list, np.ndarray))

Finally, fix another couple of errors (numpy module name, and a type error with /= operator) with the weight normalisation in _compute (line 41), by changing

            weights /= numpy.sum(weights)

            weights = weights / np.sum(weights)

I can try to commit a fixed version of the code if it's easier.

Make compatible with newer version of pip

Trying to install the package using version 10 of pip raises the following error:
AttributeError: Module Pip has no attribute 'main'

The temporary solution involves rolling back pip to version 9 (https://stackoverflow.com/a/49990889), but it would be better of course if the package is made compatible with the latest pip version.

High-dimension training problem

i have train a 30-dimension surrogate by 1000 pionts,but the different input value have the same value.
if the dismension is 2,SMT does work well .
is that mean SMT don't support high-dimension?

Deprecation warning with scikit-learn 0.22

I get the following deprecation warning for PLS imports in sklearn version 0.22.1 (under Anaconda3-2020.02-Windows-x86_64, with smt-0.4.3-cp37-cp37m-win_amd64.whl from PyPI):

C:\Anaconda3\lib\site-packages\sklearn\utils\deprecation.py:144: FutureWarning: The sklearn.cross_decomposition.pls_ module is  deprecated in version 0.22 and will be removed in version 0.24. The corresponding classes / functions should instead be imported from sklearn.cross_decomposition. Anything that cannot be imported from sklearn.cross_decomposition is now part of the private API.
  warnings.warn(message, FutureWarning)

I've patched my local copy to avoid the warning, by replacing the offending sklearn.cross_decomposition.pls_ import statements in

smt/surrogate_models/kpls.py
smt/surrogate_models/kplsk.py
smt/utils/kriging_utils.py

with the following code:

from packaging import version
from sklearn import __version__ as sklversion
if version.parse(sklversion) < version.parse("0.22"):
    from sklearn.cross_decomposition.pls_ import PLSRegression as pls
else:
    from sklearn.cross_decomposition import PLSRegression as pls

Avoid printing training details

Hi!
Is there a way to avoid printing all the details for training the surrogate models? Like a verbose=False.
Thank you very much.

The saved MFK result takes too much space

In case:
3000 sampling points
8 features
using MFK to create surrogate model
e.g.
from smt.extensions import MFK
sm_MFK = MFK(theta0=numpy.ones(8), eval_noise=True, noise0=5)
sm_MFK.set_training_values(X, Y)
sm_MFK.train()

The saved MFK model (sm_MFK) will take around 0.4GB, which is far beyond my expectation.

So is it possible to reduce the taken memory?

Thanks in advance!

installation instructions

Hi,
Can you please make installation instructions more informative for people not so versed in pip and the like.
Sorry for the many questions but here they are:

I have this error whilst installing via $ pip install git+git://github.com/SMTorg/smt.git. I am not sure what it means.

...
copying smt/extensions\mfk.py -> build\lib.win-amd64-3.7\smt/extensions
copying smt/extensions\moe.py -> build\lib.win-amd64-3.7\smt/extensions
copying smt/extensions\vfm.py -> build\lib.win-amd64-3.7\smt/extensions
copying smt/extensions\__init__.py -> build\lib.win-amd64-3.7\smt/extensions
running build_ext
building 'smt.surrogate_models.rbfclib' extension
error: Microsoft Visual C++ 14.0 is required. Get it with "Microsoft Visual C++ Build Tools": https://visualstudio.microsoft.com/downloads/


----------------------------------------
Command "C:\Users\user1\D\anaconda3\python.exe -u -c "import setuptools, tokenize;__file__='C:\\Users\\user1\\AppData\\Local\\Temp\\pip-req-build-a4wor94u\\setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record C:\Users\user1\AppData\Local\Temp\pip-record-kqmnukkv\install-record.txt --single-version-externally-managed --compile" failed with error code 1 in C:\Users\user1\AppData\Local\Temp\pip-req-build-a4wor94u\

Why can I not just use pip install smt Is it because this will fetch an old version?
I installed python via anaconda. Can I therefore use conda install?
Finally https://smt.readthedocs.io/en/latest/_src_docs/getting_started.html states

In terminal, navigate to the directory in which you want to download the source code and use the command git clone https://github.com/SMTorg/smt.git to download.

should this be the ....anaconda3\Lib\site-packages directory? I ran the pip install -e . command in a temporary directory and I think this created some troubles when it came to uninstalling.

Thanks in advance

Installation fails