redmod-team / profit Goto Github PK

The user wants to specify points where the response should be evaluated. Based on a user-supplied template she tells profit to generate a set of directories and a batch submission script.

Remarks:

1. A template for a single code run is required as well as a for the submission script, as queuing system and specific requirements of the code are not known. One could supply "template templates" for the most common queuing systems. One should not reinvent the wheel by adding a lot of options that SLURM/PBS already supply in their file format that most users know.

In some cases, to use a function in proFIt, it is required to indicate its whole path: from the root file: profit.profit. ... instead of just starting it from the current file.

Standardize surrogates. For now only Custom and GPy.

• set structure in abstract class
• implement interfaces to Config, so the surrogate to be used can easily be selected
• cleanup Custom surrogate functions and add docstrings
• make Fortran kernels user friendly
• provide standard kernels in python
• implement methods, so every surrogate has the same (e.g. train, add_training_data, predict, plot, etc.) and can be accessed by a standardized interface
• implement / revise different calculation methods in backend for Custom surrogate. Make them easily extendable by future developers

Generation of grid for 7 parameters takes 8s for sparse grid opposed to 0.05s for full grid. Test case was Gaussian quadrature rule in https://github.com/redmod-team/suruq/blob/1fc3689bf52eb1610dd450ee1363f388df761031/draft/test_sparsegrid.py#L21 .

Currently, only Uniform and LogUniform are supported after switching configuration backends to yaml.

This is related to #7 . Could be done via MPI and/or simpler local solution for the multi-process runner.

• Noise covariance matrix
• Heteroscedastic noise via linear model and/or another GP
• Individual noise for data sources

pip install -e . --user moves profit into %APPDATA%, which is usually not in %PATH% on Windows with Anaconda. Documentation should be updated to use pip install -e . on this setup.

If the option ntask = n is used for parallel computing, check the number of available cores before starting the computation.

start with PCA and work towards more generic nonlinear methods (maybe based on local sensitivity analysis)

add surface plots and slicing

A user develops a new numerical method that is faster at the same accuracy than existing methods. He wants to produce plots of accuracy vs computation time for his new code as well as an existing one.

Old code

Input parameters:

• relative tolerance, logarithmic from 1e-6 to 1e-12
  Output parameters:
• computation time
• accuracy

New code

Input parameters:

• step size, logarithmic from 1e-1 to 1e-3
  Output parameters:
• computation time
• accuracy

It should be possible to plot two outputs against each other here. So one would fit a response model with x = computation time and y = accuracy.

• create a standardized interface between 'run' and surrogates' active learning
• create actual Active Learning process (fills input.txt with points, that contribute the most information)
• create test cases and benchmark

In the file profit.profit.sur.backend the following functions have the same definition:

1. kernels.gp_matrix(x0, x1, a, K)
2. gp.gp_matrix(x0, x1, a, K)

The posterior covariance matrix (cov_f_star) isn't perfectly symmetric.

There is an error of approximatly 1e-14:
The command : np.max(cov_f_star-np.transpose(cov_f_star)) returns a value arround 1e-14

Implement / revise test cases for Custom and GPy.
Also revise the Examples to match with the cleaned up project structure. (also solves #16)

• Check shift in nll
• Avoid negative eigenvalues
• Don't to iterative eigendecomposition for too small systems

When doing distributed runs on the cluster, all output must be written in a concurrency-sage way. HDF5 with MPI communication looks like a reasonable choice.

There needs to be one simple and clearly documented way to build the Covariance Matrices: K(X_test,X_test) ; K(X_test,X_training) ; K(X_training,X_training) .

Redmod is too generic and SurUQ sounds too orcish. Instead of Surrogate the word parameter should be in focus. Suggestions:

• Paris - Parameter space regression including sensitivities
• Paras - Parameter space regression with analysis of sensitivities
• Parami - Parameter space regression with analysis ...
• Parma - Parameter space
• Supar - Surrogates and UQ via Parameter space regression
• Hypar - Handling your parameter space regression

In profit.yaml and LocalCommand troubles can arise with relative paths. The most logical way from the user would be, to relate all occurances of ../ to the study directory, i.e. replace ../ by ../../../ everywhere (study/run/XX/ instead of study/).

The best place to change this is directly in LocalCommand, since the place from which people access the Python API is usually also in study, as the profit.yaml.

The user would like to run his code independently from suruq. Therefore the user takes the following steps

1. Run profit in preprocessing mode to generate input file with a table of input parameters
2. Run code based on different parameter combinations in input file
3. Collect results in output file with format readable by profit
4. Do postprocessing in profit

Interfacing to input/output file should be easy and done by the user. For this purpose a txt and a hdf5 standard format will be supplied.

Dividing K_y by sigma_f^2 introduces an extra additive term -1/2 log(sigma_f^(-2)) in the NLL. This should be either justified or cancelled by adding it again.

In proFit, the hyperparameter vector used is: [ l=length-scale , sigma^2 = (sigma_n/sigma_f)^2 ] in order to normalize.

Adapt the written functions to this definition:

1. Replace l^2 by l in the functions' arguments.
2. Add a documentation for sigma .
3. Add an indication about the choice of sigma_f (ex: sigma_f always equal to 1) since it isn't a parameter of the kernel functions.
4. Handle the evantual different values for the same variable sigma_f given that it becomes an implicit argument for the functions which build the Covariance Matrices K(X_test,X_test) ; K(X_test,X_training) ; K(X_training,X_training) .

In the file profit.sur.backend the following functions do the same task:

• To return any of the covariance matrices K(X_train,X_train) ; K(X_test,X_test) ; K(X_test,X_train) :

1. kernels.gp_matrix(x0, x1, a, K)
2. gp.gp_matrix(x0, x1, a, K)
3. gp_functions.k(x0, x1, l)

• To return the covariance matrix K(X_train,X_train):

1. kernels.gp_matrix(x0, x1, a, K)
2. gp.gp_matrix_train(x, a, sigma_n) (the only difference is the added gaussian noise sigma_n on the diagonal of K(X_train,X_train))

see https://en.wikipedia.org/wiki/Bunch%E2%80%93Nielsen%E2%80%93Sorensen_formula

Right now, run folders are created as "0, 1, 2, 3, ..., 10, 11, ...". For better sorting in the file manager and console it should be standard to have "000, 001, 002, ..." which supports up to 1000 run folders. More generally one should put a configuration option ndigit in the run section of profit.yaml that defaults to three.

Implement possibility to shift x-axis of data such that two data sources are stitched together in the optimum way. This will require a hyperparameter that quantifies the relative shift.

Include

• Laplace approximation around MAP values (or multiple peaks) in hyperparameter space
• Variance due to (not necessarily simple) linear mean model according to Rasmussen 2.7

• Starting and managment of of runs on cluster
• Check out amzn/emukit: A Python-based toolbox of various methods in uncertainty quantification and statistical emulation: multi-fidelity, experimental design, Bayesian optimisation, Bayesian quadrature, etc.

Many codes rely on a standardized directory structure for each run. To automatically generate run directories the user provides a template file. Placeholders for input parameters in the template file are automatically replaced by values for a specific run. This feature should be usable for both, online and offline runs, and also dynamically generated parameter vectors.

For the work with Ulrich Callies from HZG a tool was developed to explore conditional distributions with one or more variables fixed in a certain range. Then the marginal distributions of the remaining variables are plotted as histograms and/or with a kernel density estimator. This way a high-dimensional probability distribution can be explored in an intuitive way.

Double brackets? Configurable?

input_mode(json)
separator = '{{'

input.json :
{
  'x': {{x}},
  'y': 4
}

config.h
int return_config()
{
  return {{u}}*x;
}


{{
 'x': {x},
 'y': 4
}}

Bring Config class and user interface in a clear form.

• enhance Config options (also solves #29)
• resolve path problems (also solves #19, #41)
• standardize code formatting
• update doc with Config options
• make variable functions easily customizable (also solves #21)
• optionally include Independent variable in inputs and treat as another parameter
• save output as .txt or .hdf5
• .py file with dict should also be a valid config file besides .yaml

After projecting to a low-order spectral basis (PCE for global UQ) one can model the residue by a GP with an additive kernel. This allows for modeling complex behavior and sensitivity analysis (ANOVA / Sobol indices)

Using scikit-learn, then scale up with PyTorch

See e.g. https://i-systems.github.io/teaching/ML/iNotes/15_Autoencoder.html