bracketjohn / kerndisc Goto Github PK
View Code? Open in Web Editor NEWAutomatic Kernel Discovery for Gaussian Processes
Automatic Kernel Discovery for Gaussian Processes
I'm not sure why this error is coming. gpflow installation has no issues
gpflow version: 2.0.3
tensorflow version: 2.2.0
Lloyd et al furthered kernel describability in his papar.
Read sections and alter kernels.
Adding new kernels will currently break tests, as all baske kernels are tested with grammar duvenaud, eventhough it only supports a subset.
gpflow
The new gpflow version changed its kernel hierarchy. Now a Sum
or Product
kernel has a kernels
child instead of all childs that are part of said Sum
or Product
. kernels
can then be used to get children.
This is breaking for kernDisc
and can currently not be supported.
AS Does the following:
def add_jitter_k(kernels, sd=0.1):
'''Adds random noise to all parameters - empirically observed to help when optimiser gets stuck'''
for k in kernels:
k.load_param_vector(k.param_vector + np.random.normal(loc=0., scale=sd, size=k.param_vector.size))
return kernels
def add_jitter(models, sd=0.1):
for a_model in models:
a_model.kernel = add_jitter_k([a_model.kernel], sd=sd)[0]
return models
Currently describe
returns it's description in an arbitrary order for all sub components. This should be improved to take into account some kind of metric. Examples of this include:
This currently does not hold:
def test_simplify_order(are_asts_equal):
"""Test whether order is also irrelevant."""
ast_one = Node(gpflow.kernels.Sum, full_name='Sum')
Node(gpflow.kernels.RBF, parent=ast_one, full_name='rbf')
Node(gpflow.kernels.Constant, parent=ast_one, full_name='constant')
ast_two = Node(gpflow.kernels.Sum, full_name='Sum')
Node(gpflow.kernels.Constant, parent=ast_two, full_name='constant')
Node(gpflow.kernels.RBF, parent=ast_two, full_name='rbf')
simpl_one = simplify(ast_one)
simpl_two = simplify(ast_two)
assert are_asts_equal(simpl_one, simpl_two)
To fix this:
if lvl_order_ast_one != lvl_order_ast_two:
return False
kerndisc
repositorykernDisc
construction into sub tasksautomated-statistician
Currently a user has to add a new file to the grammars
package and modify existing source code in order to create and use a new grammar.
This is not really "library like" and should be easier. Instead some method in kernDisc should be able to accept a method somewhere that is then used for expansion instead.
Graphing is often necessary to see what's going on and manually tune search.
This has to be done manually every time currently, a feature for this might be nice.
Multiple RBF kernels are currently already being merged, while multiple constant kernels are being ignored. This hempers performance, but shouldn't change the outcome.
max_depth
no_improvment
)0 1
like prior to oob optimizationIt seems to have an impact on a time series if it is very sparse and also stretches over a large time interval. Some of these examples include 39
data points over a duration of about 45k minutes (~75h).
This leads to:
kerndisc
tending to find constant
and white
kernels (as an end result) to deal with this,To tackle this behavior kerndisc
should be able to rescale X
to some interval during preprocessing.
This interval should still keep the same relative distance in between points.
[0, ..., 49]
for n=49
data points)Every sub function of simplify
currently uses pythons deepcopy
, in order to return an actual object.
It should be discussed whether this is necessary or whether hiding these lower levels of simplification API should be taken to full extend. This would mean creating a single deep copy in simplify
and working on this copy afterwards.
kernDisc
During search the tf
search graph bloats to thousands of variables. This happens due to the default graph not being cleaned/reset and immensely reduces search speed over time.
Is available in grammar string definition, but not in _IMPLEMENTED_KERNEL_EXPRESSIONS
.
There seems to be lazy_load errors and deprecation warnings after a fresh installation right now. This is most likely fixed by addressing #44.
Will wait for more people to experience this before fixing, as this library is in low maintanence mode right now.
def simplified(self):
k = self.copy()
k_prev = None
while not k_prev == k:
k_prev = k.copy()
k = k.collapse_additive_idempotency()
k = k.collapse_multiplicative_idempotency()
k = k.collapse_multiplicative_identity()
k = k.collapse_multiplicative_zero()
k = k.canonical()
return k
def add_random_restarts(models, n_rand=1, sd=4, data_shape=None):
new_models = []
for a_model in models:
for (kernel, likelihood, mean) in zip(add_random_restarts_single_k(a_model.kernel, n_rand=n_rand, sd=sd, data_shape=data_shape), \
add_random_restarts_single_l(a_model.likelihood, n_rand=n_rand, sd=sd, data_shape=data_shape), \
add_random_restarts_single_m(a_model.mean, n_rand=n_rand, sd=sd, data_shape=data_shape)):
new_model = a_model.copy()
new_model.kernel = kernel
new_model.likelihood = likelihood
new_model.mean = mean
new_models.append(new_model)
return new_models
Ugly stuff:
Research issue to check whether this might actually be a good idea
It might be better to use a Bayesian approach in combination with auto diff.
SPECIAL_KERNELS
are cp
and cw
kernels, currently accessible as SPECIAL_KERNELS
, which doesn't make a lot of sense. They are closer to structural
kernels or sth.
It seems like AS never really used this functionality, as best_models
is never set to anything else than None
.
if not best_models is None:
for a_model in best_models:
current_models = current_models + [a_model.copy()] + ff.add_jitter_to_models([a_model.copy() for dummy in range(exp.n_rand)], exp.jitter_sd)
Currently there is a kind of duality in the naming used internally:
gpflow
has it's own naming scheme, which capitalizes kernel names. However they go as far as also using camel case (ArcCosine
) and upper case (RBF
). Currently this is then abstracted in kerndisc
by the _kernels
module, which maintains a dict of kernel_name.lower(): kernel_class
mappings. This is useful for general coding, but also for grammar definition: The _grammar_duvenaud
allows for kernels to be excluded from being a base_kernel
. These can be passed by passing:
grammar_kwargs={'base_kernels_to_exclude': ['constant']}
as a parameter to discover
. Here it is easier for a user to just use lower case kernel names, than remembering all the different casing options.
Whether this justifies deviation from gpflow
is questionable though.
AS implements the ABCD framework for textual description/explanation of kernels generated by it. We also need this for simplified kernel expressions.
Relies on #11
explanation
package to explain and describe kernelsThere are some things that occurred during development and evaluation that might be improvements on the grammar of Duvenaud et al.
These improvements also don't seem to be covered in Lloyd et al.
It might be good to implement those in a modified
version of duvenauds grammar, some small changes were already made that move away from Duvenauds design, these must then also be moved.
_grammar_duvenaud_modified.py
grammarconstant
or white
as the "best" result)discover
might be unreasonableTODO: Add more improvements noticed during development here.
This also enables us to do textual description.
current_models = [model.simplified() for model in current_models]
current_models = ff.remove_duplicates(current_models)
with simplified being defined as:
def simplified(self):
k = self.copy()
k_prev = None
while not k_prev == k:
k_prev = k.copy()
k = k.collapse_additive_idempotency()
k = k.collapse_multiplicative_idempotency()
k = k.collapse_multiplicative_identity()
k = k.collapse_multiplicative_zero()
k = k.canonical()
return k
This epic can be closed once kerndisc
is ready for alpha.
Closing this will also move the transition of this project from using an internal, self management storyboard to an open source issue/participation style system.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.