Comments (6)
@darbour is working on implementing that. Pull request should be incoming soon.
from gpytorch.
@Balandat I think I see two problems you are running in to:
-
Additive kernel needs to check if one output is Lazy and the other isn't, and wrap the non lazy thing in a NonLazyVariable. I have a fix for this up on a
white_noise_kernel
branch. -
At test time, the way we get all the matrices we need is to concatenate the train and test data and call the kernel once (see
exact_gp.py:106
). Thus it is attempting to add a length n diagional to a (n+t) x (n+t) matrix, so needs to be padded with zeros.
Upon reflection, point 2 should change in my opinion for exact GPs. Particularly for BayesOpt getting the t x t
component is wasteful. For Sparse GPs there's not really an issue because everything involved is Lazy. @gpleiss and I are discussing this now.
from gpytorch.
This should be implemented, per #152.
Feel free to reopen if any issues arise!
from gpytorch.
Ok, so I was trying to get this to work, but I'm having a hard time getting the correct behavior in all the different contexts (training, test, eval, forward, lazy, non-lazy, batch, non-batch, in usage with AdditiveKernel, etc.). @gpleiss, @jacobrgardner, any thoughts?
The common use case is to define a new AdditiveKernel, e.g. RBFKernel() + HeteroscedasticWhiteKernel(Y_train_target_var)
from __future__ import absolute_import
from __future__ import division
from __future__ import print_function
from __future__ import unicode_literals
import torch
from gpytorch.kernels import Kernel
from gpytorch.lazy.diag_lazy_variable import DiagLazyVariable
class HeteroscedasticWhiteKernel(Kernel):
def __init__(self, variances):
super(HeteroscedasticWhiteKernel, self).__init__()
self._variances = variances
def forward(self, x1, x2, **params):
if (
x1.shape == x2.shape
and x1.shape[0] == self._variances.shape[0]
and torch.equal(x1, x2)
):
return DiagLazyVariable(self._variances)
else:
return x1.new_zeros(x1.shape[0], x2.shape[0])
def __call__(self, x1_, x2_=None, **params):
if x2_ is None:
x2_ = x1_
return self.forward(x1_, x2_, **params)
from gpytorch.
Hmm so while #141 allows to subset the kernel matrix without fully computing it, I'm still not sure how to implement the logic for the heteroscedastic white kernel in a clean & efficient way.
With all the laziness and efficient transposing for diag computations there is a lot going on. It's also not super straightforward to debug if the error you're making somewhere doesn't surface until way later in the code b/c of the lazy evaluation :(
Anyway, here's is a very dumb indefficient barebones idea for the heteroschedastic white Kernel - this trains and evaluates, but will fail when evaluating .var()
b/c @jacobrgardner was trying to be smart in #144 😄 (.covar().evaluate()
works).
class HeteroscedasticWhiteKernel(Kernel):
def __init__(self, variances):
super(HeteroscedasticWhiteKernel, self).__init__()
self.register_buffer("variances", variances)
self._train_kern = torch.diag(self.variances).unsqueeze(0)
def forward(self, x1, x2):
if self.training:
return self._train_kern.repeat(x1.shape[-3], 1, 1)
else:
out = x1.new_zeros(x1.shape[-3], x1.shape[-2], x2.shape[-2])
ntrain = self._train_kern.shape[-1]
out[:, : ntrain, : ntrain] = self._train_kern.repeat(out.shape[0], 1, 1)
return out
I guess there are ways to be smarter here, like returning a DiagLazyVariable
in training mode, or add some sort of block lazy variable (which does not necessarily require all elements to be square...).
Any thoughts are welcome.
from gpytorch.
Awesome, thanks @jacobrgardner for doing this the right way!
from gpytorch.
Related Issues (20)
- [Bug] Problems in the normalization and standardization of data HOT 1
- [Bug] Standardization of the output and inverse transform of standard deviation HOT 3
- [Docs] `get_fantasy_model` - are posterior covariances computed from scratch or using efficient cache updates? HOT 1
- [Bug] Bug in GP Regression with KeOps Kernels HOT 1
- [Bug] Extreme oscillation in loss
- [Bug] Extreme loss oscillation during training
- [Docs] Missing docs for HammingIMQKernel
- [Feature Request] Generic typing for scale kernels
- [Docs] Making sense of batch processing and tasks
- [Feature Request] Is it possible to work with Changepoint kernels?
- Nesting GPs; using the sufficient statistics from one GP as sufficient stats in another GP - variance goes to zero
- Label flattening fails with custom mean function from another GP HOT 1
- [Docs] Unexpected behavior setting kernel priors HOT 1
- [Feature Request] Allow `kwargs` to be passed to `ExactMarginalLogLikelihood.forward()` HOT 1
- [Bug] CUDA out of memory, strange numbers HOT 1
- [Docs] qKnowledgeGradient cpu usage HOT 1
- [Bug] Erroneous detaching with (custom?) mean
- [Bug] Multitask-ExactGPs seem to not use mBCG algorithm as Singletask-ExactGPs do
- [Feature Request] Choose which dimentions to differenciate with respect to in derivative multitask GPs
- [Bug] Error in tutorials and derivative GPs HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from gpytorch.