Comments (12)
jacobrgardner
commented on June 12, 2024
In theory, since they are learned parameters it should not matter much. You can think of this as learning the mixture standard deviations instead of variances, if you'd like.
In practice, the only thing it might be worth changing it for is stability, since we've noticed that directly learning the lengthscale instead of the square root lengthscale for RBF kernels is occasionally more numerically stable.
If we could come up with a test case where learning the variances directly obviously gives more sensible results than learning the standard deviations, I wouldn't be opposed to changing it.
from gpytorch.
imsrgadich
commented on June 12, 2024
thank you for the quick response. I need to experiment with standard deviations and variances before concluding which is numerically stable.
One more question, why are the scale bounds from [-100,100]? Shouldn't they be positive?
from gpytorch.
imsrgadich
commented on June 12, 2024
Just noticed those are log bounds. makes sense now.
from gpytorch.
imsrgadich
commented on June 12, 2024
Just a note here. The minimum distance computation is unstable I believe.
Line 53: min_dist = torch.min(train_x_sort[:, 1:, :] - train_x_sort[:, :-1, :], 1)[0]
There are instances where minimum is 0 and this might not work. What are your thoughts on this?
from gpytorch.
gpleiss
commented on June 12, 2024
That's a good point. It might be good to add 1e-5 or something to the minimum distance.
from gpytorch.
imsrgadich
commented on June 12, 2024
I think taking the next minimum distance would be a good starting point and stands as a better prior.
from gpytorch.
gpleiss
commented on June 12, 2024
@imsrgadich any chance you'd be willing to make a PR with these proposed changes?
from gpytorch.
imsrgadich
commented on June 12, 2024
Yes. I can. :) I will send the changes.
On 23 Mar 2018 4:13 PM, Geoff Pleiss <[email protected]> wrote:@imsrgadich any chance you'd be willing to make a PR with these proposed changes?
—You are receiving this because you were mentioned.Reply to this email directly, view it on GitHub, or mute the thread.
from gpytorch.
imsrgadich
commented on June 12, 2024
I ran the unit tests after making the changes. here it is.
(gpytorch) /anaconda3/envs/gpytorch/lib/python3.6/site-packages/gpytorch-0.1-py3.6-macosx-10.7-x86_64.egg$ python -m unittest
/anaconda3/envs/gpytorch/lib/python3.6/site-packages/gpytorch-0.1-py3.6-macosx-10.7-x86_64.egg/test/examples/test_kissgp_additive_classification.py:84: UserWarning: invalid index of a 0-dim tensor. This will be an error in PyTorch 0.5. Use tensor.item() to convert a 0-dim tensor to a Python number
self.assertLess(mean_abs_error.data.squeeze()[0], 0.15)
./anaconda3/envs/gpytorch/lib/python3.6/site-packages/gpytorch-0.1-py3.6-macosx-10.7-x86_64.egg/test/examples/test_kissgp_additive_regression.py:110: UserWarning: invalid index of a 0-dim tensor. This will be an error in PyTorch 0.5. Use tensor.item() to convert a 0-dim tensor to a Python number
self.assertLess(mean_abs_error.data.squeeze()[0], 0.15)
./anaconda3/envs/gpytorch/lib/python3.6/site-packages/gpytorch-0.1-py3.6-macosx-10.7-x86_64.egg/test/examples/test_kissgp_gp_classification.py:75: UserWarning: invalid index of a 0-dim tensor. This will be an error in PyTorch 0.5. Use tensor.item() to convert a 0-dim tensor to a Python number
self.assertLess(mean_abs_error.data.squeeze()[0], 1e-5)
.WARNING:root:The input matches the stored training data. Did you forget to call model.train()?
WARNING:root:The input matches the stored training data. Did you forget to call model.train()?
./anaconda3/envs/gpytorch/lib/python3.6/site-packages/gpytorch-0.1-py3.6-macosx-10.7-x86_64.egg/test/examples/test_kissgp_gp_regression.py:92: UserWarning: invalid index of a 0-dim tensor. This will be an error in PyTorch 0.5. Use tensor.item() to convert a 0-dim tensor to a Python number
self.assertLess(mean_abs_error.data.squeeze()[0], 0.05)
../anaconda3/envs/gpytorch/lib/python3.6/site-packages/gpytorch-0.1-py3.6-macosx-10.7-x86_64.egg/test/examples/test_kissgp_kronecker_product_classification.py:82: UserWarning: invalid index of a 0-dim tensor. This will be an error in PyTorch 0.5. Use tensor.item() to convert a 0-dim tensor to a Python number
self.assertLess(mean_abs_error.data.squeeze()[0], 1e-5)
./anaconda3/envs/gpytorch/lib/python3.6/site-packages/gpytorch-0.1-py3.6-macosx-10.7-x86_64.egg/test/examples/test_kissgp_kronecker_product_regression.py:104: UserWarning: invalid index of a 0-dim tensor. This will be an error in PyTorch 0.5. Use tensor.item() to convert a 0-dim tensor to a Python number
self.assertLess(mean_abs_error.data.squeeze()[0], 0.1)
./anaconda3/envs/gpytorch/lib/python3.6/site-packages/gpytorch-0.1-py3.6-macosx-10.7-x86_64.egg/test/examples/test_kissgp_multiplicative_regression.py:105: UserWarning: invalid index of a 0-dim tensor. This will be an error in PyTorch 0.5. Use tensor.item() to convert a 0-dim tensor to a Python number
self.assertLess(mean_abs_error.data.squeeze()[0], 0.15)
./anaconda3/envs/gpytorch/lib/python3.6/site-packages/gpytorch-0.1-py3.6-macosx-10.7-x86_64.egg/test/examples/test_kissgp_variational_regression.py:107: UserWarning: invalid index of a 0-dim tensor. This will be an error in PyTorch 0.5. Use tensor.item() to convert a 0-dim tensor to a Python number
self.assertLess(mean_abs_error.data.squeeze()[0], 0.1)
./anaconda3/envs/gpytorch/lib/python3.6/site-packages/gpytorch-0.1-py3.6-macosx-10.7-x86_64.egg/gpytorch/functions/add_diag.py:14: UserWarning: invalid index of a 0-dim tensor. This will be an error in PyTorch 0.5. Use tensor.item() to convert a 0-dim tensor to a Python number
val = diag.squeeze()[0]
/anaconda3/envs/gpytorch/lib/python3.6/site-packages/gpytorch-0.1-py3.6-macosx-10.7-x86_64.egg/test/examples/test_multitask_gp_regression.py:100: UserWarning: invalid index of a 0-dim tensor. This will be an error in PyTorch 0.5. Use tensor.item() to convert a 0-dim tensor to a Python number
self.assertLess(mean_abs_error_task_1.data.squeeze()[0], 0.05)
/anaconda3/envs/gpytorch/lib/python3.6/site-packages/gpytorch-0.1-py3.6-macosx-10.7-x86_64.egg/test/examples/test_multitask_gp_regression.py:105: UserWarning: invalid index of a 0-dim tensor. This will be an error in PyTorch 0.5. Use tensor.item() to convert a 0-dim tensor to a Python number
self.assertLess(mean_abs_error_task_2.data.squeeze()[0], 0.05)
./anaconda3/envs/gpytorch/lib/python3.6/site-packages/gpytorch-0.1-py3.6-macosx-10.7-x86_64.egg/test/examples/test_simple_gp_classification.py:90: UserWarning: invalid index of a 0-dim tensor. This will be an error in PyTorch 0.5. Use tensor.item() to convert a 0-dim tensor to a Python number
assert(mean_abs_error.data.squeeze()[0] < 1e-5)
../anaconda3/envs/gpytorch/lib/python3.6/site-packages/gpytorch-0.1-py3.6-macosx-10.7-x86_64.egg/test/examples/test_simple_gp_classification.py:125: UserWarning: invalid index of a 0-dim tensor. This will be an error in PyTorch 0.5. Use tensor.item() to convert a 0-dim tensor to a Python number
self.assertLess(mean_abs_error.data.squeeze()[0], 1e-5)
.WARNING:root:The input matches the stored training data. Did you forget to call model.train()?
WARNING:root:The input matches the stored training data. Did you forget to call model.train()?
./anaconda3/envs/gpytorch/lib/python3.6/site-packages/gpytorch-0.1-py3.6-macosx-10.7-x86_64.egg/test/examples/test_simple_gp_regression.py:109: UserWarning: invalid index of a 0-dim tensor. This will be an error in PyTorch 0.5. Use tensor.item() to convert a 0-dim tensor to a Python number
self.assertLess(mean_abs_error.data.squeeze()[0], 0.05)
..WARNING:root:The input matches the stored training data. Did you forget to call model.train()?
./anaconda3/envs/gpytorch/lib/python3.6/site-packages/gpytorch-0.1-py3.6-macosx-10.7-x86_64.egg/test/examples/test_spectral_mixture_gp_regression.py:88: UserWarning: invalid index of a 0-dim tensor. This will be an error in PyTorch 0.5. Use tensor.item() to convert a 0-dim tensor to a Python number
self.assertLess(mean_abs_error.data.squeeze()[0], 0.15)
..............................................................................................................................................................
----------------------------------------------------------------------
Ran 176 tests in 27.435s
OK
from gpytorch.
jacobrgardner
commented on June 12, 2024
Is everything here resolved by #101, or are there concerns about stability with learning standard deviations instead of variances?
from gpytorch.
imsrgadich
commented on June 12, 2024
I haven't tested that yet! #101 solves the zero minimum distance problem. i will let you know once I have more about stability. Are there any guidelines (or datasets or anything) where you faced the problem of tuning the variance? I could use that in testing.
Thank you.
from gpytorch.
gpleiss
commented on June 12, 2024
Since the spectral mixture kernel was refactored by #258 - I'm going to close this for now. Please reopen if you discover new problems!
from gpytorch.
Related Issues (20)
- [Bug] Problems in the normalization and standardization of data HOT 1
- [Bug] Standardization of the output and inverse transform of standard deviation HOT 3
- [Docs] `get_fantasy_model` - are posterior covariances computed from scratch or using efficient cache updates? HOT 1
- [Bug] Bug in GP Regression with KeOps Kernels HOT 1
- [Bug] Extreme oscillation in loss
- [Bug] Extreme loss oscillation during training
- [Docs] Missing docs for HammingIMQKernel
- [Feature Request] Generic typing for scale kernels
- [Docs] Making sense of batch processing and tasks
- [Feature Request] Is it possible to work with Changepoint kernels?
- Nesting GPs; using the sufficient statistics from one GP as sufficient stats in another GP - variance goes to zero
- Label flattening fails with custom mean function from another GP HOT 1
- [Docs] Unexpected behavior setting kernel priors HOT 1
- [Feature Request] Allow `kwargs` to be passed to `ExactMarginalLogLikelihood.forward()` HOT 1
- [Bug] CUDA out of memory, strange numbers HOT 1
- [Docs] qKnowledgeGradient cpu usage HOT 1
- [Bug] Erroneous detaching with (custom?) mean
- [Bug] Multitask-ExactGPs seem to not use mBCG algorithm as Singletask-ExactGPs do
- [Feature Request] Choose which dimentions to differenciate with respect to in derivative multitask GPs
- [Bug] Error in tutorials and derivative GPs HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from gpytorch.