Add proper support for priors about gpytorch HOT 7 CLOSED

I think my recommendation would just be to eliminate bounds entirely, and make priors a purely optional feature (with a smoothed uniform prior taking the place of bounds as an optional feature). Andrew pointed out (I think correctly) that hyperparameter priors are overwhelmingly used for BayesOpt compared to other GP applications.

Any lingering problems that bounds currently solved can probably be solved by good documentation suggesting priors in certain cases, and maybe even helpful error messages where, when we encounter NaNs that we know are caused by ridiculous hyperparameters, we suggest a prior.

from gpytorch.

How do you feel about allowing to specify one of the two for each parameter:

1. a user-defined prior (basically conforming to a simple API that more or less only requires outputting a log_prob given a parameter value)
2. bounds (or log-bounds), which we then internally translate into a smoothed-out uniform prior with full support but rapidly decreasing values outside the bounds?

Also, would you agree that it is more intuitive to put a prior on the actual lengthscales, rather than the log-lengthscales?

from gpytorch.

The bounds in GPyTorch are at least partially a left over from back when we had lots of numerical stability issues that we didn't know how to deal with due to there not being a lot of prior work to draw from that exclusively used MVM based inference methods. As the package matured, we've mostly found much more clever solutions to many of these problems.

Therefore, another option to consider is removing bounds as a requirement entirely and making priors purely optional. Barring this option, your suggestion seems sensible to me.

I absolutely that it seems more natural to place priors on the parameters directly, especially since things like uniform priors have really strange implications in log space. Do you have an interface in mind to deal with the fact that we register log parameters but define the prior over the parameters? Specifically, how when we define the prior do we specify whether the parameters being registered should be exponentiated or potentially whatever other transformation?

from gpytorch.

One other thing to consider is changing slightly the interface for initializing parameters to be more tied to the priors. Keeping a parameter initialized to a default of 0 but placing a uniform prior on (100,200) would lead to some serious trouble that may be tough to spot.

from gpytorch.

So I don't have a particular interface in mind, still exploring how to best do this.

I think getting rid of parameter bounds altogether may actually be desirable, since it would avoid a lot of the edge case handling that would be necessary if both parameter bounds and priors were present. Instead of having bounds on a parameter, why not be explicit and require a UniformPrior over the respective range? If we were to do this, we may want to internally work just in the actual (non-log) parameter space.

from gpytorch.

cc @darbour

from gpytorch.

Done in #139

from gpytorch.