First try at implementing `ngboost.distn.Bernoulli` class about ngboost HOT 6 CLOSED

Hi @wptmdoorn -- I actually had a bit of spare time this evening and finished an implementation of the Bernoulli distribution.

See the code [bernoulli.py], and I threw your example into [classification.py] which should work now. Let me know if you're happy with that.

See the derivation below; hopefully it's helpful in case there's any interest in implementing a Categorical distribution. 😄

from ngboost.

Hi @wptmdoorn -- thank you for the contribution!

You definitely have the main ideas correct, but let me make a suggestion:

My guess for why numerical issues are arising is due to the choice of parameterization -- the probability of a Bernoulli distribution must be bounded between [0,1] but in our framework the base learners can return unbounded outputs over the reals.

A common trick is to instead parameterize the Bernoulli in terms of the logit of the distribution, i.e. logit=log(p/(1-p)) and then p=1/(1+exp(-logit)). These functions are implemented in scipy here:

https://docs.scipy.org/doc/scipy/reference/generated/scipy.special.logit.html
https://docs.scipy.org/doc/scipy-0.14.0/reference/generated/scipy.special.expit.html

This will nll and D_nll functions will need to be modified accordingly, as well as the fisher_info function. No need to worry about the functions crps, crps_metric, or fisher_info_cens.

Your idea for fit is correct -- we'd just want to fit the marginal distribution (though this should change with the parameterization). Happy to follow up if any questions come up.

from ngboost.

@tonyduan -- how about incorporating Brier loss (as an analogue to CRPS) for Bernoulli (and similarly L_2 for Categorical)? Having a common name for all three would be nice :)

from ngboost.

Hi @tonyduan thank you for coming back to me on a such a short notice. Very much appreciated! I will start working on parameterizing the Bernoulli in terms of the logit of the distribution.

from ngboost.

@tonyduan wow, impressive, thanks a lot. I will start experimenting with Bernoulli class as soon as possible. Thank you also for the full mathematical derivation; these are really, really helpful for me personally.

On a side note: I guess we can close this issue now, unless you want to keep this until the bernoulli.distns.Categorical class is implemented.

from ngboost.

Let's open a new issue for Categorical

from ngboost.