Uncertainty estimation for synthetic data (3 class Gaussian distr) about dirichlet-prior-networks HOT 8 OPEN

Please fork and show modified commit diff or sent me a diff (assuming you've built on top of this code). I'm not going to check your code. I never got figure (c) vs figure (f) - although I think I could've eventually with some tuning (or discovering hidden bugs).

I'm not currently actively working on this, but I hope you have some luck with the same.

from dirichlet-prior-networks.

Thanks a lot. I have uploaded my files and added a description. I have not yet added the differential entropy code as I'm still trying to figure out why won't the precision output by the DPN (<10) have a similar precision as the target distribution (100).

from dirichlet-prior-networks.

How are you computing the "precision of the target distribution"?

from dirichlet-prior-networks.

from dirichlet-prior-networks.

I'm not possibly the expert here, I come from a programming background than stats - I coded this up to integrate with a pytorch repo I had.

1. The network trained for predicting precisions of not 100 well, which would be the OOD data (50% at any update step), where all the catagoricals are equiprobable from the Dirichlet produced. Drawing parallels with how this works for classification with softmax probabilities, I think you will need to set a threshold to create a decision rule to figure in domain or out of domain. Similar to binary classification where even if you train with 0-1 labels, you get a probability like 0.7, which using a decision rule of a cutoff of 0.5 you classify as a 1. Reiterating, I didn't manage to get the figures c and f which makes the in-domain vs out of domain distinction, which relates to precision again - but my hunch is it's more to do with hyperparameter tuning now than a bug in this code.

2. Label smoothing is to smooth the optimization surface, the predictions are at inference expected to be the actual value. Label smoothing doesn't come unless you use the loss function, which is not used during inference. (I had a hard time getting the network to converge without smoothing).

from dirichlet-prior-networks.

from dirichlet-prior-networks.

http://github.com/KaosEngineer/PriorNetworks-OLD/blob/181f74c556a39a1d7aff163b49380612fb34855d/prior_networks/dirichlet/run/step_train_synth.py#L34-L35

I was going with the values I found in the repo, so try the above default. Please do let me know how the experiments turn out and if you need any help with my part of the code.

from dirichlet-prior-networks.

from dirichlet-prior-networks.