Different between probabilistic coding and VIB about spc HOT 4 CLOSED

You claim that your "probabilistic coding integrates probabilistic encoding and task prediction into one encoder module with the network θ, and applies a non-parametric operation to obtain prediction outputs."

But there seems to be no significant difference in your code compared with VIB. Would you tell me how you deal with probabilistic coding differently compared with VIB ?

Thank you for your attention.

In the implementation, probabilistic coding (PC) only has the $\theta$ parameter in the probabilistic coding module that needs to be learned, while VIB contains two sets of parameters that need to be learned: $\theta$ in the probabilistic encoding module and $\phi$ in the task decoder. For PC, the class "StochasticTaskModule" in the "model.py" file is a stochastic head, which directly decodes to the target task space through probabilistic operations. However, for VIB, in addition to the stochastic operations, a deterministic decoder containing learnable parameters $\phi$ (e.g., a learnable linear layer) must be used afterwards to map it for target task prediction. Totally, PC allows the model to jointly learn the probabilistic encoding and the task-specific prediction, rather than separating them into distinct modules as in VIB.

Therefore, the main differences are:

1. Non-parametric Task Prediction: After the probabilistic embedding, PC applies a non-parametric operation to directly obtain the prediction outputs, without the need for an additional deterministic decoder module (with learnable parameters $\phi$) as in VIB.
2. Dimension Mapping: In VIB, the stochastic module is typically performed with a fixed dimension, and a subsequent deterministic decoder is used to map this fixed-size embedding to the target task space. In PC, the stochastic module directly maps the original feature dimensions to the label space dimensions, without an intermediate fixed-size bottleneck.

I hope this helps clarify the key differences between PC and VIB.

from spc.

Thank you for your reply! However, I am still confused about how the non-parametric operation makes predictions.

Taking a classification task with 3 labels as an example, if a sample's output tensor after using pc is [-0.1, 0.1, 0.3], then using the non-parametric operation argmax (i.e., "predicted_label = torch.argmax(model_output[0], -1)" in the "main.py" file) would yield the index of the predicted label as 2. Since the pc process has already mapped the vector dimensions to the label space dimensions during the probabilistic process, there is no need to introduce an extra parameterized classifier network.

from spc.

Thank you for your reply!
However, I am still confused about how the non-parametric operation makes predictions.

from spc.

Thank you so much!

from spc.