sphinxteam / dnner Goto Github PK

Could this library be used to get the maximum entropy of a network in a certain amount of bits knowing just the input and output sizes?

From my first read it seems that it requires a list of activation methods, however in my case the actual structure of the network is unknown. Would there be a specific set of activation methods which would maximize the entropy anyway?

Essentially my goal is to calculate the entropy of a given neural network, and the max entropy of a neural network in a given amount of memory.

Hi...
This is in regard to replication of Fig2 of your paper.

We know the identity for mutual information (MI)
I(X,Y)=H(Y)-H(Y|X) ; H=entropy
which for a deterministic system gives
I(X,Y)=H(Y) as H(Y|X)=0

Fig2 in paper plots H(Y) for such a system (neural net with random but fixed weights). While entropy can be negative, MI can not. And for deterministic mapping, H(Y) or MI should not be negative too?

Can you please explain this? or I am misinterpreting your experimental setup.

Implement iter_v, eval_i and eval_rho for a binary prior, $P_0 (x) = \rho \delta(x) + (1 - \rho) \delta(x - 1)$.

great paper!

Right now specific numerical values need to be given. This is not ideal since in the multi-layer case we actually don't know a priori what is the maximum value $\rho$ it assumes for each layer.

implement iter_theta, iter_llmse and eval_f functions for Orthogonal ensemble

Right now the ReLU and hard-tanh activations are implemented in Cython and integrals are computed quite fast. The Probit however is implemented in pure Python and leads to a quite slow iteration.