sphinxteam / dnner Goto Github PK
View Code? Open in Web Editor NEWDeep Neural Networks Entropy from Replicas
Deep Neural Networks Entropy from Replicas
Could this library be used to get the maximum entropy of a network in a certain amount of bits knowing just the input and output sizes?
From my first read it seems that it requires a list of activation methods, however in my case the actual structure of the network is unknown. Would there be a specific set of activation methods which would maximize the entropy anyway?
Essentially my goal is to calculate the entropy of a given neural network, and the max entropy of a neural network in a given amount of memory.
Hi...
This is in regard to replication of Fig2 of your paper.
We know the identity for mutual information (MI)
I(X,Y)=H(Y)-H(Y|X) ; H=entropy
which for a deterministic system gives
I(X,Y)=H(Y) as H(Y|X)=0
Fig2 in paper plots H(Y) for such a system (neural net with random but fixed weights). While entropy can be negative, MI can not. And for deterministic mapping, H(Y) or MI should not be negative too?
Can you please explain this? or I am misinterpreting your experimental setup.
Implement iter_v
, eval_i
and eval_rho
for a binary prior,
great paper!
Right now specific numerical values need to be given. This is not ideal since in the multi-layer case we actually don't know a priori what is the maximum value
implement iter_theta
, iter_llmse
and eval_f
functions for Orthogonal ensemble
Right now the ReLU and hard-tanh activations are implemented in Cython and integrals are computed quite fast. The Probit however is implemented in pure Python and leads to a quite slow iteration.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.