Comments (3)
We have not encountered any problems with the beta
... When you say they are "very large", could you be more specific? What is the mean and variance with or without MorphNet (possibly using more than one regularization strength).
In our code we are using tf.contrib version, it should behave the same. We never had problems using these settings (which I think are the same as in the paper:
'batch_norm_params': {
'decay': 0.9997,
'scale': True,
'center': True,
'epsilon': 1e-5,
'param_regularizers': {
'beta_decay': 0.0,
'gamma_decay': 5e-5
}
}
let me know if it helps.
from morph-net.
Sorry, I didn't talk about my question clearly,it's not training problem, the situation is one of the bn with morphnet, the morphnet limits the gamma to very small value (ex. 1e-8, 1e-7), but in the same channel the beta is like 0.86, 0.89(or larger), maybe it's still an important feature(or not), but morphnet cut that channel?
Or should I let my center = false to avoid this problem?
And I also got the question about the bn output, does have any example shows the value of Channel with and without morphnet?Just want to confirm it works, thank you:).
One of my bn beta mean and var with morphnet is -0.099896885 and 0.12298851.
from morph-net.
I would consider this channel to be dead as the beta is not data dependent and can be absorbed in the bias of the next layer (the bias of a filter is not beta but beta * sum(weights_i)
.
Furthermore, the next layer also has BN so this bias is actually automatically removed.
With that being said, I will be very interested in seeing what happens if you try experiments without beta
. For example I have in mind these four experiments:
flop_regularizer = {on, off} x center = {true, false}
.
Please do consider reporting back, I think it interesting to see what happens even if the result is the same.
from morph-net.
Related Issues (20)
- the morphnet can act on some layers of the model?such as act on backbone. HOT 7
- OpRegularizerManager could not handle ops HOT 7
- Getting this Error HOT 1
- Getting error like this HOT 2
- In this case, MorphNet is getting stuck because it doesn't know how to handle the Shape op. You can try a couple options:
- Applying Morph-net while training training Yolov3 using darknet HOT 1
- Query on Morph Net Structure Exporter HOT 2
- What's the tf version this repo used?
- Can morph-net output a pruned model file? HOT 3
- Bad structure HOT 3
- MorphNet not working on google colab HOT 1
- Can I use MorphNet to slim my model trained by Pytorch? HOT 1
- Regularizer Loss remains zero / BatchToSpaceND HOT 1
- Multiple alive files output HOT 1
- Batch Norm layers Assertion Error
- Latency regularizer for a target hardware. HOT 5
- LayerNorm not support?
- Performance issue in the definition of create_test_input, examples/slim/nets/resnet_v1_test.py HOT 1
- Tensorflow 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from morph-net.