Initialization weights about squeezenet HOT 7 CLOSED

Try LSUV init https://github.com/ducha-aiki/LSUVinit/blob/master/tools/extra/lsuv_init.py

from squeezenet.

It turns out that I needed to reduce the learning rate. After reducing the learning rate by 10x and increasing the effective batch size by 2x, I was able to train from scratch. Less extreme measures are most likely sufficient.

@ducha-aiki thanks. LSUV does seem to have a slightly faster start; in this case, my biggest problem was the learning rate.

from squeezenet.

@liuyipei with LSUV I was able to converge with big lr. But it is good, that other ways work as well :)
See https://github.com/ducha-aiki/caffenet-benchmark/blob/master/prototxt/architectures/SqueezeNet128_lsuv.prototxt

from squeezenet.

I like how you have trainval and solver in one file. Does Caffe accept that
as-is, or did you customize Caffe to allow it?

Anyway, it looks convenient!
On Mar 3, 2016 8:50 PM, "Dmytro Mishkin" [email protected] wrote:

@liuyipei https://github.com/liuyipei with LSUV I was able to converge
with big lr. But it is good, that other ways work as well :)
See
https://github.com/ducha-aiki/caffenet-benchmark/blob/master/prototxt/architectures/SqueezeNet128_lsuv.prototxt

—
Reply to this email directly or view it on GitHub
#4 (comment).

from squeezenet.

@forresti it accepts, see example in caffe master branch: https://github.com/BVLC/caffe/blob/master/examples/mnist/lenet_consolidated_solver.prototxt

from squeezenet.

@liuyipei
One more thing: I've run into a few problems with cuDNN and numerical correctness. I recommend trying a training run with cuDNN disabled, and seeing if you still get divergence.

from squeezenet.

@liuyipei
Update: We have been experimenting with solver configurations, and we have identified a configuration that converges more reliably. We just committed it to SqueezeNet-master: 0bc03d9

from squeezenet.