wenwei202 / terngrad Goto Github PK
View Code? Open in Web Editor NEWTernary Gradients to Reduce Communication in Distributed Deep Learning (TensorFlow)
License: Apache License 2.0
Ternary Gradients to Reduce Communication in Distributed Deep Learning (TensorFlow)
License: Apache License 2.0
"Currently, distributed-node mode only supports 32bit gradients. It will take a while to hack the highly-encapsulated SyncReplicasOptimizer to integrate TernGrad. Keep updating"
-----mentioned in the doc
hi,wenwei, thanks for the amazing job. any progress or plan about this job?
thanks!
I try in caffe.
Alexnet ,single worker. I do like this:
1). ternarize all gradients in convolution layers and fc layers
2). ternarizing method: a). m=diff/diff_max(among one layer), b) generate random num : rand_blob_ ∈(0,1) c).compare m and rand_blob_ : if( m >rand_blob_[i] ) diff_[i] = diff_max; else diff_[i]=0
when training , first 500 epoch,it behaves well ,but than,it keep from ~6.1 to ~6.7 in the later 5000 iters, then I stop training.
Because I want to train in resnet or vgg or other nets , I want to know whether the code I change is right.
Thanks!
when eval
bazel-bin/inception/imagenet_eval
--data_dir ~/dataset/imagenet-data/
--net alexnet
--image_size 224
--batch_size 50
--checkpoint_dir /tmp/imagenet_train
--restore_avg_var True
--eval_dir /tmp/imagenet_eval
error : ExponentialMovingAverage not found in checkpoint
when I run Imagenet, Error in `/home/anaconda2/envs/tensorflow/bin/python': : double free or corruption (!prev): 0x0000000000c6c780
I do such step :
i). Download and generate ImageNet TFRecord
I do it like you say and get TFrecord 1024 train and 128 validation
ii). Build and run ImageNet
I do it like you say:
cd ${TERNGRAD_ROOT}/terngrad
bazel build inception/imagenet_train
bazel build inception/imagenet_eval
bazel-bin/inception/imagenet_train
--optimizer momentum
--net alexnet
--image_size 224
--num_gpus 2
--batch_size 256
--train_dir /tmp/imagenet_train
--data_dir ~/dataset/imagenet-data/
but at this step ,
Error in `/home/anaconda2/envs/tensorflow/bin/python': : double free or corruption (!prev): 0x0000000000c6c780.
I would appreciate for your answer. thank you
$ ./build_all.sh
bazel: /home/wew57/anaconda3/lib/libstdc++.so.6: version GLIBCXX_3.4.21' not found (required by bazel) bazel: /home/wew57/anaconda3/lib/libstdc++.so.6: version
GLIBCXX_3.4.20' not found (required by bazel)
bazel: /home/wew57/anaconda3/lib/libstdc++.so.6: version `CXXABI_1.3.8' not found (required by bazel)
@wenwei202
Could help me to understand your code?
The steps to ternarize gradient is as below:
Is that right? Thanks a lot!!
If i want to run terngrad on CIFAR10 dataset for distributed system (for example: 2 worker machine, 1 parameter server, i.e, 3 machines), can you please describe the steps to execute? In readme section, it shows workers under the same localhost, but I need to run them on different machines with different IP addresses.
Thank you.
When I ran cifar10_alexnet
on cifar10 dataset, I used terngrad and set FLAGS.floating_grad_epoch=0
(leave other arguments default), achieved top_1=86.04
after 300000 steps.
And then I want to see original accuracy without terngrad and set FLAGS.floating_grad_epoch=1
,but I got top_1=85.64
after 300000 steps, which is not consistent with results in paper. I want to make sure that I used in a correct way.
By the way, if I want to test it on other models like densenet, what arguments should I modified? (weight decay? base lr?)
I'm interested in terngrad and want to try it in my own project. I want to make sure the meaning of ternarize in your paper, but I can't find it in code.
Can u specific it for me? Which part of code is for formula (1) of your Terngrad paper?
Can we install the official TensorFlow binary version and run your code based on it? Does that mean you changed the source code of TensorFlow? I find that your algorithm just inserts some tensor operation in the computation graph, but have not changed the source code of TensorFlow.
On this code:
terngrad/terngrad/inception/bingrad_common.py
Lines 100 to 107 in ec4f75e
continue
after line 101?Since -1,0,1 are still represented using float32, how you guys achieved speedup as shown in the experiment of the paper?
my pc is mac , can terngrad run with cpu only?
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.