ucla-labx / distbelief Goto Github PK

traces:
[1] 98051 segmentation fault /opt/homebrew/bin/python3.5 first_node.py

libc++abi.dylib: terminating with uncaught exception of type std::__1::system_error: mutex lock failed: Invalid argument [1] 98060 abort /opt/homebrew/bin/python3.5 second_node.py

i.e. different clients can train different models

• ideally be able to run this on something like starcluster

curious how much time is spent is send/recv for tensors vs training

• perhaps vary based on lr?

for example for parameter_request

So that clients and servers can use different optimization techniques. Right now it looks like -lr*grad is being sent over, which is sgd only

its not going to be faster :( but i hope it will be

Planning on implementing a decentralized version of asyncSGD.

Current approach is just a ring-based network, but ill investigate different structures.

Hey, I am trying to implement distbelief on a single machine with multiple-GPUs but neither GLOO nor NCCL support send or recv, so is there a workaround this? In general is point to point communication possible between GPUs using any of these backend?