Comments (7)
m not sure if u understand, if not u can directly use this code:
acc1, acc5 = accuracy(output, target, topk=(1, 5))
...
dist.all_reduce(acc1, op=dist.reduce_op.SUM)
...
top1.update(acc1[0] / 4 , images.size(0))
Btw, I don't think we really need to calculate the average accuracy during training, which is the waste of time.
from pytorch-distributed.
Nope, if u really need it, u can use .share_memory()
to share a Tensor's memory.
All in all, most distributed lib only help u to handle the synchronization of data, parameters, and gradient.
from pytorch-distributed.
Could you give a specific example?
Thanks!
from pytorch-distributed.
hm, m afraid that's not right.
Here are two ways to communicate between torch.multiprocessing
:
- if u dont care the running results, u can use
share_memory_ like
this, which is more faster:
import time
import random
import torch
import torch.distributed as dist
import torch.multiprocessing as mp
def evaluate(rank):
torch.cuda.manual_seed(rank)
local_acc = torch.randn(1)[0].cuda(rank)
print("local_acc:", local_acc)
return local_acc
def main_worker(gpu, ngpus_per_node, args):
dist.init_process_group(backend='nccl', init_method='tcp://127.0.0.1:23456', world_size=4, rank=gpu)
local_acc = evaluate(gpu)
global_acc, global_count = args['global_acc'], args['global_count']
global_acc += local_acc.cpu()
global_count += 1
print("global_acc:", global_acc / global_count)
if __name__ == '__main__':
global_acc = torch.tensor(.0)
global_count = torch.tensor(.0)
global_acc.share_memory_()
global_count.share_memory_()
args = {
'global_acc': global_acc,
'global_count': global_count
}
mp.spawn(main_worker, nprocs=4, args=(4, args))
But if you really need to synchronize the accuracy, I suggest this kind of implement or something else using all_reduce
:
import time
import random
import torch
import torch.distributed as dist
import torch.multiprocessing as mp
def evaluate(rank):
torch.cuda.manual_seed(rank)
local_acc = torch.randn(1)[0].cuda(rank)
print("local_acc:", local_acc)
return local_acc
def main_worker(gpu, ngpus_per_node, args):
dist.init_process_group(backend='nccl', init_method='tcp://127.0.0.1:23456', world_size=4, rank=gpu)
local_acc = evaluate(gpu)
dist.all_reduce(local_acc, op=dist.reduce_op.SUM)
global_acc = local_acc / ngpus_per_node
print("global:", global_acc)
if __name__ == '__main__':
args = {}
mp.spawn(main_worker, nprocs=4, args=(4, args))
from pytorch-distributed.
Thanks!
Should the code synchronize the loss between each gpus before loss.backward()
? Or the backward function will synchronize automatically?
from pytorch-distributed.
No. That's DistributedDataParallel's job.
Wrap your model with DistributedDataParallel
and just call backward()
as usual. During the backwards pass, gradients from each node are averaged (same as your saying "synchronize the loss") and parameters are synchronized automatically.
Check it here: https://github.com/pytorch/pytorch/blob/46539eee0363e25ce5eb408c85cefd808cd6f878/torch/nn/parallel/distributed.py#L378-L382
from pytorch-distributed.
thanks
from pytorch-distributed.
Related Issues (20)
- 大佬,请问如何指定gpu训练
- apex 并行加速中的data_prefetcher normalize为什么做两次? HOT 1
- 您好,我想请问一下Pytorch 在多进程分布式训练的时候,一开始加载数据集dataloader的时候,由于是多进程的,就会导致多个进程一起加载dataloader,这些dataloder全部加载到内存中使得内存爆炸,请问有没有什么办法解决呢
- 多卡显存占用不均衡 HOT 1
- No need to divide with nprocs again HOT 1
- Windows 提示Distributed package doesn't have NCCL "Distributed package doesn't have NCCL built in HOT 1
- how can i modify it
- how can i modify it ?
- RuntimeError: Timed out initializing process group in store based barrier on rank: 2, for key: store_based_barrier_key:1 (world_size=2, worker_count=4, timeout=0:30:00) HOT 2
- 2080Ti多卡训练,GPU利用率忽高忽低,这个怎么回事 HOT 2
- Can't pickle <function main_worker
- Connections time out
- DDP掉精度 HOT 3
- distributed.py seed HOT 6
- [Discussion] mp: duplicate of torch.cuda.set_device(local_rank) and images = images.cuda(local_rank, non_blocking=True) HOT 4
- About saving model in the multi-GPU training. HOT 3
- 使用mp.spawn启动比torch.distributed.launch慢很多 HOT 2
- 关于损失backward问题 HOT 1
- 请问使用DistributedSampler,各个GPU的数据是如何分配的?是连续(互不相同)的还是相同的? HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from pytorch-distributed.