Comments (3)
Hi @sunkun1997 - can you please share more information on your setup, ds_config, ds_report, and sample repro script?
from deepspeed.
repro script
I just use the https://github.com/microsoft/DeepSpeedExamples/tree/master/training/pipeline_parallelism example, but in order to fit our environment, I need to make a slight modification. In our environment, each node has four environment variables: the number of nodes WORLD_SIZE, the node rank RANK, the master node ip MASTER_ADDR, the port MASTER_PORT. So I modified run.sh
gpu=8
n=$(($WORLD_SIZE * $gpu))
start_rank=$(($RANK * $gpu))
end_rank=$((($RANK + 1) * $gpu))
for ((i=$start_rank; i<$end_rank; i++))
do
{
LOCAL_RANK=$i WORLD_SIZE=$n MASTER_ADDR=$MASTER_ADDR MASTER_PORT=$MASTER_PORT python train.py \
--p $gpu \
--steps=200
}&
done
wait
And modified the main of train.py
if __name__ == '__main__':
import json
args = get_args()
with open('./ds_config.json', 'r') as f:
args.deepspeed_config = json.loads(f.read())
args.local_rank = int(os.environ['LOCAL_RANK'])
args.world_size = int(os.environ['WORLD_SIZE'])
deepspeed.init_distributed(dist_backend=args.backend, rank=args.local_rank,
world_size=args.world_size, auto_mpi_discovery=False)
torch.cuda.set_device(args.local_rank % 8)
if args.pipeline_parallel_size == 0:
train_base(args)
else:
train_pipe(args)
Then Each node run the run.sh.
ds_report
raise Cuda error
Traceback (most recent call last):
File "train.py", line 165, in <module>
train_pipe(args)
File "train.py", line 131, in train_pipe
net = PipelineModule(layers=join_layers(net),
File "/home/ray/anaconda3/lib/python3.8/site-packages/deepspeed/runtime/pipe/module.py", line 201, in __init__
self.to(get_accelerator().device_name(self.local_rank))
File "/home/ray/anaconda3/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1145, in to
return self._apply(convert)
File "/home/ray/anaconda3/lib/python3.8/site-packages/torch/nn/modules/module.py", line 797, in _apply
module._apply(fn)
File "/home/ray/anaconda3/lib/python3.8/site-packages/torch/nn/modules/module.py", line 820, in _apply
param_applied = fn(param)
File "/home/ray/anaconda3/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1143, in convert
return t.to(device, dtype if t.is_floating_point() or t.is_complex() else None, non_blocking)
RuntimeError: CUDA error: invalid device ordinal
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.
from deepspeed.
By the way, If I modify the start train.py with
LOCAL_RANK=$((i % $gpu)) RANK=$i WORLD_SIZE=$n MASTER_ADDR=$MASTER_ADDR MASTER_PORT=$MASTER_PORT python train.py
and modify the start distributed environment with
deepspeed.init_distributed(dist_backend="nccl", rank=args.rank, world_size=args.world_size, auto_mpi_discovery=False)
.
It looks like the nodes can't communicate with each other and raise
Traceback (most recent call last):
File "train.py", line 166, in <module>
train_pipe(args)
File "train.py", line 137, in train_pipe
trainset = cifar_trainset(args.local_rank)
File "train.py", line 30, in cifar_trainset
dist.barrier()
File "/home/ray/anaconda3/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py", line 3328, in barrier
work = default_pg.barrier(opts=opts)
torch.distributed.DistBackendError: NCCL error in: ../torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:1275, internal error, NCCL version 2.14.3
ncclInternalError: Internal check failed.
from deepspeed.
Related Issues (20)
- ++[Viral@Vídeo]@@ Fotos Do Aviao Que Caiu Em Vinhedo Portal Zacarias!
- ++@@@!!full video fotos do aviao que caiu em vinhedo portal zacarias
- ++@@Ver Noelia Minchan Video Viral Telegram Full Video
- ++VIRAL Project 2025 mainly led by ex-Trump officials leaked videos reveal HOT 1
- ++>>Portal Zacarias Vídeo Do Avião Caiu Em Vinhedo Aviação Em Vinhedo Leaks On Twitter E Mais lue HOT 1
- (full video!!)VIDEO DE JHONIER LEAL MUERTE HOT 1
- !!~~Viral!!FULL ULTIMO VIDEO DE MAURICIO LEAL NUEVO HOT 1
- ++!!EL ASCENSOR DE ALBERTO HOT 1
- ++EXCLUSIVO!!VEA EL VIDEO Completo del ESTlLISTA MAURICIO LEAL MINUTOS Antes de MORIR
- ((√)) فيديو سكس هدير عبد الرازق المقطع كامل شاهد قبل الحذف
- ++Blake Lively and Justin Baldoni ARGUE In Leaked Video It Ends With Us Video Amid Feud Rumours HOT 1
- ++[[Original]] Subhashree Sahu sex Videos Full LEaked Video is trending on social media on Twitter HOT 1
- (full video!!) katie price poo on stomach & katie price photo poo on stomach
- ++Đi Giữa Trời Rực Rỡ Tập 10 HOT 1
- ++[Watch Leaked Video] katie price poo on stomach & katie price photo poo on stomach
- New Video Katie Price Poo On Stomach & Katie Price Picture Poo Twitter HOT 1
- ++((AA!!Support)) How can I get in touch with American Airlines fast?{{Guidence}}
- [Di@l_A.A]] How To Get Through To American Airlines Fast?
- How Can I Get In Touch With American Fast? [Live Person]
- nv-nightly CI test failure
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from deepspeed.