Comments (5)
What is the machine spec you use for training, e.g. how many GPU of which model and VRAM?
Does the training continue if you ignore the warning message and wait for a longer period of time?
Will the issue persist if you try to 1) change the number of workers in dataloader; 2) having more or fewer GPUs; 3) having larger or smaller batch size? 4) activating the training with mpirun
(see here) ?
The discussion here could be related.
Also checkout the discussion on Hovorod repo here and here.
from alpro.
I used 8 A100 GPUs and 1T VRAM for training. I think the allgather operation might cause the deadlock? Is there any way that might solve this?
1,0]<stderr>:[2022-04-20 02:35:43.587069: W horovod/common/stall_inspector.cc:105] One or more tensors were submitted to be reduced, gathered or broadcasted by subset of ranks and are waiting for remainder of ranks for more than 60 seconds. This may indicate that different ranks are trying to submit different tensors or that only subset of ranks is submitting tensors, which will cause deadlock. [1,0]<stderr>:Stalled ranks: [1,0]<stderr>:0: [allgather.noname.4624] [1,0]<stderr>:1: [allgather.noname.4624] [1,0]<stderr>:4: [allgather.noname.4624] [1,0]<stderr>:6: [allgather.noname.4624] [1,0]<stderr>:7: [allgather.noname.4624]
from alpro.
Could you try the possible fixes I suggested above and also check out the related threads?
This is not necessarily a deadlock. More likely it is just because some ranks are busier while keeping others waiting.
from alpro.
Thanks, I have figured out. When loading the images, I use while True to avoid non-existing images. Besides, I have a question about the weight decay factor. Did you use the weight decay during the both pre-training and finetune stage? Since I have noticed ALBEF uses the weight decay but not ALPRO.
from alpro.
Glad you've figured it out.
I'm closing the issue for now, as the original question has been resolved.
from alpro.
Related Issues (16)
- how to downsample webvid2m videos to 10% of the original FPS HOT 4
- Weight Decay HOT 3
- What is the input video feature dimension? HOT 1
- Using multiple GPUs vs single GPU HOT 1
- Inference HOT 3
- An academic issues on your paper HOT 2
- What dataset is used to pre-train the prompter? HOT 1
- Test on one GPU
- installing environment HOT 5
- I could not download the DiDeMo dataset
- After what epoch do you select the finetuning model?
- How to initialize the model with ALBEF when finetuning? HOT 2
- MSR-VTT dataset split HOT 1
- MSR-VTT Zero-shot HOT 1
- Zero-shot for MSRVTT retrieval results HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from alpro.