Comments (1)
Regarding
- Why don't you reduce losses across GPU ...
You don't need to and should not reduce loss manually. When you call loss.backward()
the gradient is reduced automatically.
As described in this answer
on loss.backward() all processes reduce their gradients. As loss.backward() returns, the gradients of your model parameters will be the same, and the optimizer in each process will perform the exact same update to the model parameters
Regarding
- When I run the code on single GPU and multiple GPUs ...
Distributed training does not always make training faster. In my opinion, the point of using distributed training is to have a larger effective batch size that is otherwise too large for a single GPU. It is normal that when you have a small batch size (i.e. 16 per GPU in your case), distributed is slower than a single GPU since there are communication overheads between different processes.
Also, distributed training could degrade performance. See this answer for detail. You might need to use a different learning rate.
from fixmatch-pytorch.
Related Issues (20)
- set_epoch with DistributedSampler HOT 1
- total numer of iterations HOT 2
- Purpose of Interleave HOT 3
- Invalid syntax in train.py HOT 1
- Performance not reproduced
- The RandAugment Implementation Is Wrong HOT 3
- Wrong use of AverageMeter HOT 1
- Lack of seed during random labeled image selection may be leading to better performance on training resumption HOT 5
- train failed when EMA mode is off HOT 5
- No Validation Set HOT 1
- v divided by 2 problem in Cutout HOT 1
- Training time without distributed training? HOT 2
- Why the batch_size of CIFAR-100 set to 64 (total number over 4 GPU) ?
- FixMatch for Multi-class Classification
- All labeled examples are as part of unlabeled data without using their labels HOT 1
- Can't reproduce the result with single GPU
- why add a interleave before the input? HOT 3
- Poor Results on CIFAR-100 with Smaller Models (e.g., ResNet34, ResNet18)
- RuntimeError: "nll_loss_forward_reduce_cuda_kernel_2d_index" not implemented for 'Int'
- 每次加载训练好的模型,结果都不一样
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from fixmatch-pytorch.