Comments (3)
easonyang1996
commented on May 13, 2024
3
I tried the default vit configuration on my own dataset, the loss didn't drop, too. Then I used smaller 'depth' and 'dim', it worked. Maybe you can try to modify the model architecture.
from vit-pytorch.
mxsurui
commented on May 13, 2024
2
also trouble with this issue, is there any tips about how to train vit on ImageNet
from vit-pytorch.
YounkHo
commented on May 13, 2024
1
I tried the default vit configuration on my own dataset, the loss didn't drop, too. Then I used smaller 'depth' and 'dim', it worked. Maybe you can try to modify the model architecture.
After I decreased the dim from 1024 to 256, num_layers and num_head from 8 to 2, the number of parameters has reduced to 679016 which is 100 times smaller than the orignal network setting. But it still did not work, is it stilll too many params or can you share your param setting? Thanks!
Here are my logs of the 5th epoch:
2020-12-03 06:39:21,075 - meter.py[line:34] - INFO: Epoch: [3][ 0/6673] Time 2.843 ( 2.843) Data 2.753 ( 2.753) Loss 6.9075e+00 (6.9075e+00) Acc@1 0.00 ( 0.00) Acc@5 0.00 ( 0.00)
2020-12-03 06:44:17,016 - meter.py[line:34] - INFO: Epoch: [3][ 500/6673] Time 1.920 ( 0.596) Data 1.860 ( 0.531) Loss 6.9071e+00 (6.9077e+00) Acc@1 0.00 ( 0.11) Acc@5 0.00 ( 0.54)
2020-12-03 06:49:20,588 - meter.py[line:34] - INFO: Epoch: [3][1000/6673] Time 0.082 ( 0.602) Data 0.022 ( 0.537) Loss 6.9074e+00 (6.9077e+00) Acc@1 0.00 ( 0.10) Acc@5 0.52 ( 0.53)
2020-12-03 06:54:45,241 - meter.py[line:34] - INFO: Epoch: [3][1500/6673] Time 0.658 ( 0.618) Data 0.600 ( 0.552) Loss 6.9077e+00 (6.9077e+00) Acc@1 0.00 ( 0.10) Acc@5 1.04 ( 0.52)
2020-12-03 07:00:25,042 - meter.py[line:34] - INFO: Epoch: [3][2000/6673] Time 0.107 ( 0.633) Data 0.019 ( 0.567) Loss 6.9081e+00 (6.9077e+00) Acc@1 0.00 ( 0.09) Acc@5 0.00 ( 0.52)
2020-12-03 07:06:20,696 - meter.py[line:34] - INFO: Epoch: [3][2500/6673] Time 0.106 ( 0.649) Data 0.020 ( 0.580) Loss 6.9073e+00 (6.9077e+00) Acc@1 0.00 ( 0.09) Acc@5 0.00 ( 0.51)
2020-12-03 07:12:51,716 - meter.py[line:34] - INFO: Epoch: [3][3000/6673] Time 0.110 ( 0.671) Data 0.021 ( 0.602) Loss 6.9077e+00 (6.9077e+00) Acc@1 0.00 ( 0.10) Acc@5 1.56 ( 0.51)
2020-12-03 07:19:59,395 - meter.py[line:34] - INFO: Epoch: [3][3500/6673] Time 0.861 ( 0.697) Data 0.800 ( 0.628) Loss 6.9069e+00 (6.9077e+00) Acc@1 0.00 ( 0.10) Acc@5 1.04 ( 0.50)
2020-12-03 07:27:54,941 - meter.py[line:34] - INFO: Epoch: [3][4000/6673] Time 0.111 ( 0.729) Data 0.019 ( 0.660) Loss 6.9075e+00 (6.9077e+00) Acc@1 0.00 ( 0.09) Acc@5 0.52 ( 0.50)
2020-12-03 07:37:17,196 - meter.py[line:34] - INFO: Epoch: [3][4500/6673] Time 0.108 ( 0.773) Data 0.018 ( 0.704) Loss 6.9070e+00 (6.9077e+00) Acc@1 0.00 ( 0.09) Acc@5 1.04 ( 0.50)
2020-12-03 07:48:40,004 - meter.py[line:34] - INFO: Epoch: [3][5000/6673] Time 0.117 ( 0.832) Data 0.018 ( 0.763) Loss 6.9078e+00 (6.9077e+00) Acc@1 0.52 ( 0.10) Acc@5 0.52 ( 0.50)
2020-12-03 08:02:10,024 - meter.py[line:34] - INFO: Epoch: [3][5500/6673] Time 0.110 ( 0.904) Data 0.019 ( 0.834) Loss 6.9072e+00 (6.9077e+00) Acc@1 0.52 ( 0.09) Acc@5 0.52 ( 0.50)
2020-12-03 08:16:39,075 - meter.py[line:34] - INFO: Epoch: [3][6000/6673] Time 5.605 ( 0.973) Data 5.545 ( 0.903) Loss 6.9067e+00 (6.9077e+00) Acc@1 0.00 ( 0.10) Acc@5 1.04 ( 0.50)
2020-12-03 08:31:23,624 - meter.py[line:34] - INFO: Epoch: [3][6500/6673] Time 0.109 ( 1.035) Data 0.019 ( 0.963) Loss 6.9074e+00 (6.9077e+00) Acc@1 0.00 ( 0.10) Acc@5 1.04 ( 0.50)
2020-12-03 08:36:24,116 - vit_train.py[line:193] - INFO: ++++++++++++++++++++++++Training for one epoch +++++++++++++++++++++++++++++++++++
2020-12-03 08:36:27,453 - meter.py[line:34] - INFO: Test: [ 0/261] Time 3.336 ( 3.336) Loss 6.9085e+00 (6.9085e+00) Acc@1 0.00 ( 0.00) Acc@5 0.00 ( 0.00)
2020-12-03 08:39:17,536 - vit_train.py[line:153] - INFO: * Acc@1 0.100 Acc@5 0.500
2020-12-03 08:39:17,537 - vit_train.py[line:207] - INFO: save checkpoint.................
2020-12-03 08:39:20,401 - meter.py[line:34] - INFO: Epoch: [4][ 0/6673] Time 2.782 ( 2.782) Data 2.714 ( 2.714) Loss
2020-12-03 08:44:16,053 - meter.py[line:34] - INFO: Epoch: [4][ 500/6673] Time 0.912 ( 0.596) Data 0.843 ( 0.517) Loss 6.9077e+00 (6.9077e+00) Acc@1 0.00 ( 0.10) Acc@5 1.04 ( 0.50)
2020-12-03 08:49:24,166 - meter.py[line:34] - INFO: Epoch: [4][1000/6673] Time 2.126 ( 0.606) Data 2.041 ( 0.527) Loss 6.9080e+00 (6.9077e+00) Acc@1 0.00 ( 0.09) Acc@5 0.00 ( 0.48)
2020-12-03 08:55:15,165 - meter.py[line:34] - INFO: Epoch: [4][1500/6673] Time 4.573 ( 0.638) Data 4.507 ( 0.564) Loss 6.9084e+00 (6.9077e+00) Acc@1 0.00 ( 0.09) Acc@5 0.00 ( 0.48)
2020-12-03 09:01:45,592 - meter.py[line:34] - INFO: Epoch: [4][2000/6673] Time 0.659 ( 0.674) Data 0.601 ( 0.602) Loss 6.9079e+00 (6.9077e+00) Acc@1 0.00 ( 0.09) Acc@5 0.00 ( 0.49)
2020-12-03 09:08:26,356 - meter.py[line:34] - INFO: Epoch: [4][2500/6673] Time 0.109 ( 0.699) Data 0.019 ( 0.627) Loss 6.9072e+00 (6.9077e+00) Acc@1 0.00 ( 0.10) Acc@5 0.00 ( 0.49)
2020-12-03 09:15:38,532 - meter.py[line:34] - INFO: Epoch: [4][3000/6673] Time 0.310 ( 0.727) Data 0.225 ( 0.654) Loss 6.9079e+00 (6.9077e+00) Acc@1 0.00 ( 0.09) Acc@5 0.00 ( 0.48)
2020-12-03 09:23:49,919 - meter.py[line:34] - INFO: Epoch: [4][3500/6673] Time 1.382 ( 0.763) Data 1.294 ( 0.690) Loss 6.9079e+00 (6.9077e+00) Acc@1 0.00 ( 0.09) Acc@5 0.00 ( 0.48)
2020-12-03 09:33:21,487 - meter.py[line:34] - INFO: Epoch: [4][4000/6673] Time 3.452 ( 0.811) Data 3.392 ( 0.737) Loss 6.9071e+00 (6.9077e+00) Acc@1 0.00 ( 0.10) Acc@5 0.52 ( 0.48)
2020-12-03 09:44:09,826 - meter.py[line:34] - INFO: Epoch: [4][4500/6673] Time 0.089 ( 0.865) Data 0.022 ( 0.792) Loss 6.9078e+00 (6.9077e+00) Acc@1 0.00 ( 0.10) Acc@5 0.52 ( 0.49)
2020-12-03 09:57:39,927 - meter.py[line:34] - INFO: Epoch: [4][5000/6673] Time 3.217 ( 0.940) Data 3.156 ( 0.868) Loss 6.9077e+00 (6.9077e+00) Acc@1 0.00 ( 0.10) Acc@5 0.00 ( 0.49)
2020-12-03 10:13:39,403 - meter.py[line:34] - INFO: Epoch: [4][5500/6673] Time 0.102 ( 1.029) Data 0.019 ( 0.957) Loss 6.9079e+00 (6.9077e+00) Acc@1 0.00 ( 0.10) Acc@5 0.00 ( 0.49)
2020-12-03 10:31:27,529 - meter.py[line:34] - INFO: Epoch: [4][6000/6673] Time 0.108 ( 1.121) Data 0.019 ( 1.049) Loss 6.9074e+00 (6.9077e+00) Acc@1 0.00 ( 0.10) Acc@5 0.00 ( 0.49)
2020-12-03 10:49:44,418 - meter.py[line:34] - INFO: Epoch: [4][6500/6673] Time 0.085 ( 1.204) Data 0.019 ( 1.131) Loss 6.9085e+00 (6.9077e+00) Acc@1 0.00 ( 0.10) Acc@5 0.00 ( 0.49)
2020-12-03 10:56:05,657 - vit_train.py[line:193] - INFO: ++++++++++++++++++++++++Training for one epoch +++++++++++++++++++++++++++++++++++
2020-12-03 10:56:14,022 - meter.py[line:34] - INFO: Test: [ 0/261] Time 8.363 ( 8.363) Loss 6.9058e+00 (6.9058e+00) Acc@1 0.00 ( 0.00) Acc@5 0.00 ( 0.00)
2020-12-03 10:59:03,169 - vit_train.py[line:153] - INFO: * Acc@1 0.100 Acc@5 0.500
2020-12-03 10:59:03,170 - vit_train.py[line:207] - INFO: save checkpoint.................
2020-12-03 10:59:05,996 - meter.py[line:34] - INFO: Epoch: [5][ 0/6673] Time 2.762 ( 2.762) Data 2.681 ( 2.681) Loss 6.9081e+00 (6.9081e+00) Acc@1 0.00 ( 0.00) Acc@5 0.52 ( 0.52)
2020-12-03 11:05:38,925 - meter.py[line:34] - INFO: Epoch: [5][ 500/6673] Time 0.089 ( 0.790) Data 0.021 ( 0.713) Loss 6.9079e+00 (6.9077e+00) Acc@1 0.00 ( 0.11) Acc@5 0.52 ( 0.51)
2020-12-03 11:12:54,162 - meter.py[line:34] - INFO: Epoch: [5][1000/6673] Time 0.079 ( 0.830) Data 0.019 ( 0.754) Loss 6.9076e+00 (6.9077e+00) Acc@1 0.00 ( 0.12) Acc@5 0.52 ( 0.51)
2020-12-03 11:20:38,089 - meter.py[line:34] - INFO: Epoch: [5][1500/6673] Time 0.099 ( 0.863) Data 0.024 ( 0.787) Loss 6.9078e+00 (6.9077e+00) Acc@1 0.52 ( 0.12) Acc@5 0.52 ( 0.51)
2020-12-03 11:28:54,603 - meter.py[line:34] - INFO: Epoch: [5][2000/6673] Time 2.232 ( 0.895) Data 2.162 ( 0.819) Loss 6.9078e+00 (6.9077e+00) Acc@1 0.00 ( 0.12) Acc@5 1.04 ( 0.52)
2020-12-03 11:37:28,869 - meter.py[line:34] - INFO: Epoch: [5][2500/6673] Time 0.096 ( 0.922) Data 0.020 ( 0.846) Loss 6.9080e+00 (6.9077e+00) Acc@1 0.00 ( 0.11) Acc@5 0.00 ( 0.51)
2020-12-03 11:46:50,222 - meter.py[line:34] - INFO: Epoch: [5][3000/6673] Time 0.087 ( 0.955) Data 0.020 ( 0.879) Loss 6.9076e+00 (6.9077e+00) Acc@1 0.00 ( 0.11) Acc@5 0.00 ( 0.50)
from vit-pytorch.
Related Issues (20)
- structural 3D ViT HOT 4
- Saving and loading model seems to be regressing to lower performance HOT 1
- vit_pytorch -> cross_vit.py(mistake) HOT 1
- Using vision transformers for different image resolutions HOT 1
- Potential regression with PT 2.0 and CUDA 12.2/CuDNN 8.9.4 HOT 1
- Not correctly understanding the Multi Head Attention part of the ViT implementation... HOT 3
- CvT with 1 channel input data HOT 2
- Layernorm in Cross attention HOT 4
- how to train HOT 2
- Questions about distill_loss HOT 1
- Trouble loading ViT - Dino structure for channels>3?
- Question regarding 1d fft use HOT 1
- Masking attention with batches
- can we use CvT model for segmentation?
- Multi-target Regression Question
- Problems regarding training 3D Vision transformer : model does not converge
- Add implementation of LongVit HOT 4
- A question with ViT 3d
- Cuda memory for 3D VIT HOT 2
- PyPi page markdown render HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from vit-pytorch.