ae-foster / pytorch-simclr Goto Github PK
View Code? Open in Web Editor NEWA PyTorch reproduction of 'A Simple Framework for Contrastive Learning of Visual Representations' by Ting Chen, et al.
License: MIT License
A PyTorch reproduction of 'A Simple Framework for Contrastive Learning of Visual Representations' by Ting Chen, et al.
License: MIT License
Hello. Thanks for your work.
By any means, would it be possible to provide the ImageNet pretrained checkpoint (without the linear evaluation head, I mean the first step only, with the projection head)? Thanks and Regards!
Hi, I met this error via executing python3 simclr.py --num-epochs 1000 --cosine-anneal --filename output.pth --base-lr 1.5
. It seems it was caused by the LARS in torchlars
. Below is my packages version:
numpy==1.18.1
torchvision==0.6.0a0+35d732a
pillow==8.2.0
torchlars==0.1.2
tqdm==4.45.0
torch==1.5.1
And here is my traceback:
==> Preparing data..
Files already downloaded and verified
Files already downloaded and verified
Files already downloaded and verified
==> Building model..
Epoch: 0
Loss: **** | 0/782 [00:00<?, ?it/s]Warning: This overload of add_ is deprecated:
add_(Number alpha, Tensor other)
Consider using one of the following signatures instead:
add_(Tensor other, *, Number alpha) (operator() at /opt/conda/conda-bld/pytorch_1591914742272/work/torch/csrc/utils/python_arg_parser.cpp:756)
Loss: **** | 0/782 [00:12<?, ?it/s]
Traceback (most recent call last):
File "/home/liuzhian/anaconda3/envs/pt1.5/lib/python3.6/contextlib.py", line 99, in __exit__
self.gen.throw(type, value, traceback)
File "/home/liuzhian/anaconda3/envs/pt1.5/lib/python3.6/site-packages/torchlars/lars.py", line 98, in hide_weight_decays
yield weight_decays
File "/home/liuzhian/anaconda3/envs/pt1.5/lib/python3.6/site-packages/torchlars/lars.py", line 145, in step
self.apply_adaptive_lrs(weight_decays)
File "/home/liuzhian/anaconda3/envs/pt1.5/lib/python3.6/site-packages/torchlars/lars.py", line 140, in apply_adaptive_lrs
p.grad.add_(weight_decay, p.data)
RuntimeError: CUDA error: invalid device function
Process finished with exit code 139 (interrupted by signal 11: SIGSEGV)
Any idea to fix that? Thanks in advance!
Hello and thanks for your work. I would like to ask regarding the “on-the-fly” linear evaluation that you do after each pre-training epoch. In the tf original implementation, they cut the gradient during linear evaluation training to ensure that labels are not backpropgated to the ResNet-50. In PyTorch, this can be done by detach(). But I see that you only include a seperate optimizer for the linear evaluation and include the parameters of this linear evaluation layer in the optimizer (which means they will be updated only). That makes sense, but have you made sure that this is the case? (Have you made sure that using a seperate optimizer ensures the gradient does not flow to the resnet-50)?
Hello, first of all thank you for sharing great project.
I am wondering if you could share the checkpoints for the resnet50 trained on ImageNet, especially for CIFAR100?
There are lots of checkpoints of resnet50 pre-trained on ImageNet, but none of them have kernel size 3 for conv1,
as far as I searched.
I am really pleased if you can share the weights, since my local computer cannot train on ImageNet due to insufficient
gpus..
Thank you for reading.
To optimize the reg weight in L-BFGS we should use proper cross validation rather than validating on the test set
Hi @ae-foster. Thanks for your work. In simclr.py, you already do linear evaluation after training for some epochs using a constant reg_weight=1e-5:
clf = train_clf(X, y, net.representation_dim, num_classes, device, reg_weight=1e-5)
In that case, is there a big difference between the results obtained here (constant reg_weight=1e-5) and the results obtained in lbfgs_linear_clf.py? May you please tell me the score you got on CIFAR10 with constant reg_weight=1e-5?
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.