ae-foster / pytorch-simclr Goto Github PK

Hello. Thanks for your work.
By any means, would it be possible to provide the ImageNet pretrained checkpoint (without the linear evaluation head, I mean the first step only, with the projection head)? Thanks and Regards!

Hi, I met this error via executing python3 simclr.py --num-epochs 1000 --cosine-anneal --filename output.pth --base-lr 1.5. It seems it was caused by the LARS in torchlars. Below is my packages version:

numpy==1.18.1 
torchvision==0.6.0a0+35d732a 
pillow==8.2.0 
torchlars==0.1.2
tqdm==4.45.0
torch==1.5.1

And here is my traceback:

==> Preparing data..
Files already downloaded and verified
Files already downloaded and verified
Files already downloaded and verified
==> Building model..

Epoch: 0
Loss: ****           | 0/782 [00:00<?, ?it/s]Warning: This overload of add_ is deprecated:
	add_(Number alpha, Tensor other)
Consider using one of the following signatures instead:
	add_(Tensor other, *, Number alpha) (operator() at /opt/conda/conda-bld/pytorch_1591914742272/work/torch/csrc/utils/python_arg_parser.cpp:756)
Loss: ****           | 0/782 [00:12<?, ?it/s]
Traceback (most recent call last):
  File "/home/liuzhian/anaconda3/envs/pt1.5/lib/python3.6/contextlib.py", line 99, in __exit__
    self.gen.throw(type, value, traceback)
  File "/home/liuzhian/anaconda3/envs/pt1.5/lib/python3.6/site-packages/torchlars/lars.py", line 98, in hide_weight_decays
    yield weight_decays
  File "/home/liuzhian/anaconda3/envs/pt1.5/lib/python3.6/site-packages/torchlars/lars.py", line 145, in step
    self.apply_adaptive_lrs(weight_decays)
  File "/home/liuzhian/anaconda3/envs/pt1.5/lib/python3.6/site-packages/torchlars/lars.py", line 140, in apply_adaptive_lrs
    p.grad.add_(weight_decay, p.data)
RuntimeError: CUDA error: invalid device function

Process finished with exit code 139 (interrupted by signal 11: SIGSEGV)

Any idea to fix that? Thanks in advance!

Hello and thanks for your work. I would like to ask regarding the “on-the-fly” linear evaluation that you do after each pre-training epoch. In the tf original implementation, they cut the gradient during linear evaluation training to ensure that labels are not backpropgated to the ResNet-50. In PyTorch, this can be done by detach(). But I see that you only include a seperate optimizer for the linear evaluation and include the parameters of this linear evaluation layer in the optimizer (which means they will be updated only). That makes sense, but have you made sure that this is the case? (Have you made sure that using a seperate optimizer ensures the gradient does not flow to the resnet-50)?

Hello, first of all thank you for sharing great project.

I am wondering if you could share the checkpoints for the resnet50 trained on ImageNet, especially for CIFAR100?
There are lots of checkpoints of resnet50 pre-trained on ImageNet, but none of them have kernel size 3 for conv1,
as far as I searched.
I am really pleased if you can share the weights, since my local computer cannot train on ImageNet due to insufficient
gpus..

Thank you for reading.

To optimize the reg weight in L-BFGS we should use proper cross validation rather than validating on the test set

Hi @ae-foster. Thanks for your work. In simclr.py, you already do linear evaluation after training for some epochs using a constant reg_weight=1e-5:
clf = train_clf(X, y, net.representation_dim, num_classes, device, reg_weight=1e-5)

In that case, is there a big difference between the results obtained here (constant reg_weight=1e-5) and the results obtained in lbfgs_linear_clf.py? May you please tell me the score you got on CIFAR10 with constant reg_weight=1e-5?