Why use custom models? Cannot reproduce with torchvision model about unlearnable-examples HOT 3 CLOSED

ResNet Implementation:
There is no specific reason for not using torchvision models. The implementation of ResNet is just copy-paste from other projects. See https://github.com/kuangliu/pytorch-cifar/blob/master/models/resnet.py
The differences between the two implementations are the kernel size, padding and stride for conv1. The torchvision's model will reduce the h,w after conv1. My guess is torchvision's model is meant for ImageNet with larger resolutions.

Performance issue:
The visualizing of the noise is by normalizing it to [0,255] so that it can be displayed, so the black image may mean the noise is all 0s. Have you examined the noise by printing them? I have reproduced the result as you mentioned by replacing torchvision's model. Validation accuracy is around 40%, but unable to produce the black image of the noise. I tested our implementation of ResNet to generate the noise and applies to torchvision.models.resnet18, the validation accuracy is around 25%. For this difference in the implementation, my guess is the hyperparameters for noise generation need an update, i.e. the termination and number of steps for the model.

Time issue:
The results shown in the notebook was run with RTX-2080-Ti. It took around 3 minutes per epoch for generating the noise, with total 10 epochs. So around 30 minutes in total. Removing mlconfig has no effect on speed. I do not see any other reason that could cause the slowdown on your machine. Can you provide more details on the hardware? Also, because the torchvision models reduce the h,w resolutions, it makes sense that this model can run faster.

from unlearnable-examples.

Thank you for the explanation.

For the visualizing noise issue, I discovered that some noises have a minimum value = maximum value, which would cause a division by 0 when running your normalizing code. Interestingly, it did not trigger a division by zero error, although the resulting normalized noise are very close to 0. Handling the minimum value = maximum value case by not dividing by the difference visualizes the noises correctly.

For the performance issue, may I clarify if your validation accuracy is computed on clean images? I can confirm that the noise generated by your Res-18 model also works well on a torchvision Res-18.

For the timing issue, I used Google Colab (Tesla T4), so maybe it just is caused by the specific GPU used. While mlconfig is not a culprit, I suggest that it be removed from the init file to avoid import errors for people cloning your repo, or include it in a requirements file.

from unlearnable-examples.

Thanks for your intrest in our work.

Yes, it is the validation accuracy computed on clean images. There may be slight variations between the noise generation with yours. But the using the torchvision.models.resnet18 to generate the noise is less effective.

Thanks for your suggestion on mlconfig. I will update the repo.

from unlearnable-examples.