Hi, Thanks for your nice work!
I am currently trying to follow your Split ImageNet-R benchmark but encounter some problems.
Firstly, I am not able to download the ImageNet-R dataset. I've got errors like this.
2022-10-25 16:43:39.764732: E tensorflow/core/platform/cloud/curl_http_request.cc:614] The transmission of request 0x668a800 (URI: https://www.googleapis.com/storage/v1/b/tfds-data/o/dataset_info%2Fimagenet_r%2F0.2.0?fields=size%2Cgeneration%2Cupdated) has been stuck at 0 of 0 bytes for 61 seconds and will be aborted. CURL timing information: lookup time: 0.004201 (No error), connect time: 0 (No error), pre-transfer time: 0 (No error), start-transfer time: 0 (No error)
It seems like a networking issue, so I access the uri(https://www.googleapis.com/storage/v1/b/tfds-data/o/dataset_info%2Fimagenet_r%2F0.2.0?fields=size%2Cgeneration%2Cupdated) through my web browser, but I got such a info:
{ "error": { "code": 404, "message": "No such object: tfds-data/dataset_info/imagenet_r/0.2.0", "errors": [ { "message": "No such object: tfds-data/dataset_info/imagenet_r/0.2.0", "domain": "global", "reason": "notFound" } ] } }
It seems that the dataset is not accessible at this time.
Secound, I tried to understand the codes you priveded in libml/input_pipeline.py and try to understand 2 things: (1) How to split the training and testing set from the original dataset. (2) What is the training order of the classes in continual learning tasks.
For (1), I found that the codes using TFDS split, in Line414-424 in input_pipeline.py.
I am not familiar with TFDS, Is this a stratified sampling (based on classes) or just a split based on the whole data?
For (2), I found that the codes permutate class order here, if the config.continual.rand_seed
is set. However in configs/imr_dualprompt.py, config.continual.rand_seed
is set to $-1$. Does this mean that the dualprompt conducts experiments in the order of natural numbers? (i.e,. 0, 1, 2, 3, ...)
Hope you counld kindly help me :)
Looking forward to your early reply and thanks again!