hyper params values about alia HOT 6 CLOSED

I hope I can calm you down: I really believe you used the test set to report the metrics / the bug was added recently. In my experiments, this (wrong) vall ACC reached 100%. Which is not the result of the paper.
Either way, good luck!

from alia.

Hello! Apologies for the late reply, I have been traveling all this month.

I pushed some fixes to the code and ran across some inconsistencies with the new results and the results from our paper. You are correct that we were still reporting test accuracy which is correct, but we were choosing our checkpoints on the train set rather than the val set which resulted in evaluating on checkpoints early on in training. I will put the links of the WandB page below but TL;DR is that our result still consistently beats the baseline, but the other data augmentation techniques seem to be performing better than before, even outperforming our method on some benchmarks like CUB. I have been working on tuning some hyper-parameters and adding in some new datasets to verify that the gains from ALIA are due to the prompt diversity and filtering as stated in the paper, and we will (hopefully) get a good idea of whether or not we need to make some significant changes to the claim in our paper by the end of next week.

Thank you again for pointing out an important error! Feel free to try out any of the configs and let me know if you are getting different results than are displayed on the project, or you can wait until I figure out exactly what is going on :)

WandB links:

• Planes
• iWildCam
• CUB

from alia.

AND on another topic,
I have found that on the data loader creation lines, you create the val loader out of the trainset.
trainset, valset, testset = get_filtered_dataset(args, transform, val_transform, flags)
trainloader = torch.utils.data.DataLoader(
trainset, batch_size=args.data.batch, shuffle=True, num_workers=2)
valloader = torch.utils.data.DataLoader(
trainset, batch_size=args.data.batch, shuffle=False, num_workers=1)
testloader = torch.utils.data.DataLoader(
testset, batch_size=args.data.batch, shuffle=False, num_workers=1)

I assume it's not on purpose?
If it is, what is the rational?

from alia.

Apologies for the late reply.

As for the HPs, the ones in the paper should be the ones to use and yes, batch size 128 is used for all exes.

As for making the valloader out of the trainset, that is definitely an error on my end, thank you so much for catching it! I am currently rerunning the experiments to see if that changes the results, I suspect this bug was also there when running the paper experiments.

from alia.

@lisadunlap , Hey, any updates? for now, I stopped experimenting until you approve that all is ok :)

from alia.

Okie dokie! I was able to rerun the experiments, looks like the planes dataset has a fair amount of issues with it so we switched it out with waterbirds. I am uploading the generated data now for it but the wandb projects and checkpoints for all the datasets are in the readme. Please let me know if you have any further issues, closing this for now :)

from alia.