Comments (6)
I hope I can calm you down: I really believe you used the test set to report the metrics / the bug was added recently. In my experiments, this (wrong) vall ACC reached 100%. Which is not the result of the paper.
Either way, good luck!
from alia.
Hello! Apologies for the late reply, I have been traveling all this month.
I pushed some fixes to the code and ran across some inconsistencies with the new results and the results from our paper. You are correct that we were still reporting test accuracy which is correct, but we were choosing our checkpoints on the train set rather than the val set which resulted in evaluating on checkpoints early on in training. I will put the links of the WandB page below but TL;DR is that our result still consistently beats the baseline, but the other data augmentation techniques seem to be performing better than before, even outperforming our method on some benchmarks like CUB. I have been working on tuning some hyper-parameters and adding in some new datasets to verify that the gains from ALIA are due to the prompt diversity and filtering as stated in the paper, and we will (hopefully) get a good idea of whether or not we need to make some significant changes to the claim in our paper by the end of next week.
Thank you again for pointing out an important error! Feel free to try out any of the configs and let me know if you are getting different results than are displayed on the project, or you can wait until I figure out exactly what is going on :)
WandB links:
from alia.
AND on another topic,
I have found that on the data loader creation lines, you create the val loader out of the trainset.
trainset, valset, testset = get_filtered_dataset(args, transform, val_transform, flags)
trainloader = torch.utils.data.DataLoader(
trainset, batch_size=args.data.batch, shuffle=True, num_workers=2)
valloader = torch.utils.data.DataLoader(
trainset, batch_size=args.data.batch, shuffle=False, num_workers=1)
testloader = torch.utils.data.DataLoader(
testset, batch_size=args.data.batch, shuffle=False, num_workers=1)
I assume it's not on purpose?
If it is, what is the rational?
from alia.
Apologies for the late reply.
As for the HPs, the ones in the paper should be the ones to use and yes, batch size 128 is used for all exes.
As for making the valloader out of the trainset, that is definitely an error on my end, thank you so much for catching it! I am currently rerunning the experiments to see if that changes the results, I suspect this bug was also there when running the paper experiments.
from alia.
@lisadunlap , Hey, any updates? for now, I stopped experimenting until you approve that all is ok :)
from alia.
Okie dokie! I was able to rerun the experiments, looks like the planes dataset has a fair amount of issues with it so we switched it out with waterbirds. I am uploading the generated data now for it but the wandb projects and checkpoints for all the datasets are in the readme. Please let me know if you have any further issues, closing this for now :)
from alia.
Related Issues (6)
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from alia.