Comparison with few/zero-shot performance and Task Specific prompts. about l2p HOT 3 CLOSED

Hi Prateek,

Thanks for your interest in our work and your great questions!

1. I did not evaluate the standard few-shot and zero-shot setting for the base ViT model, since they are not directly related to continual learning. However, one can treat one of the baselines -- GDumb -- as a few-shot learning method. To my understanding, GDumb trains on the buffered data only, which is the subsampled full dataset.

2. I believe I conducted such experiments, but not showed it in the L2P paper. However, I have to say task-specific prompts are not directly applicable to class-incremental learning, since you have no idea how to choose task-specific prompts at inference when the task ID is unknown. If I remember correctly, task-specific prompt did a bit worse than L2P on CIFAR100, but is comparable or better than L2P on 5-datasets. Intuitively, task-specific prompt does not have the ability to share knowledge between tasks, so that might be the reason. Nevertheless, feel free to add your experiments if you are interested and correct me if I am wrong.

Best,
Zifeng

from l2p.

The reason why I asked for zero/few shot number is that I suspect that the model might perform well even when we prepend some random vectors or slightly trained vectors along with the input image because the ViT model is pre-trained on ImageNet21k and CIFAR100 is very similar to it but easier. If the model has good zero/few-shot performance then it invalidates some of the claims made in the paper regarding continual learning and preventing forgetting.
Furthermore, if the performance is not better than task specific prompts then the claim regarding sharing knowledge might not be well supported as well. This comparison is completely skipped in the paper which I guess if the most important method to compare with.

Thanks,
Prateek

from l2p.

DualPrompt validates that share prefixtuning is better in paper 5.4？

from l2p.