Doubt about the correctness of testing accuracy of fine-tuning stage and adaption stage. about closerlookfewshot HOT 4 CLOSED

The feature extractor is frozen, and they re-train a linear classifier from scratch for each testing episode.
If you take a look at methods/baselinefinetune.py and methods/meta_template.py, you could see they re-initialize a linear layer for a set_forward_adaptation call in each test episode, which means the fine-tuned weight is independent between different test episode.

• CloserLookFewShot/methods/baselinefinetune.py

  Line 28 in e03aca8

  linear_clf = nn.Linear(self.feat_dim, self.n_way)

• CloserLookFewShot/methods/meta_template.py

  Line 102 in e03aca8

  linear_clf = nn.Linear(self.feat_dim, self.n_way)

from closerlookfewshot.

Thank you for your reply. So in each episode, it only use support set to fine-tune the linear layer and use query set to test. I guess the main reason to fine-tune and test in the unit of episode is to keep consistent with mete-testing stage for fair comparison.
But I think if use the whole support set for all episodes in test dataset to fine-tune(and not only the classifier but also the backbone) and use the whole query set for all episodes in test dataset to evaluation for baseline and baseline++ model, which means fine-tune in the traditional way, it may get higher accuracy. And if further use the validation dataset to fine-tune the baseline and baseline++ model, to meta-training the meta-learning model, it may further improve accuracy.
Hoping for your opinion!

from closerlookfewshot.

The purpose of sampling different episode is to evaluate the performance under few-shot learning scenario. For example, to evaluate 3-way 5-shot tasks, we want to know the performance when the training data (of target task) only contain 3 classes and 5 examples per class. Thus, there is no reason to use examples other than the support set of an episode to finetune the model in most cases.

from closerlookfewshot.

OK, I understand. Thank your for your explanation.

from closerlookfewshot.