about adding parameters about itaml HOT 5 CLOSED

Hi, @JoyHuYY1412, just a average over all the tasks. No we havent used any normalizations.

See

from itaml.

Hi, @JoyHuYY1412, just a average over all the tasks. No we havent used any normalizations.

See

iTAML/learner_task_itaml.py

Line 163 in e56e72b

ll = torch.stack(reptile_grads[i])

Thank you for your reply.
So if different tasks have parameters of different scales, e.g, A>>B, it seems the averaged network will be biased toward A. So when we try to recover the network for task B, we assume the memory samples of B can help fit? I don't know do I understand correctly.

from itaml.

Yes, that's one reason the classifier for a task is trained only in the inner loop. Also, to minimize the biased model we take a weighted average of the weights as we progress. Yes, finetuning with the memory samples helps to get a better model.

from itaml.

Yes, that's one reason the classifier for a task is trained only in the inner loop. Also, to minimize the biased model we take a weighted average of the weights as we progress.
Thank you so much. I have two more questions.

1. I read the pseudo code (algorithm 1) in your paper, so after we update phi in line 14, does the theta used in line 7 for task 1 is initialized from the updated phi ? and then task 2 initialized its theta from task 1?
2. If so, does this operation somehow relieve the imbalance between tasks? Since after each update in the outer loop, the backbone network is reset.

from itaml.

1. No, in the inner loop theta for all tasks are initialized with last updated phi, and once we learned all thetas we combine them to get the new phi, which is later used to initialize thetas for the next batch.
2. Yes, the outer loop meta updated tries to minimize the forgetting, while imbalance is minimized mostly because of the exponential averaging of the weights.

from itaml.