Question about the results on the survey paper about online-continual-learning HOT 4 CLOSED

1. Why do the results have 18 tasks instead of 20?

A large volume of CL works still tune hyperparameters in an offline manner by sweeping over the whole data sequence and selecting the best hyperparameter set with grid-search on a validation set. After that, metrics are reported on the test set with the
selected set of hyperparameters. This tuning protocol violates the online CL setting where a classifier can only make a single pass over the data, which implies that the reported results in the CL literature may be too ideal and cannot be reproduced in real online CL applications.

Thus, we use the first two tasks for hyper-parameter tuning. For more information, please check Section.4 in our paper.

2. Is there a specified random seed to arrange the class order for these two datasets?

Since the task order and task composition may impact the performance, we take the average over multiple runs
for each experiment with different task orders and compositions to reliably assess the robustness of the methods. For CORe50-NC and CORe50-NI, we follow the number of runs (i.e., 10), task order and composition provided by the authors. For Split CIFAR-100 and Split MiniImagenet, we average over15 runs, and the class composition in each task is randomly selected for each run. The random seed for each run is the same as the run id.

Please let me know if your questions are answered.

from online-continual-learning.

1. Why do the results have 18 tasks instead of 20?

A large volume of CL works still tune hyperparameters in an offline manner by sweeping over the whole data sequence and selecting the best hyperparameter set with grid-search on a validation set. After that, metrics are reported on the test set with the
selected set of hyperparameters. This tuning protocol violates the online CL setting where a classifier can only make a single pass over the data, which implies that the reported results in the CL literature may be too ideal and cannot be reproduced in real online CL applications.

Thus, we use the first two tasks for hyper-parameter tuning. For more information, please check Section.4 in our paper.

1. Is there a specified random seed to arrange the class order for these two datasets?

Since the task order and task composition may impact the performance, we take the average over multiple runs
for each experiment with different task orders and compositions to reliably assess the robustness of the methods. For CORe50-NC and CORe50-NI, we follow the number of runs (i.e., 10), task order and composition provided by the authors. For Split CIFAR-100 and Split MiniImagenet, we average over15 runs, and the class composition in each task is randomly selected for each run. The random seed for each run is the same as the run id.

Please let me know if your questions are answered.

Thanks for reply, that helps a lot. Btw, for the provided code, dose the cl type of 'nc' refers to the class-incremental setting (single-head classifier)?

from online-continual-learning.

We implemented class-incremental (nc, stands for new class) and domain-incremental (ni, stands for new instance).
Both of them are single-headed.

from online-continual-learning.

Close now, feel free to re-open if you have more questions.

from online-continual-learning.