Reproducing Plain Model Baseline Accuracies about openlongtailrecognition-oltr HOT 9 CLOSED

Thank you very much for reporting this. We will take a look at this and get back to you as soon as possible!

from openlongtailrecognition-oltr.

@ssfootball04
Sorry, I think you might need to reopen this issue. We tested both our release code and our testing code, it seems there are some bugs in our released version, which caused this issue. We thought this might was because of the learning rate, but it was not. Sorry for the bother, we will fix it as soon as possible.

Thank you very much.

from openlongtailrecognition-oltr.

Thank you for looking into it, awaiting the fix :).

p.s - I don't have permission to reopen the issue

from openlongtailrecognition-oltr.

Reopen the issue for the upcoming updates.

from openlongtailrecognition-oltr.

@ssfootball04 Thank you very much for your patience. The bug is fixed now. The first stage should work just like the paper reports. I forgot to put model.train() to the beginning of each epoch when cleaning the code. We are very sorry about this. I am also running stage 2 experiment. If there is any more changes, I will let you know. I think I will close this issue now. If you could not reproduce stage 2 results, please open a new issue. Thank you again for reporting.

from openlongtailrecognition-oltr.

@ssfootball04 And we changed the stage2 configuration file to 60 epochs as well so the stage 2 results are reproducible. Thank you very much!

from openlongtailrecognition-oltr.

Hi! I am still getting slightly lower accuracies for both open and closed-set testing. Some possible places where I might be going wrong?

from openlongtailrecognition-oltr.

@drcege @ssfootball04 @sanjanamallya Hello guys, as described in https://github.com/zhmiao/OpenLongTailRecognition-OLTR/blob/master/run_networks.py#L297 , We finally debugged the published code and current open set imagenet performance is:

============
Phase: test

Evaluation_accuracy_micro_top1: 0.361
Averaged F-measure: 0.501
Many_shot_accuracy_top1: 0.442 Median_shot_accuracy_top1: 0.352 Low_shot_accuracy_top1: 0.175

==========

This is higher than we reported in the paper. We updated some of the modules with clone() method, and set use_fc in the first stage to False. These changes will lead us to the proper results. Please have a try.

For Places, the current config won't work either. The reason why we could not get the reported results is that we forget that on the first stage, we actually did not freeze the weights. We only freeze the weights on the second stage. We will update the corresponding code as soon as possible.

from openlongtailrecognition-oltr.

@drcege @ssfootball04 @sanjanamallya Hello, we have updated configuration files for Places as well. The current reproduced results are a little better than reported, please check out the updates. Thank you very much!.

from openlongtailrecognition-oltr.