Results don't match with paper on KITTI Leaderboard about dsgn HOT 7 CLOSED

Hi, the result is weird. Did you matched the results on validation set? And which epoch did you use to inference the test set? Mine is epoch-53 for car (dec at epoch50) and epoch-28 for pedcyc (dec at epoch-25) on the validation set. For testing set, mine is epoch-36 for car (lr dec at epoch-32) and epoch-22 for pedcyc (lr dec at epoch-18). An important thing is that the results on KITTI benchmark can be unstable sometimes in my experience.
When I was formatting the code, there can be some minor difference (fixed bug) but I think they are minor to the final performance.
I think you should fine-tune the models a bit. Since you can train the model with 32G memory, I think several options in configuration can be maximized for testing the upper bound, i.e., NUM_CONVS = 4, num_3dconvs = 2. And cat_img_feature sometimes does not work effective for car category in my experiments. Besides, you are recommended to try the experiments several times.
I will later upload a config that can be trained within 24G memory since I can not get a 32G machine at hand.

from dsgn.

For the test set submission, I use mptrain_car_trainval.sh to train on the KITTI trainval set for 45 epochs with lr decrease at epoch 32. Is this different from your submission?

from dsgn.

OK, I think the epoch of the script is right for the trainval setting because trainval involves twice the train images (epoch-36 for car category (dec at epoch-32)), which is computed from the source script. The testing epoch is picked out empirically since the submission system is limited.

I do not have the available 32G machine to train the model. You might fine-tune some parameters on the validation set as I mentioned before. What is your performance on the validation set?

from dsgn.

The performance on the validation set evaluated at epoch 53 (using train set) is close to the paper. May you remember what epoch did you pick on the trainval model to submit for the leaderboard?

from dsgn.

I currently submitted at the last epoch 45

from dsgn.

The accurate epoch I submitted on the leaderboard should be about 36 epoch (learning rate is decreased at 32-epoch) (I changed the dataprovider when formatting the code so the epoch is not exactly same). I just submitted the model with the highest validation accuracy. I suggest you compare their validation accuracy before you submitted it. Choose the first epoch to get the highest accuracy to prevent over-fitting.

from dsgn.

I will close the issue now. You can re-open it once you get new problems. Thanks.

from dsgn.