Is there any trick in training? about human-pose-estimation.pytorch HOT 7 CLOSED

There is no other trick for training. If you use 4 gpus and following our steps to train, you could exactly reproduce our results, no other modification needed.

from human-pose-estimation.pytorch.

I used 4 V100s and followed the steps to train, but the loss was fluctuating at 0.0012 and the mAP equaled to 0. Have you ever seen this phenomenon before?

from human-pose-estimation.pytorch.

Please follow the steps to disable cudnn of batchnorm.

from human-pose-estimation.pytorch.

@leoxiaobin could you please explain why cudnn BN disabling is so important?

from human-pose-estimation.pytorch.

@Scitator ，for my understanding, I think that is a bug of cudnn. But we do not have the source code of cudnn, I can point which part cause the bug.

from human-pose-estimation.pytorch.

Okay, but how have you get to the idea of cudnn BN disabling?
I mean, it quite unusual operation and if it does not work without it... it is just strange, you know. Have you tried to ask nvidia for help?

from human-pose-estimation.pytorch.

I found the mean variance of BN is not normal when I trained our net using cudnn, so I tried disabled BN of cudnn, then everything was normal.
I have not tried to ask NVIDIA for help now.

from human-pose-estimation.pytorch.