Training time? about mixformer HOT 5 CLOSED

For the Base model:
--V100: ~40h
--2080Ti: ~3d20h
For the Large model:
--V100: ~3d+
We didn't train MixFormer-L on 2080Ti since it requires large gpu memory.

As for the SPM training(stage-2 training process), it takes only a few hours.

from mixformer.

Thanks a lot.

from mixformer.

May I ask the training time metioned above is under lmdb data format or not?

from mixformer.

May I ask the training time metioned above is under lmdb data format or not?

Not the lmdb data format.

from mixformer.

I am training the base_large model on a GPU with 12 GB of RAM memory. As soon as I increase the batch size to more than 1 I am getting "CUDA out of memory issue". I am wondering how much memory did you have on your V100 GPU, 32 GB? Anyway it is strange since I was expecting to settle a batch size of 4 or so.
I have also noticed that increase num_workers, slightly increase the allocated memory on GPU.
Thank you!!

from mixformer.