CRITICAL BUG: Multi GPU training possibly not implemented correctly, resulting models potentially under-trained about slotformer HOT 7 OPEN

I did it in a bad way (see this line). I should probably switch to the built-in function, but the current code should also work

from slotformer.

Hey, thank you so much for pointing this out. Indeed, there seem to be some issues with the gradient sync in DDP training. Your solution of wrapping the model to perform loss computation within the forward pass sounds very reasonable to me. Have you tried that and did you see e.g. faster convergence speed of a DDP-trained model?

from slotformer.

ok, I believe I have fixed the second issue with this commit. I check it with your function check_sampler_index_consistency and it doesn't show any overlapping indices. Thanks for pointing that out!

I might need some thought before fixing the first issue.

from slotformer.

The second issue is likely fixed in this commit. I tested with your util function and got consistent == True.

The fix is a bit hacky and ugly tho...

from slotformer.

I am training a model in this repo as a sanity-check. Unfortunately, I don't have easy access to the dataset used in SlotFormer right now, so maybe cannot test if it converges faster. @Shamdan17 will you be able to give it a try? Just pull the newest code from my nerv package

from slotformer.

Hello, I will be able to test it in a couple days, but one thing I notice is that there is no where that the self.epoch value is updated. Typically, set_epoch is called on the sampler at the beginning of each epoch to update this value, otherwise the shuffling is the same across epochs. The warning about this can be found in this doc.

from slotformer.

I got the results of that run, it at least doesn't harm the performance, tho the improvement is also not clear. Let me know if you have any results @Shamdan17. Thanks so much for the help!

from slotformer.