Memory Leak problem about persformer_3dlane HOT 13 CLOSED

So far everything works fine with our tests, and we’ll test on other machines later to see if we can reproduce your problem.

from persformer_3dlane.

Hello, just try to reproduce your problem, how much memory do you have in your machine? And after how many epoch does it happen?

from persformer_3dlane.

thanks for your reply. The memory is 256GB, and the memory problem happens in the first epoch.

I can observe in htop that the memory usage is gradually increasing from around 20GB to 200+GB, and then the process crashes. It takes about 10~20 minutes from the start of training.

from persformer_3dlane.

could you provide more information about your machine? such as pytorch version, cuda version, python version, etc. We didn't go into a memory leak when we train on 4-3090 with 128GB memory.

from persformer_3dlane.

my python version, cuda version, and PyTorch version are 3.8.13, 11.1, and 1.8.1.

I will train the model on other machines later and see if the problem still exists.

from persformer_3dlane.

it seems the same problem still exists on other machines with python version 3.6.13.

from persformer_3dlane.

Maybe you can pull the latest code and have a try to see if the problem still exists.

from persformer_3dlane.

Maybe you can pull the latest code and have a try to see if the problem still exists.

Thanks, I pull the latest code but the problem still exists.

Maybe it is caused by some unexpected environment problem. I would close this issue. If anyone else encounters this issue in the future, we may re-open this issue again.

from persformer_3dlane.

@liuzili97 I have met the same problem and my environment settings are the same as yours. Have you solved the problem?

from persformer_3dlane.

@liuzili97 I have met the same problem and my environment settings are the same as yours. Have you solved the problem?

No, I haven't

from persformer_3dlane.

I also have the same issue! It consumes almost 99% of my system memory and crashes even before training starts (after loading dataset). I reported it in a separate issue here: #33

from persformer_3dlane.

could you provide more information about your machine? such as pytorch version, cuda version, python version, etc. We didn't go into a memory leak when we train on 4-3090 with 128GB memory.

Thanks for your perfect work. Would you like to tell me the gcc --version of your environment with your 4-3090? My Server is 8-3090+CUDA11.1+pytorch1.8.0+gcc version 10.3.0 (Ubuntu 10.3.0-1ubuntu120.10), but I can't solve the problem in ‘INSTALL.md’ section when "cd models/nms/ --> python setup.py install"

from persformer_3dlane.

from persformer_3dlane.