Max Recursion Depth🐛 [BUG] about nequip HOT 11 CLOSED

It is always the exact same stack trace, the number of epochs before the error occurs is variable.

Thank you for the workaround, but as restarting works, I think I'll keep the flexibility of the training model file, for now I stop most runs before 800 epochs anyhow.

from nequip.

OK, I have submitted a PR to e3nn that should implement a workaround to resolve this issue: e3nn/e3nn#297

If this is a problem for you, please try to install my branch from the linked PR.

from nequip.

I found that training on a set of 10 configurations, the error occurs after 780 epochs (around 18 min),
my gpu is NVIDIA GeForce GTX 1050 Ti.

In the zip are the data and the config file used to get the error

Runtime_error_small.zip

from nequip.

Hi @keano130 ,

Thanks for reaching out!

This is very strange. We've seen this in our group only once before and I assumed it was some kind of corruption, but if you've seen it in a different computing environment it's definitely not.

@Nicola89 could you post your version information from when you saw this so we can compare to @keano130's?

At this point I don't really have any suspicions about the source of this, but I will look into it and let you know if I find anything or have questions.

Are you able to successfully restart the training session using nequip-restart? Does the bug reoccur after restarting?

Thanks!

from nequip.

Sure! When I encountered this error I was using the following env:

python=3.8.11
cudatoolkit=11.1.74
pytorch=1.9.0 (cu111)
pytorch-geometric=1.7.2
e3nn=0.3.3
nequip=0.3.3

I can also export and attach here the conda env if that is helpful. I concur with @keano130 about when the bug happens, i.e., deep in the training. I am attaching the error file of the test aspirin run where this happened.
recursion_depth.err.zip

from nequip.

Were either of you running under wandb, and if you were, could you check the GPU & system memory consumption?

One possibility is that this is a memory leak leading to an eventual OOM error that just isn't very informative, since new_zeros is an alloc... although in that circumstance you wouldn't expect it to consistently fail on this new_zeros...

from nequip.

I was running wandb, both GPU and system memory consumption were far from the maximum consumption in my case.

from nequip.

After the error, it is possible to just restart the training, and it trains normally until after around 800 more epochs, where it fails again.

from nequip.

interesting, thanks for the info @keano130. Does it fail in the exact same way, exact same stack trace?

A workaround appears to be enabling compile_model: True in your config. This compiles the model down to TorchScript for training. (Please note that if you do this the trained model file is somewhat less flexible / useful, since you can't go poking around in the Python module tree later, although I think the parameters can be loaded from it into a Python model if you really need to.)

from nequip.

Update on my side: restart gives the same problem after a similar number of epochs (2923 vs 2964).

from nequip.

e3nn has made a new release incorporating the bugfix: https://github.com/e3nn/e3nn/releases/tag/0.3.5

So you can now get around this just by installing e3nn==0.3.5.

from nequip.