Memory is fully eaten and training quit with errors for 40k hours ASR training about nemo HOT 5 OPEN

That Nemo version is 6 months old, can you use r1.23 and see if it persists ? We do not see constantly increasing CPU memory per epoch, but that may be because we use multiple nodes - min 4 nodes

from nemo.

Is it GPU or CPU memory that is exhausted ? And how many nodes are you using ?

What version of NeMo are you using ?
Without sufficient details it's not possible to debug.

What I can say is we train on nodes with 400 GB ram per node and A100 with 80GB gpu memory and train on 90-400K hours of speech without oom in either CPU or GPU memory.

If you can visibly see CPU ram constantly increase during training, a pseudo fix could be to use exp_manager.max_time_per_run and set it to a reasonable value like a day, then the job stops after a day and you can restart it and avoid memory leak. It's not a fix but a temporary solution

from nemo.

Is it GPU or CPU memory that is exhausted ? And how many nodes are you using ?

1. CPU not GPU
2. Just single node
   We just added one row
   self.log('loss', loss_value, on_step=True, prog_bar=True, on_epoch=False,)
   in file:
   nemo/collections/asr/models/ctc_models.py
   Previously, we used on_epoch=True, but now the problem still remains after changine to False.

What version of NeMo are you using ?

git log
commit 0d3d8fa (HEAD -> main)
Author: anteju [email protected]
Date: Wed Nov 15 16:56:29 2023 -0800

[ASR] GSS-based mask estimator (#7849)

* Added GSS-based mask estimator for multispeaker scenarios

Signed-off-by: Ante Jukić <[email protected]>

* Addressed PR comments

Signed-off-by: Ante Jukić <[email protected]>

---------

Signed-off-by: Ante Jukić <[email protected]>
Co-authored-by: Taejin Park <[email protected]>

Actually, it's very easy to verify: you just submit a training task with, say librispeech data, you can observe you CPU memory keeps increasing within an epoch.
But such memory increase won't hurt since memory increase slow and after an epoch, memory usage somehwo is going down again. Here, if we decrease our training data down to 30k, for 1.2T cpu memory, we can finish an epoch normally.

from nemo.

from nemo.

Hi, is this issue resolved? I've been running into the same issue. (I can confirm that it happens on 1.23 as well)

from nemo.