Problem with max-duration during training about icefall HOT 6 CLOSED

You might want to filter out some cuts that are too long. You likely have outliers with large duration.

from icefall.

You might want to filter out some cuts that are too long. You likely have outliers with large duration.

Yeah, I got it.

I think if I set max-duration, the total duration in a batch will not exceed this value, right?
So maybe some transcript in this batch is too long?

from icefall.

Yeah something like that, I’d say too long audio/features.

BTW The total duration of supervised chunks won’t exceed max duration. It means there could be actually more due to padding. But with BucketingSampler the amount of padding is negligible.

from icefall.

Yeah something like that, I’d say too long audio/features.

BTW The total duration of supervised chunks won’t exceed max duration. It means there could be actually more due to padding. But with BucketingSampler the amount of padding is negligible.

So for the same data, icefall will need more gpu memory compared to kaldi, right?

If I remember correctly, in kaldi's training, there is a param egs.chunk-width (such as 150,110,100) to control the frames per batch. I usually use minibatch=64, so the max duration per batch may be larger than 64s. And I never see OOM, and the gpu usage is usually > 90%.

from icefall.

In general any alignment-free training will require more memory due to padding. I think that Kaldi was able to optimize the memory usage because alignment gives you frame-level supervision, so you can take chunks of utterances. On the other hand, for CTC or attention decoder you have to use the whole utterance.

from icefall.

Thanks, I'll close this issue.

By the way, for docker user, --shm-size is also needed to set for parallel training.

from icefall.