OOM issue during training of classifier on GPU instance about aspect-based-sentiment-analysis HOT 2 CLOSED

Thanks!
I tried reducing the batch size before but this did not change a lot. The sequence length was a good intuition ;) Reducing it to max 30 tokens allows much larger datasets.

Best,
Tobias

from aspect-based-sentiment-analysis.

Hey Tobias,
if you have problems with the OOM error, you need to inspect how your training batches look like ☺️

1. reduce the batch size
2. reduce the input sequence length (too long texts => use a text splitter, take a look how the Pipeline works)
3. distribute the training (few GPUs)

I think you should check the training dataset and remove/trim outliers (in terms of the input sequence length). In other words, your training dataset probably contains texts that are too long. This is a common problem of processing sequences (you need to adjust a batch to the longest example and then we have problems allocating a tensor even if there are mainly zeros).

This is my quick intuition. You need to provide more details ☺️

from aspect-based-sentiment-analysis.