MemoryError when process "beer" dataset about unsupervised-aspect-extraction HOT 2 CLOSED

Memory size of the machine I used is 65840088 KiB, so I didn't encounter this problem. But now the data preparing part is generally not well written as it takes a long time (around 20 mins) on large dataset like "beer".

You can try a few ways to make it work on your machine. For example, randomly sample a subset (e.g. size=50000) before padding. Or you can pad the sentence inside the batch/negative generator instead of the beginning. In this case, you need to modify the network a little bit to set the sequence length of input to "None" to allow the shape of different batches to be different.

from unsupervised-aspect-extraction.

Thanks very much for your reply, It works well on a new machine now.

from unsupervised-aspect-extraction.