Bad alloc about sentencepiece HOT 3 CLOSED

Yes, it is simply out-of-memory.

SentencePiece loads entire lines into memory before training.

You might want to try --mining_sentence_size=1000000

If it doesn't work, you can make the value smaller.

from sentencepiece.

It worked! Thanks for the help.

from sentencepiece.

I got a similar error after this operation:

Final character coverage=0.999511
trainer_interface.cc(435) LOG(INFO) Done! preprocessed 8266227 sentences.
terminate called after throwing an instance of 'std::bad_alloc'

would be also requiring a larger amount of ram?

how could I prevent this?

Best regards

from sentencepiece.