terminate called after throwing an instance of 'Darts::Details::Exception' about sentencepiece HOT 5 CLOSED

Thank you for uploading the model.

Seems the the model contains a null (\0) token, implying that the training corpus contains '\0'.
One possibility is that the corpus is not encoded in utf-8. Trivial work around is to escape '\0' in the corpus into some safe character like ' '.

Anyway, I will put some workaround (or simply emitting an error) to prevent the model from having '\0'.

Thank you.

from sentencepiece.

Updated the code to refuse null character. Please try the latest spm_train.

from sentencepiece.

Thank you for using SentencePiece

Seems the model accidentally contains an empty token, which is not allowed.

How did you make/train 'm.model'. Could you upload 'm.model' so I can investigate the error?

from sentencepiece.

Thank you for this amazing repo since it makes word segmentation on Cantonese much better and easier. It is really hard to find an unsupervised method that this good in terms of accuracy, speed, and low memory usage.

Attaching the model file.
m3.model.zip

from sentencepiece.

Thanks, I will first try to encode my corpus in utf-8. Will report back to here.

from sentencepiece.