Frequency weighted training sentences about sentencepiece HOT 2 CLOSED

Thank you for using SentencePiece.

Though it is not documented, spm_train --input=tsv_file --input_format=tsv .. allows for loading a corpus in tsv format where the first column is the sentence and the second column is frequency. --input_format=tsv is originally prepared to train the model from the word list with frequency.

As for the training data, I feel that the default size 100M is enough for SPM training as long as they are sampled appropriately from the full corpus. SentencePiece adopts a simple uni-gram language model with relatively smaller vocab size. We would be able to get a reliable statistics only from the small subset of the corpus. However, it would be nicer to check the output segmentation changing the size of corpus.

from sentencepiece.

Thank you Kudo-san for the satisfying answers and thank you for continuously developing SentencePiece! Cheers.

from sentencepiece.