Symbols in sentencepieces about sentencepiece HOT 2 CLOSED

Hi, thank you for using sentencepiece.

The default behavior is always adding "_" to the begging of sentence. This is intended as we want to treat words independently from the positions ("foo" and "bar foo" are handled as "_foo" and "_bar_foo" and there are two "_foo").

You can suppress to add "_" --add_dummy_prefix-=false to spm_train.

EncodeAs(Ids|Pieces) are the methods to tokenize the sentences. If you want to obtain the ids of bos/eos/pad, you might want to use unk_id(), bos_id(), eos_id(), pad_id() methods.

from sentencepiece.

I see. I will try the --add_dummy_prefix-=false. Thank you very much

from sentencepiece.