Decoding from ids with UNK tokens about sentencepiece HOT 4 CLOSED

This is somehow expected.

Short answer is that U+2047 https://www.compart.com/en/unicode/U+2047 is a sort of reserved token meaning unknown.

First, we no longer know the original surface string when converting the input into id sequences.
So EncodeIds([0]) will not return "^^".

The workaround for this situation is either

1. Decode 0 to "<unk>"
2. Decode 0 to empty string ""
3. Decode 0 to other reserved string.

We don't want to show internal representation like <unk>, as they are tricky for users. Returning empty string will make the debugging harder for developer. So we now use a special token ?? which can be interpretable both for user and developer.

Any comments for this behavior are appreciated.
It would be nicer to allow user change the behavior like --unk-surface=<unk>

Thank you.

from sentencepiece.

Thanks for clarifying.

I think that allowing the user to set this option would be great, but for now I can just replace it I guess.

One more thing that is interesting for me is why is there a trailing space in the decoded result, and a double space between the first word and the ?? special token?

>>> p.DecodeIds(p.EncodeAsIds('test ^^'))
'test  \u2047 '

from sentencepiece.

I forgot to mention that the unk id (0) is decoded " ?? ", containing heading/tailing spaces.

This is because we are not sure the unknown token is either a part of word or independent token. However, it would be safer to handle it as an independent token.

from sentencepiece.

Added --unk_surface= flag to spm_train.

Note that this is not a flag of spm_encode, meaning that the unk_surface is embedded in the model file.

from sentencepiece.