Trouble with reversibility about sentencepiece HOT 4 CLOSED

Thank you for using sentencepiece.

That looks strange. The tokenization is basically reversible except for the case when the input contains unknown symbols and the input is encoded into id sequences with EncodeAsIds or DecodeId APIs. However, that seems not the case in your experiments.

Could you share the model file you are using? I will take a look.

from sentencepiece.

Given your comment, I further looked into that last input line I provided by copying from the input file (opened in vim, copied with Mac copy) into python interpreters:

>>> x = "Mon emprisonnement m'a contraint à me pencher sur l'essentiel quant à moi-même, mon engagement politique et mon pays."
>>> x
"Mon emprisonnement m'a contraint \xc3\xa0 me pencher sur l'essentiel\xc2\xa0quant \xc3\xa0 moi-m\xc3\xaame, mon engagement politique et mon pays."
>>> x.encode()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 33: ordinal not in range(128)

>>> x = "Mon emprisonnement m'a contraint à me pencher sur l'essentiel quant à moi-même, mon engagement politique et mon pays."
>>> x
"Mon emprisonnement m'a contraint à me pencher sur l'essentiel\xa0quant à moi-même, mon engagement politique et mon pays."
>>> x.encode()
b"Mon emprisonnement m'a contraint \xc3\xa0 me pencher sur l'essentiel\xc2\xa0quant \xc3\xa0 moi-m\xc3\xaame, mon engagement politique et mon pays."

There is clearly something funky going on with the space token between "l'essentiel" and "quant". I'm not sure why they are portrayed differently across python2 and python 3 as "\xc2\xa0" and "\xa0", but these non-breaking spaces seem to get eaten up by sentencepiece.

I don't know whether you want sentencepiece to behave differently, but for now I'm just going to replace non-breaking spaces with normal spaces.

from sentencepiece.

Hi,

I've just remembered that I introduced a bug around unicode handling in the latest python wrapper.
453fd9a

The master branch has been already fixed, but the pip package is not updated yet.
Could you try the attached whl package?
sentencepiece-0.1.4-cp35-cp35m-manylinux1_x86_64.whl.gz

% gzip -d sentencepiece-0.1.4-cp35-cp35m-manylinux1_x86_64.whl
% pip3 install sentencepiece-0.1.4-cp35-cp35m-manylinux1_x86_64 --user

from sentencepiece.

Let me close this issue. Please reopen it if this issue still persists.

from sentencepiece.