Error when trying to import Tokenizer about sacremoses HOT 11 CLOSED

a quick workaround is explicitly stating the encoding as being utf-8 in corpus.py line 37 i.e. change
with open(self.datadir+category+'.txt') as fin:
to
with open(self.datadir+category+'.txt', encoding='utf-8') as fin:
although this only works with python 3 (with py2 you'd need to use codecs)

from sacremoses.

Interesting, Windows is reading it as cp1252 instead of utf8. Are you using Python 3 or 2?

from sacremoses.

3.6

from sacremoses.

Very interesting now!

Looks like an upsteam bug/feature in CPython vs Windows... https://stackoverflow.com/questions/42070668/python-3-default-encoding-cp1252

My suggestion is to set the proper locale before the Python interpreter.

If you're on cygwin: https://stackoverflow.com/questions/24255407/permanently-set-python-path-for-anaconda-within-cygwin

If natively and globally on windows, see https://www.java.com/en/download/help/locale.xml

from sacremoses.

BTW, do you get the same when you import nltk?

from sacremoses.

I have the same issue. But not when importing nltk

from sacremoses.

Are you using Windows too?

from sacremoses.

Yes, Windows 10. Python 3.6.5

from sacremoses.

@lukedorney Hmmm.. It's weird that in Python3 the default encoding is already utf8 but Windows is doing something strange in the locale such that it's not the default.

from sacremoses.

@lukedorney @janwendt @sleighsoft I've added the patch and updated the package.

Please tell me if you still face the same problems after

pip install -U sacremoses

from sacremoses.

Going to close this issue. If there's any error in Windows from encoding problems again, please feel free to reopen this issue.

from sacremoses.