Jenkins node 4 has problem with Python 3.6 and locales about buildout.coredev HOT 8 CLOSED

Well, the Dutch locale is not needed, it was just meant as an example of how to set locales. But anyway, it looks okay. So maybe there is something else that unsets these variables at some point.

But I have merged the two PRs (plone.app.locales and plone.dexterity), so it should be fine now.

from buildout.coredev.

I see this always happens on Jenkins Node 4 (syslab). See also older job.

Maybe something is configured differently there, only on Python 3.6?
Or there is a bad download of this package somewhere.

@gforcada @ale-rt Is this something you can take a look at?

from buildout.coredev.

Not sure how to proceed debugging.
The list of installed python3.6 packages looks identica on both node 3 and 4 (the one I can ssh into).
Should I temporarily shut down node4?
Or is there a way to not use it for the Python3. jobs?

from buildout.coredev.

I have removed the buildout.requirements extension, so this is solved now.
Replaced with an easy script.
Done on 5.2. I will do the same on 6.0.

Eventually, like spoken about in that PR, plone.versioncheck could be extended instead of using the script.

from buildout.coredev.

Found it I think. The same problem happens with plone.app.locales on Python 3.6 on this node, see job 1 and job 2:

Traceback (most recent call last):
  File "/tmp/tmp_oacv4nx", line 14, in <module>
    exec(compile(f.read(), '/home/jenkins/workspace/plone-6.0-python-3.6/src/plone.app.locales/setup.py', 'exec'))
  File "/home/jenkins/workspace/plone-6.0-python-3.6/src/plone.app.locales/setup.py", line 11, in <module>
    long_description=(open('README.rst').read() + '\n' +
  File "/home/jenkins/shiningpanda/jobs/641b788a/virtualenvs/d41d8cd9/lib/python3.6/encodings/ascii.py", line 26, in decode
    return codecs.ascii_decode(input, self.errors)[0]
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 111: ordinal not in range(128)

I can reproduce this locally on my Mac. I have this in a shell:

$ echo $LANG 
en_GB.UTF-8
$ echo $LC_CTYPE 
UTF-8
$ echo $LC_ALL 
en_GB.UTF-8

With that it works fine on all Python versions (specifically 2.7, 3.6, 3.7).

When I call unset on these three environment variables, it still works on 2.7 and 3.7, but not on 3.6:

$ python3.6
Python 3.6.10 (default, Feb  8 2020, 17:39:49) 
[GCC 4.2.1 Compatible Apple LLVM 11.0.0 (clang-1100.0.33.17)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> x = open("README.rst").read()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/maurits/.pyenv/versions/3.6.10/lib/python3.6/encodings/ascii.py", line 26, in decode
    return codecs.ascii_decode(input, self.errors)[0]
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 111: ordinal not in range(128)

So on Jenkins node 4, some locale settings should be configured.
Some hints from the wiki page of a customer:

• In /etc/environment add the following lines:

  LC_ALL=en_US.UTF-8
  LANG=en_US.UTF-8
  

• (Dutch specific: )There has to be a locale nl_NL.utf8. Check wih locale -a if this is in the list. Otherwise: sudo locale-gen nl_NL.utf8.

Meanwhile, I will make a PR for plone.app.locales to catch the error. It looks like open("README.rst", "rb") helps.

from buildout.coredev.

It seems it is like that since quite some time:

root@jenkins-plone:~# grep en_US /etc/environment 
LC_ALL=en_US.UTF-8
LANG=en_US.UTF-8
root@jenkins-plone:~# locale -a | grep nl_NL
nl_NL@euro
nl_NL.iso885915@euro

I anyway ran:

Generating locales (this might take a while)...
  nl_NL.UTF-8... done
Generation complete.
root@jenkins-plone:~# locale -a | grep nl_NL
nl_NL@euro
nl_NL.iso885915@euro
nl_NL.utf8

from buildout.coredev.

Update: there are still more packages that give problems.

• buildout.requirements was fixed by replacing it with a script
• plone.app.locales
• plone.outputfilters
• plone.restapi: plone/plone.restapi#879
• various dependent scripts via zest.releaser: #646
• plone.app.robotframework fails for the same reason. Locally it works because I have a working egg in my egg cache. plone/plone.app.robotframework#119

Okay, that actually seems to be the last one. I have cleaned my egg cache locally, and with the above, it all works again.
I will make Jenkins jobs with the last remaining PRs

from buildout.coredev.

Checked: all PRs have been merged last month, and I have not seen the problem anymore. Closing the issue.

from buildout.coredev.