Git Product home page Git Product logo

python-build-standalone's Introduction

python-build-standalone's People

Contributors

bcmyers avatar charliermarsh avatar chenxu2048 avatar clnperez avatar danielshaulov avatar ddeville avatar dependabot[bot] avatar edward-knight avatar edwardcwang avatar f0rmiga avatar indygreg avatar jonathan-albrecht-ibm avatar kornholi avatar melroyvandenberg avatar mp4096 avatar namrata-ibm avatar polkovnikov avatar uranusjr avatar zanieb avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

python-build-standalone's Issues

Other pythons versions

This is more a question than an actual issue: is it planned to build other versions of Python? Python 3.4 is now deprecated but some project may still be using it (I am ๐Ÿ˜ž ), and 3.5 and 3.6 versions would be great.

3.7 introduced the async keyword so it is not a drop in replacement to previous version, and my projects with which I wanted to try PyOxidizer need some work before being able to use python 3.7โ€ฆ

Anyway, I'm very excited by this project!

build/versions/VERSION.*: No such file or directory

I try to compile on Debian8.10. Its clean system, just installed.
I was install Python3.6 and run this command:

python3.6 ./build-linux.py --python cpython-3.7

But after install some dependencies i catch this error

Makefile:78: .../python-build-standalone/build/versions/VERSION.*: No such file or directory
Makefile:289: warning: overriding recipe for target '.../python-build-standalone/build/cpython--x86_64-unknown-linux-gnu-noopt.tar'
Makefile:286: warning: ignoring old recipe for target '.../python-build-standalone/build/cpython--x86_64-unknown-linux-gnu-noopt.tar'
make: *** No rule to make target '.../python-build-standalone/build/versions/VERSION.*'.  Stop.

What i do wrong? Debian8 not supported?

libpython_link_mode=shared for unix platforms ?

Hi @indygreg,

I working on a python binding for the Godot game engine
Godot works as a standalone application that can be extended by loading shared libraries, hence my binding must embed a python interpreter.

So currently I build CPython from the source as part of the binding build, but (as you very well know ๐Ÿ˜„ ) this is a tedious and error-prone step.
Hence I'll be really interested in using your pre-build releases instead ;-)

However my binding is written in Cython which produces native modules that depend on libpythonXY.so, but your builds only provide libpythonXY.so for Windows :'(

So what the reasons shared library is not available for Unix ? Is there something I can do to help ? ;-)
Does a cheap hack like gcc -shared -o libpythonXY.so build/Object/**.so or recompiling libpythonX.Ym.a into a shared lib could work ?

Another important requirement for godot-python is to allow game developer to be able to use pip to access the full Python ecosystem. Is there some limitations I would face with your builds ?
(From why I've seen in the documentation, the main issues are the shebang in the pip binary which isn't a big deal and the fact that plenty of native lib expect to have a libpython.so available but my project already have this requirement by itself so I have to solve this anyway)

PS: I've been following your Pyoxidizer project since it announcement, it's an incredible work you're doing at addressing one of the biggest weakness of Python. I guess we can confidently say you are building a killer feature for Python ๐Ÿ˜ƒ

Skip packaging regression tests?

Currently all of the regression tests are being packaged. It adds up to quite a bit of space:

21M     install/lib/python3.7/test

My debian installation only has a small subset:

$ du -sh /usr/lib/python3.7/test
516K    /usr/lib/python3.7/test
$ tree /usr/lib/python3.7/test
/usr/lib/python3.7/test
โ”œโ”€โ”€ __init__.py
โ”œโ”€โ”€ __main__.py
โ”œโ”€โ”€ __pycache__
โ”‚ย ย  โ”œโ”€โ”€ __init__.cpython-37.pyc
โ”‚ย ย  โ”œโ”€โ”€ __main__.cpython-37.pyc
โ”‚ย ย  โ”œโ”€โ”€ ann_module.cpython-37.pyc
โ”‚ย ย  โ”œโ”€โ”€ ann_module2.cpython-37.pyc
โ”‚ย ย  โ”œโ”€โ”€ ann_module3.cpython-37.pyc
โ”‚ย ย  โ”œโ”€โ”€ regrtest.cpython-37.pyc
โ”‚ย ย  โ””โ”€โ”€ test_support.cpython-37.pyc
โ”œโ”€โ”€ ann_module.py
โ”œโ”€โ”€ ann_module2.py
โ”œโ”€โ”€ ann_module3.py
โ”œโ”€โ”€ libregrtest
โ”‚ย ย  โ”œโ”€โ”€ __init__.py
โ”‚ย ย  โ”œโ”€โ”€ __pycache__
โ”‚ย ย  โ”‚ย ย  โ”œโ”€โ”€ __init__.cpython-37.pyc
โ”‚ย ย  โ”‚ย ย  โ”œโ”€โ”€ cmdline.cpython-37.pyc
โ”‚ย ย  โ”‚ย ย  โ”œโ”€โ”€ main.cpython-37.pyc
โ”‚ย ย  โ”‚ย ย  โ”œโ”€โ”€ refleak.cpython-37.pyc
โ”‚ย ย  โ”‚ย ย  โ”œโ”€โ”€ runtest.cpython-37.pyc
โ”‚ย ย  โ”‚ย ย  โ”œโ”€โ”€ runtest_mp.cpython-37.pyc
โ”‚ย ย  โ”‚ย ย  โ”œโ”€โ”€ save_env.cpython-37.pyc
โ”‚ย ย  โ”‚ย ย  โ”œโ”€โ”€ setup.cpython-37.pyc
โ”‚ย ย  โ”‚ย ย  โ””โ”€โ”€ utils.cpython-37.pyc
โ”‚ย ย  โ”œโ”€โ”€ cmdline.py
โ”‚ย ย  โ”œโ”€โ”€ main.py
โ”‚ย ย  โ”œโ”€โ”€ refleak.py
โ”‚ย ย  โ”œโ”€โ”€ runtest.py
โ”‚ย ย  โ”œโ”€โ”€ runtest_mp.py
โ”‚ย ย  โ”œโ”€โ”€ save_env.py
โ”‚ย ย  โ”œโ”€โ”€ setup.py
โ”‚ย ย  โ””โ”€โ”€ utils.py
โ”œโ”€โ”€ regrtest.py
โ”œโ”€โ”€ support
โ”‚ย ย  โ”œโ”€โ”€ __init__.py
โ”‚ย ย  โ”œโ”€โ”€ __pycache__
โ”‚ย ย  โ”‚ย ย  โ”œโ”€โ”€ __init__.cpython-37.pyc
โ”‚ย ย  โ”‚ย ย  โ”œโ”€โ”€ script_helper.cpython-37.pyc
โ”‚ย ย  โ”‚ย ย  โ””โ”€โ”€ testresult.cpython-37.pyc
โ”‚ย ย  โ”œโ”€โ”€ script_helper.py
โ”‚ย ย  โ””โ”€โ”€ testresult.py
โ””โ”€โ”€ test_support.py

5 directories, 38 files

However this seems to be different everywhere I look...

The docker images don't have it at all:

$ docker run --rm -it python:3.7 python -c "import test"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ModuleNotFoundError: No module named 'test'

But my macOS homebrew install includes everything:

 20M	/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/test/

CI/CD

This repo should at least have CI, and ideally CD also.

Patching makesetup

https://github.com/indygreg/python-build-standalone/blob/274f549/pythonbuild/cpython.py#L294 refers to a problem I just hit when trying to use https://github.com/indygreg/python-build-standalone/blob/master/cpython-unix/static-modules.linux64 , in that CPython makesetup breaks when Setup.local entries contain =.

I am actually trying to setup suitable static builds on Open Build Service, so borrowing only the logic found here, which has been quite useful.

It seems that a bunch of hacks here could be removed by patching makesetup.

Upstream issue at https://bugs.python.org/issue37839 , with more details at https://bugs.python.org/issue35184

I've also encountered it on Python 3.7, and the above bpo issues suggest it pre-dates that which would block #8

wrong interpreter path in files

python/install/bin/
Python 3.7.7

musl

==> 2to3-3.7 <==
#!/install/bin/python3.7

==> easy_install <==
#!/build/out/python/install/bin/python3

==> easy_install-3.7 <==
#!/build/out/python/install/bin/python3

==> idle3.7 <==
#!/install/bin/python3.7

==> pip <==
#!/build/out/python/install/bin/python3

==> pip3 <==
#!/build/out/python/install/bin/python3

==> pip3.7 <==
#!/build/out/python/install/bin/python3

==> pydoc3.7 <==
#!/install/bin/python3.7

==> pyvenv-3.7 <==
#!/install/bin/python3.7

pgo

==> 2to3-3.7 <==
#!/install/bin/python3.7

==> easy_install <==
#!/build/out/python/install/bin/python3

==> easy_install-3.7 <==
#!/build/out/python/install/bin/python3

==> idle3.7 <==
#!/install/bin/python3.7

==> pip <==
#!/build/out/python/install/bin/python3

==> pip3 <==
#!/build/out/python/install/bin/python3

==> pip3.7 <==
#!/build/out/python/install/bin/python3

==> pydoc3.7 <==
#!/install/bin/python3.7

==> pyvenv-3.7 <==
#!/install/bin/python3.7

Include libffi in Windows image

Branching off from #22

Currently the Windows image provides some libffi symbols in ./python/build/extensions/_ctypes/ffi.obj and cfield.obj , but it doesnt provide libffi as a separate library like mac and linux images.

This means it is troublesome for any project to use libffi (which is indirectly almost all serious Python projects via cffi), as doing so will cause linker conflicts.

Failure during gcc-linux64 didnt resume

Brief internet problems caused a failure during finalisation of gcc-linux64, and restarting resulted in job starting at image-clang.

gcc-linux64> make[8]: Leaving directory `/build/gcc-objdir/x86_64-unknown-linux-gnu/32/libsanitizer'
gcc-linux64> make[7]: Leaving directory `/build/gcc-objdir/x86_64-unknown-linux-gnu/32/libsanitizer'
gcc-linux64> make[6]: Leaving directory `/build/gcc-objdir/x86_64-unknown-linux-gnu/32/libsanitizer'
gcc-linux64> make[5]: Leaving directory `/build/gcc-objdir/x86_64-unknown-linux-gnu/libsanitizer'
gcc-linux64> make[4]: Leaving directory `/build/gcc-objdir/x86_64-unknown-linux-gnu/libsanitizer'
gcc-linux64> make[3]: Leaving directory `/build/gcc-objdir/x86_64-unknown-linux-gnu/libsanitizer'
gcc-linux64> make[2]: Leaving directory `/build/gcc-objdir/x86_64-unknown-linux-gnu/libsanitizer'
gcc-linux64> make[1]: Leaving directory `/build/gcc-objdir'
gcc-linux64> ~
gcc-linux64> copying container files to /home/jayvdb/projects/python/python-build-standalone/build/gcc-9.1.0-linux64.tar
Traceback (most recent call last):
  File "/home/jayvdb/projects/python/python-build-standalone/build/venv.linux/lib64/python3.7/site-packages/urllib3/connectionpool.py", line 421, in _make_request
    six.raise_from(e, None)
  File "<string>", line 3, in raise_from
  File "/home/jayvdb/projects/python/python-build-standalone/build/venv.linux/lib64/python3.7/site-packages/urllib3/connectionpool.py", line 416, in _make_request
    httplib_response = conn.getresponse()
  File "/usr/lib64/python3.7/http/client.py", line 1333, in getresponse
    response.begin()
  File "/usr/lib64/python3.7/http/client.py", line 305, in begin
    version, status, reason = self._read_status()
  File "/usr/lib64/python3.7/http/client.py", line 266, in _read_status
    line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
  File "/usr/lib64/python3.7/socket.py", line 589, in readinto
    return self._sock.recv_into(b)
socket.timeout: timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/jayvdb/projects/python/python-build-standalone/build/venv.linux/lib64/python3.7/site-packages/requests/adapters.py", line 449, in send
    timeout=timeout
  File "/home/jayvdb/projects/python/python-build-standalone/build/venv.linux/lib64/python3.7/site-packages/urllib3/connectionpool.py", line 720, in urlopen
    method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
  File "/home/jayvdb/projects/python/python-build-standalone/build/venv.linux/lib64/python3.7/site-packages/urllib3/util/retry.py", line 400, in increment
    raise six.reraise(type(error), error, _stacktrace)
  File "/home/jayvdb/projects/python/python-build-standalone/build/venv.linux/lib64/python3.7/site-packages/urllib3/packages/six.py", line 735, in reraise
    raise value
  File "/home/jayvdb/projects/python/python-build-standalone/build/venv.linux/lib64/python3.7/site-packages/urllib3/connectionpool.py", line 672, in urlopen
    chunked=chunked,
  File "/home/jayvdb/projects/python/python-build-standalone/build/venv.linux/lib64/python3.7/site-packages/urllib3/connectionpool.py", line 423, in _make_request
    self._raise_timeout(err=e, url=url, timeout_value=read_timeout)
  File "/home/jayvdb/projects/python/python-build-standalone/build/venv.linux/lib64/python3.7/site-packages/urllib3/connectionpool.py", line 331, in _raise_timeout
    self, url, "Read timed out. (read timeout=%s)" % timeout_value
urllib3.exceptions.ReadTimeoutError: UnixHTTPConnectionPool(host='localhost', port=None): Read timed out. (read timeout=60)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/jayvdb/projects/python/python-build-standalone/cpython-unix/build.py", line 899, in <module>
    sys.exit(main())
  File "/home/jayvdb/projects/python/python-build-standalone/cpython-unix/build.py", line 752, in main
    build_gcc(client, get_image(client, ROOT, BUILD, "gcc"))
  File "/home/jayvdb/projects/python/python-build-standalone/cpython-unix/build.py", line 158, in build_gcc
    build_env.get_tools_archive(archive_path("gcc", "linux64"), "host")
  File "/usr/lib64/python3.7/contextlib.py", line 119, in __exit__
    next(self.gen)
  File "/home/jayvdb/projects/python/python-build-standalone/pythonbuild/buildenv.py", line 201, in build_environment
    container.remove()
  File "/home/jayvdb/projects/python/python-build-standalone/build/venv.linux/lib64/python3.7/site-packages/docker/models/containers.py", line 347, in remove
    return self.client.api.remove_container(self.id, **kwargs)
  File "/home/jayvdb/projects/python/python-build-standalone/build/venv.linux/lib64/python3.7/site-packages/docker/utils/decorators.py", line 19, in wrapped
    return f(self, resource_id, *args, **kwargs)
  File "/home/jayvdb/projects/python/python-build-standalone/build/venv.linux/lib64/python3.7/site-packages/docker/api/container.py", line 994, in remove_container
    self._url("/containers/{0}", container), params=params
  File "/home/jayvdb/projects/python/python-build-standalone/build/venv.linux/lib64/python3.7/site-packages/docker/utils/decorators.py", line 46, in inner
    return f(self, *args, **kwargs)
  File "/home/jayvdb/projects/python/python-build-standalone/build/venv.linux/lib64/python3.7/site-packages/docker/api/client.py", line 238, in _delete
    return self.delete(url, **self._set_request_timeout(kwargs))
  File "/home/jayvdb/projects/python/python-build-standalone/build/venv.linux/lib64/python3.7/site-packages/requests/sessions.py", line 615, in delete
    return self.request('DELETE', url, **kwargs)
  File "/home/jayvdb/projects/python/python-build-standalone/build/venv.linux/lib64/python3.7/site-packages/requests/sessions.py", line 533, in request
    resp = self.send(prep, **send_kwargs)
  File "/home/jayvdb/projects/python/python-build-standalone/build/venv.linux/lib64/python3.7/site-packages/requests/sessions.py", line 646, in send
    r = adapter.send(request, **kwargs)
  File "/home/jayvdb/projects/python/python-build-standalone/build/venv.linux/lib64/python3.7/site-packages/requests/adapters.py", line 529, in send
    raise ReadTimeout(e, request=request)
requests.exceptions.ReadTimeout: UnixHTTPConnectionPool(host='localhost', port=None): Read timed out. (read timeout=60)
make: *** [Makefile:95: /home/jayvdb/projects/python/python-build-standalone/build/gcc-9.1.0-linux64.tar] Error 1
```
```console
~python-build-standalone> ./build-linux.py
Requirement already satisfied: certifi==2019.9.11 in ./build/venv.linux/lib/python3.7/site-packages (from -r /home/jayvdb/projects/python/python-build-standalone/requirements.txt (line 7)) (2019.9.11)
Requirement already satisfied: cffi==1.13.1 in ./build/venv.linux/lib/python3.7/site-packages (from -r /home/jayvdb/projects/python/python-build-standalone/requirements.txt (line 11)) (1.13.1)
Requirement already satisfied: chardet==3.0.4 in ./build/venv.linux/lib/python3.7/site-packages (from -r /home/jayvdb/projects/python/python-build-standalone/requirements.txt (line 43)) (3.0.4)
Requirement already satisfied: click==7.0 in ./build/venv.linux/lib/python3.7/site-packages (from -r /home/jayvdb/projects/python/python-build-standalone/requirements.txt (line 47)) (7.0)
Requirement already satisfied: docker==4.1.0 in ./build/venv.linux/lib/python3.7/site-packages (from -r /home/jayvdb/projects/python/python-build-standalone/requirements.txt (line 51)) (4.1.0)
Requirement already satisfied: idna==2.8 in ./build/venv.linux/lib/python3.7/site-packages (from -r /home/jayvdb/projects/python/python-build-standalone/requirements.txt (line 54)) (2.8)
Requirement already satisfied: jinja2==2.10.3 in ./build/venv.linux/lib/python3.7/site-packages (from -r /home/jayvdb/projects/python/python-build-standalone/requirements.txt (line 58)) (2.10.3)
Requirement already satisfied: markupsafe==1.1.1 in ./build/venv.linux/lib/python3.7/site-packages (from -r /home/jayvdb/projects/python/python-build-standalone/requirements.txt (line 61)) (1.1.1)
Requirement already satisfied: pip-tools==4.2.0 in ./build/venv.linux/lib/python3.7/site-packages (from -r /home/jayvdb/projects/python/python-build-standalone/requirements.txt (line 91)) (4.2.0)
Requirement already satisfied: pycparser==2.19 in ./build/venv.linux/lib/python3.7/site-packages (from -r /home/jayvdb/projects/python/python-build-standalone/requirements.txt (line 94)) (2.19)
Requirement already satisfied: requests==2.22.0 in ./build/venv.linux/lib/python3.7/site-packages (from -r /home/jayvdb/projects/python/python-build-standalone/requirements.txt (line 97)) (2.22.0)
Requirement already satisfied: six==1.12.0 in ./build/venv.linux/lib/python3.7/site-packages (from -r /home/jayvdb/projects/python/python-build-standalone/requirements.txt (line 101)) (1.12.0)
Requirement already satisfied: urllib3==1.25.6 in ./build/venv.linux/lib/python3.7/site-packages (from -r /home/jayvdb/projects/python/python-build-standalone/requirements.txt (line 105)) (1.25.6)
Requirement already satisfied: websocket-client==0.56.0 in ./build/venv.linux/lib/python3.7/site-packages (from -r /home/jayvdb/projects/python/python-build-standalone/requirements.txt (line 109)) (0.56.0)
Requirement already satisfied: zstandard==0.12.0 in ./build/venv.linux/lib/python3.7/site-packages (from -r /home/jayvdb/projects/python/python-build-standalone/requirements.txt (line 113)) (0.12.0)
You are using pip version 19.0.3, however version 19.3.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
/home/jayvdb/projects/python/python-build-standalone/cpython-unix/build.py image-clang
image-clang> Step 1/9 : FROM debian@sha256:37103c15605251b2e35b70a3214af626a55cff39abbaadccd01ff828ee7005e0
```

Move config-3.7m-x86_64-linux-gnu folder outside of install ?

Following issue #44, I've successfully end up with a POC on linux 64 of embedding python-build-standalone into my project (see touilleMan/godot-python#181 if you're curious about it ^^)

One small pain-point I've encountered is the python/install/lib/python3.7/config-3.7m-x86_64-linux-gnu/ folder.
This folder contains the libpython.a and some other compilation related stuff. Hence this folder is not useful for running python so I'm a bit puzzled why it is in the install part.
On top of that this folder is ~32mo big (so about 1/4 of the total size of install) which make it very tempting to remove. However this is not trivial given it name depends of python version & target triple.

So I think it would be better to move this folder outside of the install directory, or if it's not possible to document it path inside the PYTHON.json manifest so removing it wouldn't require to use globing ;-)

Python 3.8 build invalid with missing python3.8m

I tried building from the source for a Python 3.8 build. I tried either of the following commands:

./build-linux.py --musl --libressl --python cpython-3.8
./build-linux.py --optimized --python cpython-3.8

The build completed the binary executables link to the python3.8m file which as of Python 3.8 appears to have been removed. Also the lib/ folder also contains the invalid libpython3.8m.a.

Is there a proper way to build for 3.8 or is it not yet supported all together.

Issue with build cpython-3.8 on windows 10

Hi, I try to build cpython-3.8 on Windows 10 and got the issue:

Traceback (most recent call last):
  File "build.py", line 22, in <module>
    from pythonbuild.utils import (
  File "C:\Work\python-build-standalone\pythonbuild\utils.py", line 15, in <module>
    import zstandard
  File "C:\Work\python-build-standalone\build\venv\lib\site-packages\zstandard\__init__.py", line 36, in <module>
    import zstd
ModuleNotFoundError: No module named 'zstd'

Could someone suggest what to do?

Support MinGW builds

#13 outlines some of the reasons why MinGW builds might be advantageous, but there is also the benefit of gcc/clang toolchain being more managable, and the possibility of using cross-compiling.

The MinGW builds of Python are quite mature, and well maintained with a long list of patches at

https://github.com/msys2/MINGW-packages/tree/master/mingw-w64-python3

There are around 350 Python packages being built and tested in that repo, some with extra patches required.

I have built around around 150 myself on top of wine at https://build.opensuse.org/project/show/home:jayvdb:wine-python3:py37:packages & https://build.opensuse.org/project/show/home:jayvdb:wine-python3:py37:packages-p2 , most with tests running. Builds there disabled to avoid unnecessary load on OBS, but I was able to build standalone CLI apps with it.
(Also Nuitka-built apps at https://build.opensuse.org/project/show/home:jayvdb:wine-python3:py37:packages-Nuitka ;-) )

Unable to get libXau from its mirror

Received an error (FTP -> ConnectionRefused) when trying to obtain libXau-1.0.7.tar.gz from its provided mirror.

Replaced that mirror with Xorg's primary mirror in Oregon to circumvent. Considered a patch that turns entry["url"] into entry["urls"] and handles this failure by trying the next mirror in a list, but the handling is a bit bizarre in urllib and ftplib.

cpython-3.7.7-linux64-20200409T0045 links against libnsl.so.1, which Fedora no longer has

I'm not sure what to do about this offhand, but it looks like Fedora dropped libnsl.so.1 in Fedora 28 or so (32 is current). Filing this so we at least don't lose track. STR:

[greenwood:Programming/hg/crew] augie% pyoxidizer build --path ./rust/hgcli --release
resolving 1 targets
resolving target app_posix
resolving target distribution_posix
resolving target exe_posix
resolving Python distribution Url { url: "https://github.com/indygreg/python-build-standalone/releases/download/20200408/cpython-3.7.7-linux64-20200409T0045.tar.zst", sha256: "74799ae3b7f3ddc2d118516d65d46356fb3ef3ff3c4c4591a0dde073c413aff0" }
Python distribution available at /home/augie/Programming/hg/crew/build/pyoxidizer/python_distributions/cpython-3.7.7-linux64-20200409T0045.tar.zst
reading data from Python distribution...
/home/augie/Programming/hg/crew/build/pyoxidizer/python_distributions/python.74799ae3b7f3/python/install/bin/python3.7m: error while loading shared libraries: libnsl.so.1: cannot open shared object file: No such file or directory
error[PYOXIDIZER_BUILD]: EOF while parsing a value at line 1 column 0
  --> ./rust/hgcli/pyoxidizer.bzl:26:11
   |
26 |       exe = dist.to_python_executable(
   |  ___________^
27 | |         name = "hg",
28 | |         resources_policy = "prefer-in-memory-fallback-filesystem-relative:lib",
29 | |         config = config,
...  |
32 | |         extension_module_filter = "all",
33 | |     )
   | |_____^ resolve_distribution()


error: EOF while parsing a value at line 1 column 0

Build failure for x86_64-unknown-linux-musl on Ubuntu 18.04.2-LTS

Hey there! First, thanks for this project. It will be quite a helpful building block for me on a few projects of mine.

For now however, I'm unable tomusl build for Linux x64.

Environment

I'm building using Purism's Librem15 (product page, product review), which sports an Intel i7-6500U @ 2.5Ghz x4 and 7.7GiB of RAM.

Problem

I've invoked the build from the HEAD of master using:

~/code/python-build-standalone $ python3 -V
Python 3.6.9
~/code/python-build-standalone $ pip -V
pip 20.1 from /home/user/.pyenv/versions/3.7.5/envs/build_standalone/lib/python3.7/site-packages/pip (python 3.7)
~/code/python-build-standalone $ python3 build-linux.py --python=cpython-3.8 --optimizations=pgo+lto --target=x86_64-unknown-linux-musl

I was able to see the build pull/build docker images and build most of the tools in the toolchain. But when it was working on clang, it choked with an error while trying to compile SemaDeclAttr.cpp.o:

clang> [3021/4071] Building CXX object tools/clang/lib/Sema/CMakeFiles/obj.clangSema.dir/SemaConcept.cpp.o
clang> [3022/4071] Building CXX object tools/clang/lib/Sema/CMakeFiles/obj.clangSema.dir/SemaCUDA.cpp.o
clang> [3023/4071] Building CXX object tools/clang/lib/Sema/CMakeFiles/obj.clangSema.dir/SemaCodeComplete.cpp.o
clang> [3024/4071] Building CXX object tools/clang/lib/Sema/CMakeFiles/obj.clangSema.dir/SemaChecking.cpp.o
clang> [3025/4071] Building CXX object tools/clang/lib/Sema/CMakeFiles/obj.clangSema.dir/SemaDeclAttr.cpp.o
clang> FAILED: tools/clang/lib/Sema/CMakeFiles/obj.clangSema.dir/SemaDeclAttr.cpp.o
clang> /tools/host/bin/g++  -DGTEST_HAS_RTTI=0 -D_GNU_SOURCE -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -D__STDC_LIMIT_MACROS -Itools/clang/lib/Sema -I/build/llvm/tools/clang/lib/Sema -I/build/llvm/tools/clang/include -Itools/clang/include -Iinclude -I/build/llvm/include -Wno-cast-function-type -fPIC -fvisibility-inlines-hidden -Werror=date-time -Wall -Wextra -Wno-unused-parameter -Wwrite-strings -Wcast-qual -Wno-missing-field-initializers -pedantic -Wno-long-long -Wimplicit-fallthrough -Wno-maybe-uninitialized -Wno-class-memaccess -Wno-redundant-move -Wno-noexcept-type -Wdelete-non-virtual-dtor -Wno-comment -fdiagnostics-color -ffunction-sections -fdata-sections -fno-common -Woverloaded-virtual -fno-strict-aliasing -O3 -DNDEBUG    -fno-exceptions -fno-rtti -std=c++14 -MD -MT tools/clang/lib/Sema/CMakeFiles/obj.clangSema.dir/SemaDeclAttr.cpp.o -MF tools/clang/lib/Sema/CMakeFiles/obj.clangSema.dir/SemaDeclAttr.cpp.o.d -o tools/clang/lib/Sema/CMakeFiles/obj.clangSema.dir/SemaDeclAttr.cpp.o -c /build/llvm/tools/clang/lib/Sema/SemaDeclAttr.cpp
clang> g++: fatal error: Killed signal terminated program cc1plus
clang> compilation terminated.
clang> [3026/4071] Building CXX object tools/clang/lib/Sema/CMakeFiles/obj.clangSema.dir/SemaExceptionSpec.cpp.o
clang> [3027/4071] Building CXX object tools/clang/lib/Sema/CMakeFiles/obj.clangSema.dir/SemaDeclObjC.cpp.o
clang> [3028/4071] Building CXX object tools/clang/lib/Sema/CMakeFiles/obj.clangSema.dir/SemaDecl.cpp.o
clang> [3029/4071] Building CXX object tools/clang/lib/Sema/CMakeFiles/obj.clangSema.dir/SemaDeclCXX.cpp.o
clang> [3030/4071] Building CXX object tools/clang/lib/Sema/CMakeFiles/obj.clangSema.dir/SemaExpr.cpp.o
clang> ninja: build stopped: subcommand failed.

Workarounds I've tried

This StackOverflow thread suggested that the problem was memory limitations. As such, I tried:

  1. Compiling without the --optimizations flag, which produced the same failure.
  2. Modifying pythonbuild/buildenv.py to add mem_limit="4g" (as documented here) to the container instantiation, which led to the same failure yet again.

Here are the logs produced by the latest build -- with the mem_limit="4g" modification, but many of the images have not changed since their first successful runs (and the logs for them may not have been updated, either? I'm not certain)

Support WinXP

If possible and not to difficult. Would be great to have static python for XP.

For XP support one needs:

  1. Use Python 3.4 sources, this is the last python version with XP support.
  2. Do needed tweaks in pcbuild+msbuild+vcxproj files. PCBuild folder in sources has different .vcxproj/.props files than in Py 3.7.
  3. Use any MSVC version, 2019 is fine also.
  4. Compile all .c/.cpp files with MSVC cl flag /D_USING_V110_SDK71_, link all binaries with MSVC link flag /SUBSYSTEM:WINDOWS,5.01 (for win apps) and /SUBSYSTEM:CONSOLE,5.01 (for console apps).
  5. Also check what versions of libs-dependencies support XP. Or just try to compile and run in XP.

pip issues using 3.7.6 build

Initial tests with the python interpreter looks OK but for some reason, pip doesn't seem to work:

[root@bfat3-fedora-1 bin]# ./pip3 install distro
-bash: ./pip3: /build/out/python/install/bin/python3: bad interpreter: No such file or directory
[root@bfat3-fedora-1 bin]# ./python3 -m pip install distro
ERROR: Could not install packages due to an EnvironmentError: Dynamic loading not supported

I'm not sure why the absolute path of I suppose the original build location is referenced.

Improve Windows build process

A couple of things I have in mind, will send a PR if I am successful in implementing them:

  • Use Strawberry Perl to build OpenSSL, instead of depending on the PERL environment variable. Strawberry Perl has a nice portable distribution.
  • Automatically set environment variables a la distutils, so I donโ€™t forget to run vcvarsall.bat before building.

macOS linked Tcl/Tk versions

Hello!

First and foremost, thanks for sharing this work. :-)

My name is Tiago and I've been contributing to Mu Editor for a while now. It's a beginner friendly Python Editor and one of the challenges we face is, like many other projects, packaging it for end-user distribution. In particular on macOS and Windows, we ship a packaged version that looks and feels like pretty much other desktop applications.

Context

As of today, Mu is packaged with BeeWare's Briefcase on macOS and with pynsist on Windows. It is also packaged for some Linux distros, including Raspbian, and as a pure Python wheel, on PyPI.

Recent progress on BeeWare's Briefcase project lead me to re-evaluate it as more general and unified tool for packaging Mu for all/most targets.

Briefcase uses the concept of a "support package" which is a redistributable and relocatable Python installation, much like what this project is sharing. For that reason the BeeWare team also puts efforts into creating such redistributable Python installations.

My Experiment

While exploring the recent progress in Briefcase I confirmed BeeWare's Python still does not include modules such as tkinter, pip, or venv -- these are required by Mu (see beeware/briefcase#383). Up until today, we have been using a taylor-made Briefcase "support package" (Python installation), but that does feel like a waste of efforts -- how many projects out there are building redistributable Python installations?

Wondering if I could avoid the burden of managing Mu's own "support package", I decided to try out Python Build Standalone -- would it include the bits and pieces we need? how easy would it be to feed into Briefcase?

What I tried:

  • Downloaded the most recent 3.7.7 release for both macOS and Windows.
  • Played around with it interactively, at the REPL, in particular with tkinter, pip, and venv.
  • All working out of the box. Great!

Then, on macOS only:

  • Slighly adjusted the payloads to make them usable with Briefcase.
  • Successfully packaged a tkinter based GUI application on macOS with your Python build.
  • Great!

Where I Stumbled

Pushing it a little further, I created a very minimal tkinter based application -- a single window, displaying an embedded image file, in PNG format.

It kept failing on macOS until I realized that the underlying Tcl/Tk versions are 8.5, with no PNG image support, instead of 8.6 like the ones included in the CPython distributions from www.python.org. I then checked your Windows distribution only to see that it bundles Tcl/Tk 8.6.

I then skimmed through the build scripts in this project and, seeing references to Tcl/Tk downloads, I ran a build on my macOS system, hoping it would bring in Tcl/Tk 8.6. It did not.. (in retrospect, why would it, assuming your releases are built the same way?) :-)

Question

Not being into the nitty gritty details about the whole build process, how feasible/easy would it be to update the macOS builds to include updated Tcl/Tk? I suspect the current versions use the system included ones which are, at least on my system, running Mojave 10.14.x, in fact, 8.5.

Thanks again for sharing this project and for any feedback and pointers in how to attain my objective -- having a common, up to date, Tcl/Tk version across the different platforms.

cat: Setup.local: Permission denied with build-linux

I'm getting a cat: Setup.local: Permission denied when I run ./build-linux.py. It happens at the cpython-linux64 stage during the build. Full error log:

cpython-linux64> cat: Setup.local: Permission denied
Traceback (most recent call last):
  File "/Users/zakichammaa/Documents/workspace/obkio/python-build-standalone/cpython-unix/build.py", line 867, in <module>
    sys.exit(main())
  File "/Users/zakichammaa/Documents/workspace/obkio/python-build-standalone/cpython-unix/build.py", line 858, in main
    version=os.environ["PYBUILD_PYTHON_VERSION"][0:3],
  File "/Users/zakichammaa/Documents/workspace/obkio/python-build-standalone/cpython-unix/build.py", line 596, in build_cpython
    build_env.run("build-cpython.sh", environment=env)
  File "/Users/zakichammaa/Documents/workspace/obkio/python-build-standalone/pythonbuild/buildenv.py", line 64, in run
    container_exec(self.container, program, user=user, environment=environment)
  File "/Users/zakichammaa/Documents/workspace/obkio/python-build-standalone/pythonbuild/docker.py", line 128, in container_exec
    raise Exception("exit code %d from %s" % (inspect_res["ExitCode"], command))
Exception: exit code 1 from /build/build-cpython.sh
make: *** [/Users/zakichammaa/Documents/workspace/obkio/python-build-standalone/build/cpython-3.7.4-linux64.tar] Error 1

I've tried to run the command with sudo in the build-cpython.sh script, but it requires a password (adding a password to the user creates other problems). Just wondering if you've encountered this problem, and if so, how did you solve it. Thanks!

Compile/Link static distribution with static C/C++ Run Time Lib

Would be great if everything in static python distribution is compiled/linked with static CRT. So that libs are really static, not needing to install any extra C/C++ redistributable package in the system.

E.g. in Win/MSVC compile all libs with /MT flag instead of /MD.

In GCC use -static/-static-libstdc++/-static-libgcc flags for compile. Although maybe static builds in Linux world is not that necessary as in Win, but still will be good.

BTW, at least for MSVC if a lib was compiled statically (with /MT) then final application can use both /MT and /MD and link that library. While if a lib was dynamical (with /MD) then application can't use /MT, there will be link errors. In that sense /MT library is more universal and useful.

Windows python.exe should support shared libs

The provided python.exe is unable to load shared extension libs, defining Py_NO_ENABLE_SHARED, and the hacked pyport.h for Windows replaces all the nested extern sym declare alternatives with only the definitions needed to create the .exe.

#define PyAPI_FUNC(RTYPE) __declspec(dllexport) RTYPE
#define PyAPI_DATA(RTYPE) extern __declspec(dllexport) RTYPE
#define PyMODINIT_FUNC __declspec(dllexport) PyObject*

i.e. excluding the definitions like __declspec(dllimport) needed to build dlls.

This presents a problem especially for built extensions which rely on cffi to build, which includes cryptography and gevent and many more packages in the core of the Python ecosystem. cffi often isnt needed after the build -- it only needs to exist while running setup.py for the package.

afaic, the python.exe inside the distribution is not used directly the PyOxidizer app re-packaging, but is used for running pip and the compile scripts. As a result, python.exe can be a shared-enabled binary, and even be a thin exe with shared pythonXY.dll, so that it can be more useful for running pip to build shared libraries.

(Another hack around this would be to also provide _cffi_backend as part of the python-build-standalone/ distribution, similar to how PyPy provides it as a core component, but build it into the static python.exe)

Just a heads up

There's a project tackling portable builds on linux called build-anywhere located here: https://github.com/theopolis/build-anywhere

I don't know if it is useful, just thought i should tell. Before i found python-build-standalone project i tried to build the _ctypes extension statically(still can't get it to work) through build-anywhere project.

There's another thing though. Following this: https://wiki.python.org/moin/BuildStatically in addition to build-anywhere i can build a python binary with no deps. I can't get _ctypes to compile though so i can't really test it. But isn't this better than the binary depending on a lot of stuff gotten through this project or am i confused. Maybe those deps are to static builds?

pip install fail because of missing musl-clang

I could not find musl-clang package for linux. Do you know how can I get this package?

Python 3.7.7 musl version
Error message:
unable to execute 'musl-clang': No such file or directory
error: command 'musl-clang' failed with exit status 1

Fails to build on OSX: `dyld: Library not loaded: @executable_path/../.Python`

Cannot build on OSX.

$ git clone https://github.com/indygreg/python-build-standalone.git
$ cd python-build-standalone
$ ./build-macos.py
Collecting certifi==2019.6.16 (from -r /Users/alanjds/src/git/python-build-standalone/requirements.txt (line 7))
  Downloading https://files.pythonhosted.org/packages/69/1b/b853c7a9d4f6a6d00749e94eb6f3a041e342a885b87340b79c1ef73e3a78/certifi-2019.6.16-py2.py3-none-any.whl (157kB)
    100% |โ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆ| 163kB 2.4MB/s
Collecting cffi==1.12.3 (from -r /Users/alanjds/src/git/python-build-standalone/requirements.txt (line 11))
  Cache entry deserialization failed, entry ignored
  Downloading https://files.pythonhosted.org/packages/f0/48/5aa4ea664eba26dd5142558d04762f5065c02220b4665b3f7eecb9bb614e/cffi-1.12.3-cp37-cp37m-macosx_10_9_x86_64.whl (169kB)
    100% |โ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆ| 174kB 467kB/s
Collecting chardet==3.0.4 (from -r /Users/alanjds/src/git/python-build-standalone/requirements.txt (line 41))
  Using cached https://files.pythonhosted.org/packages/bc/a9/01ffebfb562e4274b6487b4bb1ddec7ca55ec7510b22e4c51f14098443b8/chardet-3.0.4-py2.py3-none-any.whl
Collecting click==7.0 (from -r /Users/alanjds/src/git/python-build-standalone/requirements.txt (line 45))
  Downloading https://files.pythonhosted.org/packages/fa/37/45185cb5abbc30d7257104c434fe0b07e5a195a6847506c074527aa599ec/Click-7.0-py2.py3-none-any.whl (81kB)
    100% |โ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆ| 81kB 2.8MB/s
Collecting docker==4.0.2 (from -r /Users/alanjds/src/git/python-build-standalone/requirements.txt (line 49))
  Downloading https://files.pythonhosted.org/packages/95/47/5560c9cf0c92b50da24216f0e7733250fbed5a497f69e3c70e1be62143fe/docker-4.0.2-py2.py3-none-any.whl (138kB)
    100% |โ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆ| 143kB 482kB/s
Collecting idna==2.8 (from -r /Users/alanjds/src/git/python-build-standalone/requirements.txt (line 52))
  Using cached https://files.pythonhosted.org/packages/14/2c/cd551d81dbe15200be1cf41cd03869a46fe7226e7450af7a6545bfc474c9/idna-2.8-py2.py3-none-any.whl
Collecting jinja2==2.10.1 (from -r /Users/alanjds/src/git/python-build-standalone/requirements.txt (line 56))
  Using cached https://files.pythonhosted.org/packages/1d/e7/fd8b501e7a6dfe492a433deb7b9d833d39ca74916fa8bc63dd1a4947a671/Jinja2-2.10.1-py2.py3-none-any.whl
Collecting markupsafe==1.1.1 (from -r /Users/alanjds/src/git/python-build-standalone/requirements.txt (line 59))
  Downloading https://files.pythonhosted.org/packages/ce/c6/f000f1af136ef74e4a95e33785921c73595c5390403f102e9b231b065b7a/MarkupSafe-1.1.1-cp37-cp37m-macosx_10_6_intel.whl
Collecting pip-tools==4.0.0 (from -r /Users/alanjds/src/git/python-build-standalone/requirements.txt (line 89))
  Downloading https://files.pythonhosted.org/packages/49/91/69286bbcf185be26a1d3fa555bfa8b54ea4cb916b50485e9dfc0dd9fa638/pip_tools-4.0.0-py2.py3-none-any.whl
Collecting pycparser==2.19 (from -r /Users/alanjds/src/git/python-build-standalone/requirements.txt (line 92))
  Cache entry deserialization failed, entry ignored
  Using cached https://files.pythonhosted.org/packages/68/9e/49196946aee219aead1290e00d1e7fdeab8567783e83e1b9ab5585e6206a/pycparser-2.19.tar.gz
Collecting requests==2.22.0 (from -r /Users/alanjds/src/git/python-build-standalone/requirements.txt (line 95))
  Using cached https://files.pythonhosted.org/packages/51/bd/23c926cd341ea6b7dd0b2a00aba99ae0f828be89d72b2190f27c11d4b7fb/requests-2.22.0-py2.py3-none-any.whl
Collecting six==1.12.0 (from -r /Users/alanjds/src/git/python-build-standalone/requirements.txt (line 99))
  Using cached https://files.pythonhosted.org/packages/73/fb/00a976f728d0d1fecfe898238ce23f502a721c0ac0ecfedb80e0d88c64e9/six-1.12.0-py2.py3-none-any.whl
Collecting urllib3==1.25.3 (from -r /Users/alanjds/src/git/python-build-standalone/requirements.txt (line 103))
  Downloading https://files.pythonhosted.org/packages/e6/60/247f23a7121ae632d62811ba7f273d0e58972d75e58a94d329d51550a47d/urllib3-1.25.3-py2.py3-none-any.whl (150kB)
    100% |โ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆ| 153kB 1.7MB/s
Collecting websocket-client==0.56.0 (from -r /Users/alanjds/src/git/python-build-standalone/requirements.txt (line 107))
  Using cached https://files.pythonhosted.org/packages/29/19/44753eab1fdb50770ac69605527e8859468f3c0fd7dc5a76dd9c4dbd7906/websocket_client-0.56.0-py2.py3-none-any.whl
Collecting zstandard==0.11.1 (from -r /Users/alanjds/src/git/python-build-standalone/requirements.txt (line 111))
  Downloading https://files.pythonhosted.org/packages/33/5f/f01995dd1946f89fcc4a56bddd980fdab21c333afb99bb052cef7221a02e/zstandard-0.11.1-cp37-cp37m-macosx_10_6_intel.whl (632kB)
    100% |โ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆ| 634kB 1.8MB/s
Installing collected packages: certifi, pycparser, cffi, chardet, click, six, websocket-client, urllib3, idna, requests, docker, markupsafe, jinja2, pip-tools, zstandard
  Running setup.py install for pycparser ... done
Successfully installed certifi-2019.6.16 cffi-1.12.3 chardet-3.0.4 click-7.0 docker-4.0.2 idna-2.8 jinja2-2.10.1 markupsafe-1.1.1 pip-tools-4.0.0 pycparser-2.19 requests-2.22.0 six-1.12.0 urllib3-1.25.3 websocket-client-0.56.0 zstandard-0.11.1
You are using pip version 10.0.1, however version 19.2.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
dyld: Library not loaded: @executable_path/../.Python
  Referenced from: /Users/alanjds/src/git/python-build-standalone/build/venv/bin/python
  Reason: image not found

And needs to run on system Python. On a virtualenv the script outputs nothing.

Windows 10 cpython-3.8 build fails with autoreconf

I try to build python.exe build-windows.py --python cpython-3.8 --sh C:\Soft\cygwin64\bin\sh.exe --profile static-noopt and got the following exception:

'.toreconf-2.69: unrecognized option '- autoreconf-2.69: Try '/usr/bin/autoreconf-2.69 --help' for more information. Traceback (most recent call last): File "build.py", line 2267, in <module> sys.exit(main()) File "build.py", line 2241, in main build_libffi( File "build.py", line 1485, in build_libffi subprocess.run(args, env=env, check=True) File "C:\Soft\Python\Python38\lib\subprocess.py", line 512, in run raise CalledProcessError(retcode, process.args, subprocess.CalledProcessError: Command '['C:\\Users\\MyUser\\AppData\\Local\\Temp\\libffi-build-hxqnsout\\Python-3.8.3\\PCBuild\\prepare_libffi.bat', '-pdb', '-x64']' returned non-zero exit status 1.

Could someone help? When I build cpython-3.7 everything is ok.

Unable to compile cpython-3.x with optimizations on Ubuntu

Summary

I've tried to build Python on both of the Ubuntu platforms listed below, and the process always fails on cpython. Over several iterations on the Ubuntu server VM:

  • Tried to compile with flags --python=cypthon-3.8 and --optimizations=pgo+lto, but failed (gist).
  • Tried to compile with flags --python=cypthon-3.7 and --optimizations=pgo+lto, but failed (gist).
  • Tried to compile with flags --python=cypthon-3.7 and --optimizations=pgo, but failed (gist).
  • Tried to compile with the --python=cpython-3.7 flag but no optimizations and succeeded! (download and slice off the .zip from the end, sha256 is f994d12161adcf3ced5ad2daf964cab6e8f70913cc90534ee3ac1feb41420404).
  • Tried to compile with the --python=cpython-3.8 flag but no optimizations and succeeded! (download and slice off the .zip from the end, sha256 is 9743d8927e82503bd464bf645014ade886a01f29a8d20999081d4acb95050f41)

With this evidence, I'd have to say that @indygreg is probably right about musl not liking optimization.

Host Information

Platform Laptop
Notes Default development machine
System Model Purism Librem 15v3 w/ TPM
Operating System Ubuntu 18.04 LTS x86_64
Kernel 5.3.0-51-generic #44~18.04.2-Ubuntu SMP Thu Apr 23 14:27:18 UTC 2020
CPU Intel Core i7-6500U @ 2.5GHz (x4)
RAM 8GB
Platform Tower PC
Notes Used only to host the Ubuntu server VM, via VirtualBox v6.1.6 r137129 (Qt5.6.2); not used for building itself!
System Model Beefy (custom) build
Operating System Windows 10 Pro 64-bit, v1909 (build 18363.836)
CPU AMD Ryzen 9 3900X (x12)
RAM 32GB
Platform Ubuntu server (virtualized)
Notes AMD-V hypervisor, Nested paging, PAE/NX, and KVM paravirtualization are all enabled
Operating System Ubuntu Server 20.04 LTS x86_64
Kervel 5.4.0-29-generic #33-Ubuntu SMP Wed Apr 29 14:32:27 UTC 2020
CPU AMD Ryzen 9 3900X (x12)
RAM 16GB

python3.dll is missing on windows

python3.dll seems to be needed by binary wheels (see touilleMan/godot-python#228 for more details), but it is missing from the install folder:

$ ls C:\Users\gbleu\AppData\Local\Programs\Python\Python38-32
DLLs/  Doc/  LICENSE.txt  Lib/  NEWS.txt  Scripts/  Tools/  include/  libs/  python.exe*  python3.dll*  python38.dll*  pythonw.exe*  tcl/  vcruntime140.dll*

$  ls cpython-3.8.3-x86_64-pc-windows-msvc-shared-pgo-20200517T2207\python\install\
DLLs/  LICENSE.txt  Lib/  Scripts/  include/  libs/  python.exe*  python.pdb  python38.dll*  python38.pdb  pythonw.exe*  pythonw.pdb  tcl/  vcruntime140.dll*  vcruntime140_1.dll*

I'm a bit puzzled for the reason of this omission given it seems to be already taken care of here:

if static:
static_replace_in_file(
pcbuild_proj, b'<Projects Include="python3dll.vcxproj" />', b""
)

Static test modules

I was surprised to see the test modules are being built in. Wouldn't that increase the size for no particular benefit?

I didnt see the test suite being run, although I didnt look very hard.

$ git grep test
..
static-modules.linux64:_ctypes_test _ctypes/_ctypes_test.c -I/tools/deps/lib/libffi-3.2.1/include -lm
static-modules.linux64:_testbuffer _testbuffer.c
static-modules.linux64:_testimportmultiple _testimportmultiple.c
static-modules.linux64:_testmultiphase _testmultiphase.c
static-modules.linux64:_xxtestfuzz _xxtestfuzz/_xxtestfuzz.c _xxtestfuzz/fuzzer.c
static-modules.macos:_ctypes_test _ctypes/_ctypes_test.c -lm
static-modules.macos:_testbuffer _testbuffer.c
static-modules.macos:_testimportmultiple _testimportmultiple.c
static-modules.macos:_testmultiphase _testmultiphase.c
static-modules.macos:_xxtestfuzz _xxtestfuzz/_xxtestfuzz.c _xxtestfuzz/fuzzer.c

Missing lib-dynload in macOS builds

install/lib/python3.7/lib-dynload is missing in the macOS builds, breaking exec_prefix detection:

Could not find platform dependent libraries <exec_prefix>
Consider setting $PYTHONHOME to <prefix>[:<exec_prefix>]
Python 3.7.3 (default, Jun 17 2019, 22:24:24)
[Clang 6.0.1 (tags/RELEASE_601/final)] on darwin

Creating an empty directory fixes it, but I wonder if anything else is missing as well? Running
build-macos.py fails for me on macOS 10.14.5, so I couldn't track this down:

cpython-macos>  CC='clang' LDSHARED='clang -bundle -undefined dynamic_lookup -L/private/var/folders/jp/gp7mqwc963vc1487847t8b180000gn/T/tmpajyuh1zi/deps/lib   ' OPT='-DNDEBUG -g -fwrapv -O3 -Wall' 	_TCLTK_INCLUDES='' _TCLTK_LIBS='' 	./python.exe -E ./setup.py  build
cpython-macos> running build
cpython-macos> running build_ext
cpython-macos> error: [Errno 2] No such file or directory: '/install/include/python3.7m/pyconfig.h'
cpython-macos> make[1]: *** [sharedmods] Error 1

Windows runtime dependencies

The README doesnt list the runtime requirements on Windows. ping @uranusjr, I guess you can help with this?

This is a significant omission contrasting with the care given to the licensing choices and documentation for Linux/macOS.

I guess that the 'standalone' runtime still depends on Microsoft redistributables, which have shrink-wrap licenses that need to be carefully considered, so they should be documented.

Ideally any non-OS runtime dependencies are eliminated where possible.

Build 32-bit linux with musl

Hello!
In order to install python on quite old hardware, I want to build 32-bit package with musl (binary libraries are also very old).

In my case it is not even 686, it is 486 without FPU.

Python REPL (on Linux) has strange behaviour for delete and arrow keys

Hi, I've only tested this on Linux (Ubuntu 18.04) using this build.

When I enter the Python REPL and try to interact with it via the delete and arrow keys (for history) I get all sorts of strange behaviour. Delete simply doesn't update the screen (although the characters in the buffer are deleted, they don't look that way), and the command history accessed via the arrow keys doesn't update the screen correctly either.

You can see what I mean in the following screenie:

static_python

Turtle (tkinter) causes errors

Hi, thanks for this project. It's going to be really useful for some of my projects used in educational settings..!

Using this build I tried to use the turtle module. It results in a failure, by the looks of it, a missing file or absence of a path. When running from python/install/bin:

$ ./python3.7m
Python 3.7.7 (default, Apr  9 2020, 00:46:21) 
[Clang 9.0.1 ] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import turtle
>>> turtle.forward(100)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<string>", line 6, in forward
  File "/home/ntoll/src/mu-editor/Mu-naco/python/linux64/python/install/lib/python3.7/turtle.py", line 3812, in __init__
    Turtle._screen = Screen()
  File "/home/ntoll/src/mu-editor/Mu-naco/python/linux64/python/install/lib/python3.7/turtle.py", line 3662, in Screen
    Turtle._screen = _Screen()
  File "/home/ntoll/src/mu-editor/Mu-naco/python/linux64/python/install/lib/python3.7/turtle.py", line 3678, in __init__
    _Screen._root = self._root = _Root()
  File "/home/ntoll/src/mu-editor/Mu-naco/python/linux64/python/install/lib/python3.7/turtle.py", line 434, in __init__
    TK.Tk.__init__(self)
  File "/home/ntoll/src/mu-editor/Mu-naco/python/linux64/python/install/lib/python3.7/tkinter/__init__.py", line 2023, in __init__
    self.tk = _tkinter.create(screenName, baseName, className, interactive, wantobjects, useTk, sync, use)
_tkinter.TclError: Can't find a usable init.tcl in the following directories: 
    /tools/deps/lib/tcl8.6 /home/ntoll/src/mu-editor/Mu-naco/python/linux64/python/install/lib/tcl8.6 /home/ntoll/src/mu-editor/Mu-naco/python/linux64/python/lib/tcl8.6 /home/ntoll/src/mu-editor/Mu-naco/python/linux64/python/install/library /home/ntoll/src/mu-editor/Mu-naco/python/linux64/python/library /home/ntoll/src/mu-editor/Mu-naco/python/linux64/python/tcl8.6.10/library /home/ntoll/src/mu-editor/Mu-naco/python/linux64/tcl8.6.10/library



This probably means that Tcl wasn't installed properly.

I've not tested this yet on Windows or OSX. My feeling is some asset or other isn't getting copied to the correct location when the build is generated. Happy to help in any way I can... got questions, just ask..! Thank you in advance for your help..! ๐Ÿ‘

removing PYTHONHOME dependency from a static libpython?

Hello, First thank you for a great package. I spent quite a bit of time trial and erroring what python-build-standalone does well.

My question is that I have a simple c program which is essentially the example from the python doc on how to embed libpython.
I've managed to compile a fully static exec. As you've already solved the issue of module importing with PyOxidizer for a python executable, would you have any thoughts or suggestions on the libpython embedded side? I mean beyond the trivial mechanism of storing the files in the exec and using local disk or shm to extract out the files when needed? I do not want to modify
python source if I can help it as that would make any method highly dependent on the specific python version. But perhaps modifying the Python module initialization to load from memory instead of a file is the easiest way forward.

Any feedback would be highly appreciated.

Thank you,

-mp

Segmentation Fault in base docker image build

Can you possibly add some documentation on this to the README @indygreg or migrate to a minimum of jessie as the base image?

For host machines running kernels after 4.11, vsyscall has been disabled by default for security reasons. However, we can emulate the vsyscall by adding an option to the bootloader.

  • For GRUB: Append vsyscall=emulate to the GRUB_CMDLINE_LINUX_DEFAULT

GRUB_CMDLINE_LINUX_DEFAULT="... vsyscall=emulate"

  • For systemd-boot:

Edit your /boot/loader/entries/*.conf and append vsyscall=emulate to the options line

No static library available for macOS

This is more of a question but I wasn't sure how to contact the author any other way.

I've built for macOS but I'm not seeing there is a static library available. I need an absolute barebones Python 3.x library for use as a scripting bridge in my program. No other packages are needed. Is this possible using your project?

Thank you for your time.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.