PySide6 fails to load shiboken.Shiboken6 with Python3.12 about nuitka HOT 19 CLOSED

Since I am rebasing a lot, I don't think I would appreciate that if they were notified each time I rebase factory and even develop and main. But I never really tried it. Honestly, looking up the issue number is an extra step that I am skipping due to the sheer mass of things I am working on simultaneously, not on branches. I cannot keep up with that, I feel and the commit you linked will become dangling, because I will do my usual hotfix workflow, where it's rebased onto the hotfix merge where it may or may not be in cherry-picked. Likely not for this one though, but the commit is subject to change.

Mercurial had a thing, where commits had a history, but that didn't make it into git unfortunately. I think I might have liked that.

from nuitka.

While I am working on a MWE, I have come across this additional error:

project.bin: static_src/CompiledFunctionType.c:1267: Nuitka_Function_New: Assertion `Py_REFCNT((&_Py_NoneStruct)) == _Py_IMMORTAL_REFCNT' failed.
Aborted

from nuitka.

That's very strange, at least our_tuple_richcompare must be a false positive, since it's only True or False and these values must be immortal. Unless of course, other bad code, corrupts that state, which could still be in Nuitka, but also elsewhere. Or maybe the assertion is incorrect on some arches, but you got the typical one, so I don't know.

from nuitka.

Will continue working on an MWE. Just takes a while, with each round of compilation taking 20-30 minutes...

For now, verified that this repros under Python 3.12.4 with PySide6 6.7.2 (today's CI version from https://download.qt.io/snapshots/ci/pyside/6.7.2/latest/pyside6/), so it is likely not https://bugreports.qt.io/browse/PYSIDE-2780.

Potentially related: https://bugreports.qt.io/browse/PYSIDE-2781.

from nuitka.

It seems that the indeed PySide6 corrupts recounts of immortal objects, at least in some versions, are now caught by Nuitka debug mode. I guess we need to make the PySide6 plugin disable the immortal checks, which might be required with some C extension modules for a while. I think python-dbg reports those at program exit even.

I think immortal corruption ought to be hard to have an effect. The check is mostly for Nuitka to not assume to be working with an immortal object, when it's not to catch Nuitka making false assumptions about what's immortal.

I will try and add a flag for use of yours.

from nuitka.

MWE creation was slowed down by the fact that the segmentation faults are non-deterministic. Out of 11 runs of the same executable, I get maybe 8 good runs and 3 segfaults. So most of my earlier, seemingly unsuccessful tries of reducing my 10k+ LOC code base may actually have been successful without me noticing...

from nuitka.

This is indeed terrible. I think if you were after another bug to be discovered, you ought to use the new flag to disable the checks for now. I might even add that myself to the PySide plugins. This is of course all beyond my control. I don't think any Nuitka issue comes out of it, and it's not very critical if I understand things correctly, immortal values when no longer being immortal will incur a performance regression, but it takes a lot of leaked or lost references to actually cause trouble.

from nuitka.

I think if you were after another bug to be discovered

Well, I think I am. My executable sporadically does not start under Windows with no error message at all; and I get segmentation faults under Linux; both without --debug. I switched to --debug only to maybe get some information on which component is segfaulting, and maybe I should try that once more now with ----no-debug-immortal-assumptions in 46b58b1. Thanks for pointing me to it! (By the way, if you tag the issues in a comment message, the commits appear in the issue as well, and that makes it very nice to see code changes related to issues.)

from nuitka.

So now that I can create a debug build and all the false positives are out of the way, I see this:

(gdb) run
Starting program: /mnt/c/ws/project/.nuitka/onedir/nuitka.dist/nuitka.bin
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".

Program received signal SIGSEGV, Segmentation fault.
0x00007ffff74ab95c in Shiboken::Module::create(char const*, void*) () from /mnt/c/ws/project/.nuitka/onedir/nuitka.dist/shiboken6/../libshiboken6.abi3.so.6.7

This is looking exactly like https://bugreports.qt.io/browse/PYSIDE-2781 (same Python/nuitka version, segfaults in same lib, similar strack trace, only when deployed, also sporadically) - will wait for a resolution to that one (which has reproduction steps and should be easier to investigate) before continuing to investigate here.

#0  0x00007ffff74ab95c in Shiboken::Module::create(char const*, void*) () from /mnt/c/ws/project/.nuitka/onedir/nuitka.dist/shiboken6/../libshiboken6.abi3.so.6.7      
#1  0x00007ffff74de3f1 in PyInit_Shiboken () from /mnt/c/ws/project/.nuitka/onedir/nuitka.dist/shiboken6/Shiboken.so
#2  0x000055555dd9569d in callIntoExtensionModule (tstate=0x55556082ff18 <_PyRuntime+459704>, full_name=0x55555e76f84f "shiboken6.Shiboken",
    filename=0x7fffffff8090 "/mnt/c/ws/project/.nuitka/onedir/nuitka.dist/shiboken6/Shiboken.so", is_package=false)
    at /home/bers/.pyenv/versions/project3.12/lib/python3.12/site-packages/nuitka/build/static_src/MetaPathBasedLoader.c:840
#3  0x000055555dd95e55 in loadModule (tstate=0x55556082ff18 <_PyRuntime+459704>, module=0x7ffff77c2110, module_name=0x7ffff76e0bf0,
    entry=0x5555606ba840 <meta_path_loader_entries+32128>)
    at /home/bers/.pyenv/versions/project3.12/lib/python3.12/site-packages/nuitka/build/static_src/MetaPathBasedLoader.c:1078
#4  0x000055555dd960d2 in _EXECUTE_EMBEDDED_MODULE (tstate=0x55556082ff18 <_PyRuntime+459704>, module=0x7ffff77c2110, module_name=0x7ffff76e0bf0,
    name=0x7ffff76e0c18 "shiboken6.Shiboken")

from nuitka.

I think I saw this myself at some point, when I had a wrong LD_LIBRARY_PATH pointing to a work Python installation.

from nuitka.

See what I wrote there, I think gdb is to blame, however, maybe there is something to be said about Nuitka being more robust against environment variables that must be opposed to onefile execution.

from nuitka.

Here's an MWE for you to play around with to see if environment variables are to blame. I kind of doubt it - my LD_LIBRARY_PATH is unset or empty.

bug.py

# nuitka-project: --standalone
# nuitka-project: --enable-plugin=pyside6
# nuitka-project: --assume-yes-for-downloads
# nuitka-project: --debug
# https://github.com/Nuitka/Nuitka/issues/2917
# nuitka-project: --experimental=allow-c-warnings
# https://github.com/Nuitka/Nuitka/issues/2920
# nuitka-project: --no-debug-immortal-assumptions
# uv pip install https://github.com/Nuitka/Nuitka/archive/46b58b1e6b2f213972a59b3dd8fa7e278018f756.zip

from PySide6 import QtCore  # noqa

bug.sh

#!/bin/bash
python -m nuitka bug.py
gdb -ex=run bug.dist/bug.bin

from nuitka.

I am working on it now, this seems relevant. First, for my convenience, I am ensuring that --run leaks nothing, so this can become an actual identity test one liner.

from nuitka.

So, after doing this, I see no evidence of the running program seeing any environment variables anymore, and I don't see the Python of gdb leak through. I have seen for sure in the past though, that the standalone environment of Nuitka crashes the Python inside of gdb.

With Python3.11, my current PySide6 tests works fine. With Python3.12 a newly added non-deployment catcher for segfaults triggers:

Nuitka: A segmentation fault has occurred. This is highly unusual and can
have multiple reasons. Please check https://nuitka.net/info/segfault.html
for solutions.

With --debugger I then see the stack as known:

0x00007ffff6ede80c in Shiboken::Module::create(char const*, void*) () from /home/hayen/repos/Py2C/PySide6Using.dist/shiboken6/../libshiboken6.abi3.so.6.7
#0  0x00007ffff6ede80c in Shiboken::Module::create(char const*, void*) () from /home/hayen/repos/Py2C/PySide6Using.dist/shiboken6/../libshiboken6.abi3.so.6.7
#1  0x00007ffff6f113f1 in PyInit_Shiboken () from /home/hayen/repos/Py2C/PySide6Using.dist/shiboken6/Shiboken.so
#2  0x0000555555647069 in callIntoExtensionModule (tstate=0x555555c868c8 <_PyRuntime+459656>, full_name=0x555555bc312d "shiboken6.Shiboken", filename=0x7fffffff8170 "/home/hayen/repos/Py2C/PySide6Using.dist/shiboken6/Shiboken.so", is_package=false) at nuitka/build/static_src/MetaPathBasedLoader.c:838

My immortality corruption checks cannot trigger, since the code doesn't obviously execute on. The line triggering is this:

from PySide6.QtCore import QMetaObject, QObject, Signal, Slot

So, for me, on my self-compiled Python3.12.2 it's not working at all, PySide6 from PyPi package. I looked at compilation reports for the two compilations, and the results were very similar modulo stdlib, and included DLL order, not sure why the order changes for a few places.

from nuitka.

The issue is totally with shiboken

from shiboken6 import Shiboken crashes for me already. The Nuitka PySide6 plugin is not needed for it. Previously, I tried to remove its early import from PySide6 and remove the PySide6 post-load code, all to no avail, which, of course, is not surprising at all.

The new minimal reproducer is from shiboken6 import Shiboken

from nuitka.

The code in question is this one:

https://code.qt.io/cgit/pyside/pyside-setup.git/tree/sources/shiboken6/libshiboken/sbkmodule.cpp#n459

I think I will have to re-activate PySide6 compilation setup, to get a backtrace in shiboken6 there.

from nuitka.

I have release a hotfix for this, turns out the frame f_lasti is now calculated in Python 3.12, but Nuitka didn't set the necessary value for that to work, so often -1 was returned (which is OK), but sometimes large values, and a bytecode scan done by shiboken fails therefore, causing the crash.

from nuitka.

Thank you! 2.3.7 is the first version ever that is working for me on Python 3.12 (technically, 2.3.6 should also work, but its release overlapped with the numpy 2.0 release).

from nuitka.

yeah, setuptools, PySide6 (although that was a Python 3.12 incompletness,, numpy, an unusual amount of package releases have been hitting us lately, and Nuitka-Watch is still not in place as well as it should be. But my CI testing PySide6 standalone e.g. never saw the crash, it's too defined a setup I guess.

from nuitka.