Nuitka extension file loader needs to load extension modules in "create_module" not only "exec_module" was (Pymongo lazy import) about nuitka HOT 17 OPEN

@blink1073 created a workaround: mongodb/mongo-python-driver@ff950f0
The workaround solves my immediate issue, so there is no rush on the Nuitka fix. I do think it will be good to have it fixed in Nuitka in case this happens again with a different 3rd party code package that isn't as responsive.
Thanks @kayhayen @KRRT7 and @blink1073 for working this so quickly.

from nuitka.

I have the nuitka-crash-report.xml file, but github gives me an error when I try to attach it.
Github error when I try to attach the xml file:
We don’t support that file type.
Try again with GIF, JPEG, JPG, MOV, MP4, PNG, SVG, WEBM, CPUPROFILE, CSV, DMP, DOCX, FODG, FODP, FODS, FODT, GZ, JSON, JSONC, LOG, MD, ODF, ODG, ODP, ODS, ODT, PATCH, PDF, PPTX, TGZ, TXT, XLS, XLSX or ZIP.

from nuitka.

pymongo developer here, as best as I can tell Nuitka does not fully support lazy imports.

Example code:

import importlib.util
import sys
def lazy_import(name):
    spec = importlib.util.find_spec(name)
    loader = importlib.util.LazyLoader(spec.loader)
    spec.loader = loader
    module = importlib.util.module_from_spec(spec)
    sys.modules[name] = module
    loader.exec_module(module)
    return module
lazy_zlib = lazy_import("zlib")
print(lazy_zlib.crc32)

$ python3 hello.py
<built-in function crc32>
$ python3 -m nuitka --onefile hello.py
$ ./hello.bin
Traceback (most recent call last):
  File "/tmp/onefile_6892_1714611657_437491/hello.py", line 12, in <module>
    print(lazy_zlib.crc32)
  File "/tmp/onefile_6892_1714611657_437491/importlib/util.py", line 250, in __getattribute__
ValueError: module object for 'zlib' substituted in sys.modules during a lazy load

Is there a way we can detect that we are running under Nuitka and eagerly import instead?

from nuitka.

Another aspect of the problem is the handling of ModuleNotFoundError.

$ cat hello.py
e = ModuleNotFoundError(name="foo")
print(e.name)
$ python3 hello.py
foo
$ python3 -m nuitka --onefile hello.py
$ ./hello.bin
Traceback (most recent call last):
  File "/tmp/onefile_7078_1714612025_862116/hello.py", line 1, in <module>
    e = ModuleNotFoundError(name="foo")
TypeError: exceptions.ModuleNotFoundError does not take keyword arguments

from nuitka.

@blink1073 if you'd like to add in nuitka support from within pymongo, then it'd look like something like this

    if spec is None:
        cond = "__compiled__" in globals()
        if cond:
            sys.exit(f"Nuitka: For this environment to load, need to use this as an option to compile with --include-module={name}` with Nuitka to include it")
        raise ModuleNotFoundError(name=name)

from nuitka.

@kayhayen
the signature says

class ModuleNotFoundError(
    *args: object,
    name: str | None = ...,
    path: str | None = ...
)

so it needs to accept those.

from nuitka.

There is special handling for ImportError and its arg only, I don't think I noticed ModuleNotFoundError yet.

I also never saw importlib.util.LazyLoader so far, I will check it out. Seems also old, since you saw that on 3.9, given the function you need to use there, I wonder

I don't want there to be a need to change the code for Nuitka of course. Compiled modules interact themselves with sys.modules and put themselves there, somehow LazyLoader seems to dislike that, but we will find ways to cooperate I guess.

from nuitka.

I will start to add support for the ModuleNotFoundError keyword arguments. Since we got a good precedent with ImportError surely that cannot be too hard, although it's besides the point, but visibility of that issue will go away once we fix the other bug.

from nuitka.

Thanks all, here is what I have for now as a workaround:

def lazy_import(name: str) -> ModuleType:
    """Lazily import a module by name

    From https://docs.python.org/3/library/importlib.html#implementing-lazy-imports
    """
    # Eagerly import on Nuitka
    if "__compiled__" in globals():
        return importlib.import_module(name)

    try:
        spec = importlib.util.find_spec(name)
    except ValueError:
       raise ImportError(name) from None  # use ImportError instead of ModuleNotFoundError
    if spec is None:
        raise ImportError(name)
    ...

@sullivan50909, I'll update my PR accordingly

from nuitka.

Thanks for your report, this is worked on the factory branch, which is a development version under rapid development. You can try it out by going here: https://nuitka.net/doc/factory.html

Feedback on whether this is working is very welcome. Please do not share plans to do it; only confirm or deny that it is working.

from nuitka.

So, it seems lazy loader for the example code is working as expected. As for lazy loading, I would normally recommend to use the lazy package, that works out of the box. If you have a minimal reproducer for this in pymongo, that would be great, the example you give is not doing it for me, maybe because zlib is built-in or not something for Fedora Python vs. my Ubuntu Python, I have to check.

from nuitka.

Yes, being built-in, means Nuitka doesn't step it, trying with my badly self-compiled 3.9 now, that probably doesn't do it, since it's an extension module there. Extension module loading will also be different from compiled module loading.

from nuitka.

Yeah, I managed to reproduce it, checking the source code now, I came across this:

    def __getattribute__(self, attr):
        """Trigger the load of the module and return the attribute."""
        # All module metadata must be garnered from __spec__ in order to avoid
        # using mutated values.
        # Stop triggering this method.
        self.__class__ = types.ModuleType

Seriously?

from nuitka.

Thanks all, here is what I have for now as a workaround:

def lazy_import(name: str) -> ModuleType:
    """Lazily import a module by name

    From https://docs.python.org/3/library/importlib.html#implementing-lazy-imports
    """
    # Eagerly import on Nuitka
    if "__compiled__" in globals():
        return importlib.import_module(name)

    try:
        spec = importlib.util.find_spec(name)
    except ValueError:
       raise ImportError(name) from None  # use ImportError instead of ModuleNotFoundError
    if spec is None:
        raise ImportError(name)
    ...

@sullivan50909, I'll update my PR accordingly

if you keep the snippet i have, it'll be true lazy loading and will also help with exe's debloat, snappy i found to be a large dependency which is why i did it that way.

from nuitka.

So, it seems there is an incompatibility between the meta path-based loader of Nuitka and the one of Python3.9 or higher. Nuitka loads the extension module and executes a "def" if found (not the case for zlib) during exec_module. That changes the module in sys.modules then, as zlib does that during its execution. In the ExtensionFileLoader at that point, nothing is happening anymore, there is no "def" to execute anymore, this has happened when create_module is called, and there it re-uses the value in sys.modules it seems, but it doesn't do that for Nuitka somehow in exec_module, so I don't know yet, what to do yet.

Somehow, imp.create_dynamic must be capable of enforcing load of the module within the existing value.

from nuitka.

So, this can be distilled down to this:

import importlib.util
import sys
spec = importlib.util.find_spec("zlib")
module = importlib.util.module_from_spec(spec)
sys.modules["zlib"].crc32(b"a")

Basically, module_from_spec, which only calls create_module and, depending on the extension module, may or may not need exec_module to be called, should already populate sys.modules, but in Nuitka, it never does that until exec_module is called.

from nuitka.

This is an invasive change, but not too bad, I will try and tackle it for the next round of hotfixes, 2.2.1 has to go out now, 2.2.2 will be a week later likely.

from nuitka.