Custom package namespace about maturin HOT 10 CLOSED

There seems to be only very few information online about ext_package. Could you describe a bit more in detail how this argument would change the structure of the wheel or unpacked package?

from maturin.

@konstin of course! It isn't well documented, you are right. An example is here: python/typed_ast#85. Essentially it moves the _ast27 and _ast3 modules into a typed_ast package. So instead of import _ast3 one must from typed_ast import _ast3, which means your extension modules end up namespaced as they probably should be.

from maturin.

I believe this shouldn't be too hard to accomplish similar to how distutils does it, which is to just create an additional directory between the files to be installed and the toplevel. I can work on a PR.

from maturin.

I've built typed_ast and got the following wheels:

Without ext_package:

├── _ast27.cpython-36m-x86_64-linux-gnu.so
├── _ast3.cpython-36m-x86_64-linux-gnu.so
├── typed_ast
│   ├── _ast27.cpython-36m-x86_64-linux-gnu.so
│   ├── ast27.py
│   ├── _ast3.cpython-36m-x86_64-linux-gnu.so
│   ├── ast3.py
│   ├── conversions.py
│   ├── __init__.py
│   └── tests
│       └── test_basics.py
└── typed_ast-1.3.2.dev0.dist-info
    ├── LICENSE
    ├── METADATA
    ├── RECORD
    ├── top_level.txt
    └── WHEEL

With ext_package:

├── typed_ast
│   ├── _ast27.cpython-36m-x86_64-linux-gnu.so
│   ├── ast27.py
│   ├── _ast3.cpython-36m-x86_64-linux-gnu.so
│   ├── ast3.py
│   ├── conversions.py
│   ├── __init__.py
│   └── tests
│       └── test_basics.py
└── typed_ast-1.3.2.dev0.dist-info
    ├── LICENSE
    ├── METADATA
    ├── RECORD
    ├── top_level.txt
    └── WHEEL

So what happens is that python_ast without ext_package installs two top-level modules that don't match the package name, which it shouldn't.

This doesn't happen with pyo3-pack, because the shared libraries module name is equal to the distribution name. What we could do however is to put the shared libraries into a folder of the same module name and reexport everything through __init__.py, so we get more consistency with other packages and the pyo3-pack's cffi wheels.

from maturin.

So what happens is that python_ast without ext_package installs two top-level modules that don't match the package name, which it shouldn't.

Sorry I'm not sure I understand what you are saying. Are you saying it shouldn't install the extension modules into the global namespace? I agree, but that is unfortunately the default (and I think there is a reason for this? Otherwise where should they go if there are multiple packages...).

What we could do however is to put the shared libraries into a folder of the same module name and reexport everything through __init__.py, so we get more consistency with other packages and the pyo3-pack's cffi wheels.

Hm, I'm not sure this allows me to accomplish what I want, which is native extensions alongside Python code in a single package.

from maturin.

Sorry I'm not sure I understand what you are saying. Are you saying it shouldn't install the extension modules into the global namespace? I agree, but that is unfortunately the default (and I think there is a reason for this? Otherwise where should they go if there are multiple packages...).

What I meant is that a package foo should only be installing a module called foo in the top-level namespace not also a module called bar, i.e. after pip install foo I shouldn't be able to import bar. This is enforced in other languages, but unfortunately changing would break backwards compatibility with many wheels.

Hm, I'm not sure this allows me to accomplish what I want, which is native extensions alongside Python code in a single package.

Oh, that's totally different story and than wheel layout switches like ext_package. Unfortunately mixing python and rust in a single package is currently not supported. I've written #88 with a proposal how that could work. In the meantime setuptools-rust can do that.

from maturin.

What I meant is that a package foo should only be installing a module called foo in the top-level namespace not also a module called bar, i.e. after pip install foo I shouldn't be able to import bar. This is enforced in other languages, but unfortunately changing would break backwards compatibility with many wheels.

Ah ok, this is what I thought you mean, but I was unsure.

Oh, that's totally different story and than wheel layout switches like ext_package. Unfortunately mixing python and rust in a single package is currently not supported. I've written #88 with a proposal how that could work. In the meantime setuptools-rust can do that.

Yeah, #88 seems like a plausible solution to this, I just want to make sure that with your example, if the lib name is say "test" then one would import my_proj.test, and not be able to import test? If so we can close this in favor of #88.

from maturin.

@konstin I think this issue should remain open, as #88 seems to have been differed/abandoned. I would still like to be able to namespace the final binary only modifying a Cargo.toml. The current solution seems to be to use setuptools-rust, which I'd like to avoid if possible.

from maturin.

I've implemented #88, which will be part of the next release, so there shouldn't be any need for namespacing anymore.

from maturin.

Ah, apologies, I missed the linked PR, thanks for implementing that!

from maturin.