I've come up with a solution proposal to a known problem I'm facing, and I would like to know your thoughts on it and if it's possible to implement it on Pyapp
I've tried to be as detailed as possible, and I'm open to any questions or suggestions, thanks!
Context, Research and Development old Comment (Expand)
Context
I have a pretty standard Monorepo structure that provides a main package for all the other Projects but don't know them
A project refers to the monorepo using path dependencies, and they even might refer to other projects as well
Note: I'm doing this to separate dependencies e.g. not all projects need pytorch
It all works nicely under development mode.. until I want to build a release of any project
In the past, I have implemented convoluted solutions using Pyinstaller or Nuitka which ended up working to a certain extent but wasn't ideal (long story), so I decided to give Pyapp a try
Problem
As I saw somewhere, Python wheels aren't standardized for path dependenciesyet?, so whenever building a pyproject.toml, the wheel won't be installable on other machines as the builder's local path is hardcoded
I don't really want to upload the code to PyPI as it is very specific to my use case, much like other monorepo opinions; even if that was the use case, Poetry can't have a versioned dependency on the main section and a path dependency on the dev section simultaneously of the same package
Ultimately, this yields either spaghetti solutions or the lack of it
What I have tried
I've spent two days of intensive digging through the documentation and issues everywhere, trying many build backends such as Poetry, Hatch, PDM and proposed Poetry plugins or solutions, but ultimately I couldn't get it to work. Raw Pyapp was the closest I gotI know it's used in Hatch!
Attempt 1: Source distribution
I honestly don't remember much of what I tried yesterday, but I can say this wasn't ideal as including the packages as sdist isn't "safe" and annoying to define the glob imports, also the monorepo package isn't on the subpath of the projects, and Poetry fails
Attempt 2: Custom distribution
Long story short, I zipped the Poetry's Virtual Environment and set the proper relative paths on Pyapp variables for the executables, and used the full isolated mode, skip install.
It fails as the Python included there is a symlink to system Python. Setting poetry.virtualenvs.options.always-copy to true didn't do it as well(Consider this as a bug report? To embed some proper Python distribution on top of a local one?)
I'm not a fan of this solution as yours fetching and installation of the Python distribution feels more reliable and arguably universal
Attempt 3: Hatch
I ported the pyprojects.toml to Hatch syntax and force-included the main package Broken under ../../Broken to the wheel. The embedding I'm using is PYAPP_PROJECT_PATH as the built wheel
This failed as I didn't "inherit" the dependencies of the main package, the Virtual Environment contained properly ShaderFlow and Broken package, but not the dependencies of Broken (the monorepo root's package)
This solution feels non ideal as I had to unset safety flags on Hatch, like the allow direct references and, well, including some other package on the wheel
Proposed solution
After all the digging, I think this could be solved by the following:
- Have the path dependencies as
dev-dependencies on the pyproject.toml of the project:
[tool.poetry.dependencies]
python = ">=3.10,<3.13"
moderngl = "^5.8.2"
# ...
[tool.poetry.dev-dependencies]
broken = {path="../../", develop=true}Building a wheel for this project won't include the broken package, but it's ok
- Find all path dependencies and build their wheel, recursively
This isn't something you can implement on Pyapp, but a process users would need to define on their own
A pseudo code / implementation would be something like this (I didn't run nor test the logic):
from pathlib import Path
from dotmap import DotMap
import toml
def build_projects(path: Path, found: Set[Path]=None):
path = Path(path).resolve()
# Initialize empty set
found = found or set()
# Skip if already found
if path in found:
return
# Skip if no pyproject.toml exists
if not (path/"pyproject.toml").exists():
return
# Load pyproject.toml dictionary
pyproject = DotMap(toml.reads((path/"pyproject.toml").read_text()))
# Iterate and find all path dependencies
for name, dependency in pyproject.tool.poetry["dev-dependencies"].items():
# Find only path= dictionaries
if isinstance(data, str):
continue
if not dependency.path:
continue
# Dependency is a path
dependency = Path(data.path).resolve()
found.add(dependency)
# Build the wheel
with pushd(dependency):
subprocess.run(["poetry", "build", "--format", "wheel"])
# Recursively find wheels
wheels(dependency, found)
return found
# Build all wheels
projects = build_projects(Path.cwd())
# We can now get the wheels from the projects
wheels = [next(project.glob("dist/*.whl")) for project in projects]Why we need all of this? For the next step and the proposed solution
By installing the wheel of the main project and all other Path Dependencies wheels, we:
![dependabot[bot] avatar](https://avatars.githubusercontent.com/in/29110?v=4 "dependabot[bot]")
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent