frostming / findpython Goto Github PK

• findpython 0.2.0 & 0.2.1
• pyenv v2.3.5

Actual:

Installing pyenv and trying to find all Python environments will make findpython raise a FileNotFoundError:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/usr/local/lib/python3.9/site-packages/findpython/finder.py", line 103, in find_all
    matched_python = set(self._find_all_python_versions())
  File "/usr/local/lib/python3.9/site-packages/findpython/finder.py", line 136, in _find_all_python_versions
    yield from provider.find_pythons()
  File "/usr/local/lib/python3.9/site-packages/findpython/providers/pyenv.py", line 27, in find_pythons
    for version in self.root.joinpath("versions").iterdir():
  File "/usr/local/lib/python3.9/pathlib.py", line 1160, in iterdir
    for name in self._accessor.listdir(self):
FileNotFoundError: [Errno 2] No such file or directory: '/root/.pyenv/versions'

Can be reproduced building this docker image:

FROM python:3.9.7-slim AS base

RUN apt-get update && apt-get install -y git curl 
ENV PYENV_GIT_TAG v2.3.5
RUN curl https://pyenv.run | bash
RUN pip install findpython==0.2.1
RUN python -c "from findpython.finder import Finder; Finder().find_all()"

Expected:

• findpython skips the clean pyenv installation without raising an exception

Remarks:
Detected in pdm 2.1.5 which uses findpython 0.2.0, doing pdm init: pdm-project/pdm#1468

ref: https://repology.org/project/python:findpython/badges

rtx is an asdf compatible runtime executor written in rust. Executables are installed by default under ~/.local/share/rtx (ref here) though it can be changed. Would you be interested in findpython supporting rtx too? I could volunteer to work on the PR too.

Previous discussion prompted this issue.

The goal is ultimately for pdm to provide an option to be more directed in finding python- from a particular provider, for example pyenv. The reason is the user wants a predictable configuration - for example, all my projects use python installed by one designated provider.

The design I'd like to propose is that

• Every provider defines its configuration name:

  • PyenvProvider.name == "pyenv"
  • PathProvider.name == "path"
  • etc

• Finder class takes a configuration input provider_selection: list[str]

  • Input needed: __init__ parameter or find parameter? Either can work, I think.

• Only listed providers (one or more) are used for finding python

• Providers are used in the given order (if possible)

I hope this is sounds halfway reasonable or can be made reasonable

When I found that in my set of packaged python modules as rpm packages findpython is not updated for switching pdm-pdm-517 to pdm-backend I've started looking what is is using this module and so far found that is is used only by pdm.

Looking on the pdm code I have impression that it should be possible to replace findpython use by standard python modules.
Q: is this modules really needs to be used? 🤔

Hi! I'm currently looking into packaging this software (and pdm) for Arch Linux.

The vendoring of the unreleased and seemingly abandoned pep514tools is suboptimal from a downstream point of view, as it makes it not possible to devendor the dependency (easily).

Would it be possible to use different tooling as direct dependency that replace the aforementioned project or to become a new stable upstream of the tooling so that there can be releases of it (a related issue has been open since 2019)?

Steps to reproduce

• Install pdm by scoop in a Windows device installed with Miniconda.
• Run pdm init.

Actual behavior

> pdm init -v
Creating a pyproject.toml for PDM...
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "C:\Users\Leo\scoop\apps\pdm\current\venv\Scripts\pdm.exe\__main__.py", line 7, in <module>
  File "C:\Users\Leo\scoop\apps\pdm\2.14.0\venv\Lib\site-packages\pdm\core.py", line 359, in main
    return core.main(args or sys.argv[1:])
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Leo\scoop\apps\pdm\2.14.0\venv\Lib\site-packages\pdm\core.py", line 277, in main
    raise cast(Exception, err).with_traceback(traceback) from None
  File "C:\Users\Leo\scoop\apps\pdm\2.14.0\venv\Lib\site-packages\pdm\core.py", line 272, in main
    self.handle(project, options)
  File "C:\Users\Leo\scoop\apps\pdm\2.14.0\venv\Lib\site-packages\pdm\core.py", line 208, in handle
    command.handle(project, options)
  File "C:\Users\Leo\scoop\apps\pdm\2.14.0\venv\Lib\site-packages\pdm\cli\commands\init.py", line 237, in handle
    self.do_init(project, options=options)
  File "C:\Users\Leo\scoop\apps\pdm\2.14.0\venv\Lib\site-packages\pdm\cli\commands\init.py", line 40, in do_init
    self.set_python(project, options.python, hooks)
  File "C:\Users\Leo\scoop\apps\pdm\2.14.0\venv\Lib\site-packages\pdm\cli\commands\init.py", line 214, in set_python
    python_info = UseCommand().do_use(
                  ^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Leo\scoop\apps\pdm\2.14.0\venv\Lib\site-packages\pdm\cli\commands\use.py", line 111, in do_use
    selected_python = self.select_python(
                      ^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Leo\scoop\apps\pdm\2.14.0\venv\Lib\site-packages\pdm\cli\commands\use.py", line 73, in select_python
    found_interpreters = list(dict.fromkeys(project.iter_interpreters(python, filter_func=version_matcher)))
                              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Leo\scoop\apps\pdm\2.14.0\venv\Lib\site-packages\pdm\project\core.py", line 652, in iter_interpreters
    for interpreter in self.find_interpreters(python_spec, search_venv):
  File "C:\Users\Leo\scoop\apps\pdm\2.14.0\venv\Lib\site-packages\pdm\project\core.py", line 722, in find_interpreters
    for entry in finder.find_all(finder_arg, allow_prereleases=True):
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Leo\scoop\apps\pdm\2.14.0\venv\Lib\site-packages\findpython\finder.py", line 122, in find_all
    matched_python = set(self._find_all_python_versions())
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Leo\scoop\apps\pdm\2.14.0\venv\Lib\site-packages\findpython\finder.py", line 171, in _find_all_python_versions
    yield from provider.find_pythons()
  File "C:\Users\Leo\scoop\apps\pdm\2.14.0\venv\Lib\site-packages\findpython\providers\winreg.py", line 41, in find_pythons
    Version(py_version) if py_version else None,
    ^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Leo\scoop\apps\pdm\2.14.0\venv\Lib\site-packages\packaging\version.py", line 200, in __init__
    raise InvalidVersion(f"Invalid version: '{version}'")
packaging.version.InvalidVersion: Invalid version: 'py39_4.12.0'

According above error log, I try to locate at file "C:\Users\Leo\scoop\apps\pdm\2.14.0\venv\Lib\site-packages\findpython\providers\winreg.py and print path of invalid Python.

# ......
if path.exists():
  py_version = getattr(version.info, "version", None)
  print(py_version, path)
  py_ver = self.version_maker(
      path,
      Version(py_version) if py_version else None,
      getattr(version.info, "sys_architecture", SYS_ARCHITECTURE),
      path,
  )
  yield py_ver

After that, run pdm init again. And the log shows Python in Miniconda (F:\Miniconda\python.exe) is invalid.

> pdm init
Creating a pyproject.toml for PDM...
py39_4.12.0 F:\Miniconda\python.exe
[InvalidVersion]: Invalid version: 'py39_4.12.0'

Python in Miniconda (F:\Miniconda\python.exe) is removed temporarily then pdm init works successfully. Obviously findpython cannot recognize Python version of Miniconda, which causes pdm initilization crash.

> pdm init -v
Creating a pyproject.toml for PDM...
3.7.4 D:\Python37\python.exe
3.9.10 D:\Python39\python.exe
Please enter the Python interpreter to use
 0. cpython@3.12 (C:\Users\Leo\scoop\apps\python\current\python.EXE)
 1. cpython@3.12 (C:\Users\Leo\scoop\shims\python3.exe)
 2. cpython@3.9 (D:\Python39\python.exe)
 3. cpython@3.7 (D:\Python37\python37.exe)
 4. cpython@3.7 (D:\Python37\python.exe)

Expected behavior

It seems Miniconda uses custom Python version format which is not supported. I hope findpython can successfully recognize miniconda Python and conflict of packaging.version.InvalidVersion: Invalid version can be solved.

installer module is sensitive about such issues and it fails like below

+ /usr/bin/python3 -sBm installer dist/findpython-0.2.5-py3-none-any.whl --destdir /home/tkloczko/rpmbuild/BUILDROOT/python-findpython-0.2.5-2.fc35.x86_64
Traceback (most recent call last):
  File "/usr/lib64/python3.8/runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib64/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/usr/lib/python3.8/site-packages/installer/__main__.py", line 98, in <module>
    _main(sys.argv[1:], "python -m installer")
  File "/usr/lib/python3.8/site-packages/installer/__main__.py", line 94, in _main
    installer.install(source, destination, {})
  File "/usr/lib/python3.8/site-packages/installer/_core.py", line 109, in install
    record = destination.write_file(
  File "/usr/lib/python3.8/site-packages/installer/destinations.py", line 207, in write_file
    return self.write_to_fs(scheme, path_, stream, is_executable)
  File "/usr/lib/python3.8/site-packages/installer/destinations.py", line 167, in write_to_fs
    raise FileExistsError(message)
FileExistsError: File already exists: /home/tkloczko/rpmbuild/BUILDROOT/python-findpython-0.2.5-2.fc35.x86_64/usr/lib/python3.8/site-packages/findpython/__version__.py

Here is content of the .whl:

[tkloczko@pers-jacek findpython-0.2.5]$ unzip -l dist/findpython-0.2.5-py3-none-any.whl
Archive:  dist/findpython-0.2.5-py3-none-any.whl
  Length      Date    Time    Name
---------  ---------- -----   ----
     4966  01-01-2016 00:00   findpython-0.2.5.dist-info/METADATA
       90  01-01-2016 00:00   findpython-0.2.5.dist-info/WHEEL
       57  01-01-2016 00:00   findpython-0.2.5.dist-info/entry_points.txt
     1075  01-01-2016 00:00   findpython-0.2.5.dist-info/licenses/LICENSE
     1928  01-01-2016 00:00   findpython/__init__.py
     2564  01-01-2016 00:00   findpython/__main__.py
       22  01-01-2016 00:00   findpython/__version__.py      <<<<=== here
       22  01-01-2016 00:00   findpython/__version__.py      <<<<=== here
     6667  01-01-2016 00:00   findpython/finder.py
      442  01-01-2016 00:00   findpython/pep514tools/__init__.py
      254  01-01-2016 00:00   findpython/pep514tools/__main__.py
     6567  01-01-2016 00:00   findpython/pep514tools/_registry.py
     4638  01-01-2016 00:00   findpython/pep514tools/environment.py
      846  01-01-2016 00:00   findpython/providers/__init__.py
     1058  01-01-2016 00:00   findpython/providers/asdf.py
     1823  01-01-2016 00:00   findpython/providers/base.py
      759  01-01-2016 00:00   findpython/providers/macos.py
      662  01-01-2016 00:00   findpython/providers/path.py
     1249  01-01-2016 00:00   findpython/providers/pep514.py
     1072  01-01-2016 00:00   findpython/providers/pyenv.py
      913  01-01-2016 00:00   findpython/providers/rye.py
     6364  01-01-2016 00:00   findpython/python.py
     4374  01-01-2016 00:00   findpython/utils.py
     2010  01-01-2016 00:00   findpython-0.2.5.dist-info/RECORD
---------                     -------
    50422                     24 files

LegacyVersion object returned from packaging module doesn't have the attributes being required from this package. There should be a check for that.

Originally reported: pdm-project/pdm#1849

I'm wondering, would it make sense to support a SpecifierSet, such as >=3.10, and select the best version of Python available based on that? I think it might be interesting for supporting requires-python inside PEP 723, for example. I think you could detect the =>< chars which are not currently used. CC wntrblm/nox#814.

Just a thought, feel free to close if it's not a good idea! :)

With the output from a local build packaging fails to parse the version:

>>> from packaging.version import Version
>>> Version('3.11.0+')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "…/.venv/lib/python3.11/site-packages/packaging/version.py", line 266, in __init__
    raise InvalidVersion(f"Invalid version: '{version}'")
packaging.version.InvalidVersion: Invalid version: '3.11.0+'

% python --version
Python 3.11.0+
% python --version --version
Python 3.11.0+ (heads/3.11:7c9c993945, Nov 10 2022, 10:26:35) [GCC 12.2.0]

This likely causes pdm to not detect this Python from the actually used venv, and also causes an error with pdm config:

…
Project configuration (…/.pdm.toml):
Traceback (most recent call last):
  File "…/project/.venv/bin/pdm", line 7, in <module>
    sys.exit(main())
             ^^^^^^
  File "…/project/.venv/lib/python3.11/site-packages/pdm/core.py", line 258, in main
    return Core().main(args)
           ^^^^^^^^^^^^^^^^^
  File "…/project/.venv/lib/python3.11/site-packages/pdm/core.py", line 204, in main
    check_update(project)
  File "…/project/.venv/lib/python3.11/site-packages/pdm/cli/actions.py", line 868, in check_update
    latest_version = get_latest_version(project)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "…/project/.venv/lib/python3.11/site-packages/pdm/cli/actions.py", line 854, in get_latest_version
    latest_version = get_latest_pdm_version_from_pypi(project)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "…/project/.venv/lib/python3.11/site-packages/pdm/cli/actions.py", line 836, in get_latest_pdm_version_from_pypi
    with environment.get_finder([project.default_source]) as finder:
  File "/home/user/.local/opt/python/lib/python3.11/contextlib.py", line 137, in __enter__
    return next(self.gen)
           ^^^^^^^^^^^^^^
  File "…/project/.venv/lib/python3.11/site-packages/pdm/models/environment.py", line 160, in get_finder
    target_python=self.target_python,
                  ^^^^^^^^^^^^^^^^^^
  File "/home/user/.local/opt/python/lib/python3.11/functools.py", line 1001, in __get__
    val = self.func(instance)
          ^^^^^^^^^^^^^^^^^^^
  File "…/project/.venv/lib/python3.11/site-packages/pdm/models/environment.py", line 116, in target_python
    python_version = self.interpreter.version_tuple
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "…/project/.venv/lib/python3.11/site-packages/pdm/models/python.py", line 64, in version_tuple
    return (self.major, self.minor, self.micro)
            ^^^^^^^^^^
  File "…/project/.venv/lib/python3.11/site-packages/pdm/models/python.py", line 52, in major
    return self._py_ver.major
           ^^^^^^^^^^^^^^^^^^
  File "…/project/.venv/lib/python3.11/site-packages/findpython/python.py", line 71, in major
    return self.version.major
           ^^^^^^^^^^^^
  File "…/project/.venv/lib/python3.11/site-packages/findpython/python.py", line 65, in version
    self._version = self._get_version()
                    ^^^^^^^^^^^^^^^^^^^
  File "…/project/.venv/lib/python3.11/site-packages/findpython/python.py", line 165, in _get_version
    return Version(version)
           ^^^^^^^^^^^^^^^^
  File "…/project/.venv/lib/python3.11/site-packages/packaging/version.py", line 266, in __init__
    raise InvalidVersion(f"Invalid version: '{version}'")
packaging.version.InvalidVersion: Invalid version: '3.11.0+'

I am not sure this is a bug with packaging / if it should handle it better, but certainly causes issues here.