landscapeio / requirements-detector Goto Github PK

requirements-detector assumes version will always exist for each dependency in pyproject.toml, however this is not the case for git, path or url dependencies:

https://python-poetry.org/docs/dependency-specification/#git-dependencies

For example, git dependencies in pyproject.toml can be specified without any version property (poetry assumes latest commit on main), or using branch, rev or tag, but not version.

mytestdependency = {git = "ssh://[email protected]/MyOrg/MyRepo.git",rev = "1.2.3"}

In this case, requirements-detector fails with a KeyError.

  File "/repo/.venv/lib/python3.8/site-packages/requirements_detector/detect.py", line 68, in find_requirements
    requirements = from_pyproject_toml(poetry_toml)
  File "/repo/.venv/lib/python3.8/site-packages/requirements_detector/detect.py", line 116, in from_pyproject_toml
    spec = spec["version"]
KeyError: 'version'

The expected output would be to list mytestdependency without a version, or to attempt to parse the rev as the version

Relates to landscapeio/prospector#556

I've started seeing requirements-detector crash when I run prospector, though it was working fine before:

tox -e lint
lint create: /home/seanh/Projects/lms/.tox/lint
lint installdeps: prospector, mock, pytest, factory-boy, -rrequirements.txt
lint installed: alembic==0.9.6,astroid==2.0,atomicwrites==1.1.5,attrs==18.1.0,certifi==2017.11.5,chardet==3.0.4,dodgy==0.1.9,factory-boy==2.11.1,Faker==0.8.17,flake8==3.5.0,flake8-polyfill==1.0.2,gunicorn==19.7.1,httplib2==0.10.3,hupper==1.0,idna==2.6,isort==4.3.4,Jinja2==2.10,lazy-object-proxy==1.3.1,Mako==1.0.7,MarkupSafe==1.0,mccabe==0.6.1,mock==2.0.0,more-itertools==4.2.0,newrelic==2.98.0.81,oauth2==1.9.0.post1,oauthlib==2.0.6,PasteDeploy==1.5.2,pbr==4.1.0,pep8-naming==0.7.0,plaster==1.0,plaster-pastedeploy==0.4.2,pluggy==0.6.0,prospector==0.12.11,psycopg2==2.7.3.2,py==1.5.4,pycodestyle==2.3.1,pycrypto==2.4,pydocstyle==2.1.1,pyflakes==1.6.0,PyJWT==1.5.3,pylint==1.9.2,pylint-common==0.2.5,pylint-plugin-utils==0.4,PyLTI==0.5.1,pyramid==1.9.1,pyramid-jinja2==2.7,pyramid-services==1.1,pyramid-tm==2.2,pytest==3.6.3,python-dateutil==2.6.1,python-editor==1.0.3,PyYAML==3.13,raven==6.4.0,repoze.lru==0.7,requests==2.18.4,requests-oauthlib==0.8.0,requirements-detector==0.5.2,setoptconf==0.2.0,six==1.11.0,snowballstemmer==1.2.1,SQLAlchemy==1.1.15,text-unidecode==1.2,transaction==2.1.2,translationstring==1.3,typed-ast==1.1.0,urllib3==1.22,venusian==1.1.0,WebOb==1.7.4,wrapt==1.10.11,zope.deprecation==4.3.0,zope.interface==4.4.3,zope.sqlalchemy==0.7.7
lint runtests: PYTHONHASHSEED='2814062973'
lint runtests: commands[0] | prospector
Traceback (most recent call last):
  File "/home/seanh/Projects/lms/.tox/lint/bin/prospector", line 7, in 
    from prospector.run import main
  File "/home/seanh/Projects/lms/.tox/lint/lib/python3.6/site-packages/prospector/run.py", line 9, in 
    from prospector.config import configuration as cfg
  File "/home/seanh/Projects/lms/.tox/lint/lib/python3.6/site-packages/prospector/config/__init__.py", line 7, in 
    from prospector.autodetect import autodetect_libraries
  File "/home/seanh/Projects/lms/.tox/lint/lib/python3.6/site-packages/prospector/autodetect.py", line 4, in 
    from requirements_detector import find_requirements
  File "/home/seanh/Projects/lms/.tox/lint/lib/python3.6/site-packages/requirements_detector/__init__.py", line 1, in 
    from requirements_detector.detect import find_requirements
  File "/home/seanh/Projects/lms/.tox/lint/lib/python3.6/site-packages/requirements_detector/detect.py", line 5, in 
    from astroid import MANAGER, CallFunc, Name, Assign, Keyword, List, Tuple, Const, AssName
ImportError: cannot import name 'CallFunc'

I was trying to use this tool on a setup.py file with valid syntax.

# -*- coding: UTF-8 -*-
from distutils.core import setup

if foo:
    # just for a test with indentation
    bar()

setup(
    name=u'prospector-test-4-üéø',
    version='0.0.1',
    install_requires=[
        'Django==1.5.0',
        'django-gubbins==1.1.2'
    ]
)

Unfortunately this hack destroys the line indentation, and therefore produces invalid python code. This leads to an exception in AstroidBuilder (being catched here).
Since parsing setup.py yielded an error, it falls back to parse requirements.txt - which in my case is empty.

Proposed solution:
Don't remove leading whitespace during _load_file_contents. I would have tried working on a fix myself, but I didn't quite understand the unicode hack involved here - I didn't want to break anything else...
Or is it enough to simply remove the .strip() from here?

Hi,

I just got this exception:

  File "/home/mdk/Downloads/pystyle/pystyle/pystyle/pystyle_update.py", line 175, in infer_requirements
    requirements_detector.find_requirements(path)]
  File "/home/mdk/.venvs/pystyle/lib/python3.5/site-packages/requirements_detector/detect.py", line 89, in find_requirements
    requirements.sort()
  File "/home/mdk/.venvs/pystyle/lib/python3.5/site-packages/requirements_detector/requirement.py", line 101, in __gt__
    return (self.name or "") > (other.name or "")
AttributeError: 'NoneType' object has no attribute 'name'

I'll try to find with which repository to tell you more.

Bests

setup.py files can also specify dependecies in a requires parameter to setup()

As Python 3.6 was released more than 17 months ago, it should be supported.

Hello,
could you please consider to add tests/ into pypi tarball?

A setup.py which declares an encoding (PEP-263) is not handled correctly at the moment. ASCII is assumed and this breaks when non-ASCII characters are used.

Found an issue when trying to update prospector to be compatible with black 23.1.0:

landscapeio/prospector#588 (closed duplicate of the following)
landscapeio/prospector#579

It looks like requirements-detector depends on packaging=^21.0, but black 23.1.0 requires packaging >=22.0. I would open a PR but it appears that poetry is not happy about this project continuing to support 3.6 when moving to packaging >=22.0.

$ poetry add "packaging=^22.0"

Updating dependencies
Resolving dependencies... (0.0s)

The current project's Python requirement (>=3.6.2,<4.0) is not compatible with some of the required packages Python requirement:
  - packaging requires Python >=3.7, so it will not be satisfied for Python >=3.6.2,<3.7

Because no versions of packaging match >22.0,<23.0
 and packaging (22.0) requires Python >=3.7, packaging is forbidden.
So, because requirements-detector depends on packaging (^22.0), version solving failed.

  • Check your dependencies Python requirement: The Python requirement can be specified via the `python` or `markers` properties

    For packaging, a possible solution would be to set the `python` property to ">=3.7,<4.0"

    https://python-poetry.org/docs/dependency-specification/#python-restricted-dependencies,
    https://python-poetry.org/docs/dependency-specification/#using-environment-markers

The obvious solution is to drop python 3.6 support, but I'm not sure if there's a desire to retain python 3.6 support at this time.

You should search for the specified build tool first and fall back to legacy stuff then:

1. Parse PEP 621 static metadata
2. If no PEP 621 metadata exists, read pyproject.toml’s build-system.build-backend (PEP 517) and parse tool specific metadata
3. If no standard metadata exists at all, fall back to setup.py or requirements.txt

Hey guys, I am in the process of updating support for python311 for our package and since we use prospector, and prospector depends on requirements-detector, I went about and started updating the conda packages for both prospector and your package; the one problem that I am facing while updating the conda package for requirements-detector is that you depend on, and use poetry-semver which doesn't have a conda package candidate, nor does it have a feedstock, nor is it developed any more, it looks like it's now archived. Would you mind if I opened a PR to replace it and its functionality with semver instead? Many thanks in advance, cheers 🍺

The Debian packaging pulls the Github release tarballs, so getting the latest version of requirements-detector in to Debian (and therefore Ubuntu) would be a lot easier with a tagged release. :-)

Thanks!

As Python 3.7 was released last month, it should be supported.

cf. #18 (Python 3.6)

If the value of install_requires comes from another variable, and that variable is updated later in the code, the requirements are not fully detected, eg:

requires = []
requires.add('Django==1.5.5')

setup(
   install_requires=requires
   ...
)

Example: https://github.com/sigmavirus24/github3.py/blob/a728fffa49e0da2a404ee2c8651b2f0eb3579cb7/setup.py

The license file says MIT. However, setup.py says GPL 2. FWIW it seems astroid is the old requirement and seems to be LGPL for the version specified (unless we are talking 0.x versions). So it doesn't seem to constrain this at all. Given all of this, what is the license of requirements-detector?

pyproject.toml is the new, but only standard way to build packages. Wheels offer a standard way to specify them (using a METADATA file)

So technically the only fool proof way to do this is to build a wheel using pyproject.toml and check that.

But alternatively we can just handle all known users:

if config['build-backend'] == 'flit.buildapi':
    flit_meta = config['tool']['flit']['metadata']
    requirements = flit_meta['requires'] + [
        dep
        for extra, deps in flit_meta['requires-extra'].items()
        for dep in deps
    ]
elif config['build-backend'] == 'poetry.masonry.api':
    ... ['tool']['poetry']['dependencies']
...

Setup.cfg is a proper place to setup whole thing for setuptools

example (taken from setuptools page)

# setup.py

from setuptools import setup

setup()  # All details are in setup.cfg

#setup.cfg

[metadata]
name = my_package
version = attr: my_package.VERSION
author = Author Name
author_email [email protected]
description = My package description
long_description = file: README.rst, CHANGELOG.rst, LICENSE.rst
keywords = one, two
license = BSD-3-Clause
classifiers =
    Framework :: Django
    Programming Language :: Python :: 3

[options]
zip_safe = False
include_package_data = True
packages = find:
python_requires = >=3.7
install_requires =
    requests
    importlib-metadata; python_version<"3.8"

[options.package_data]
* = *.txt, *.rst
hello = *.msg

[options.entry_points]
console_scripts =
    executable-name = my_package.module:function

[options.extras_require]
pdf = ReportLab>=1.2; RXP
rest = docutils>=0.3; pack ==1.1, ==1.3

[options.packages.find]
exclude =
    examples*
    tools*
    docs*
    my_package.tests*

Hi all,
I think I found a bug in the latest release(1.0.2), on 0.7 the following code would run without any problem:

import os
from requirements_detector import find_requirements
find_requirements(os.getcwd())

But on 1.0.2 we get the following error:

TypeError: unsupported operand type(s) for /: 'str' and 'str'

It seems the function find_requirements does not accept a string anymore, only a Path object from pathlib. Can we make it compatible with 0.7.0 and accept both a Path object and a string?

Something like this should work:

def find_requirements(path: Union[Path, str]) -> List[DetectedRequirement]:
    requirements = []
    if isinstance(path, str):
        path = Path(path)
    setup_py = path / "setup.py"
    if path.exists() and path.is_file():
        try:
            requirements = from_setup_py(setup_py)
            requirements.sort()
            return requirements

Thanks.

I tried to detect requirements in setup.py file at https://github.com/landscapeio/requirements-detector/blob/master/setup.py, but I'm getting the error
Unable to find requirements at /***/Downloads/requirements-detector.

if the setup.py is in this format
setup( install_requires=install_requires,)
it isn't working, it works only with this format
setup(install_requires=['requests', 'CAGE'],)

astroid published a major update with 3.0.0 (https://github.com/pylint-dev/astroid/releases/tag/v3.0.0) awhile back.

requirements-detector requires poetry-semver which has been archived recently: https://github.com/python-poetry/semver.

Maybe the more active https://github.com/python-semver/python-semver/ could be a good alternative?

Due to missing ignore of pre-release packages in setuptools pypa/setuptools#855 and the existence of astroid pre-released 2.0.0.dev0, we should encounter the following failure when some packages (e.g., pylint) are installed together.

setup.py

setuptools.setup(
    install_requires=[
        'pylint==1.9.1',
        'requirements-detector==0.5.2'
    ],
)

Results with failures

See https://readthedocs.org/projects/prospector/builds/7247785/ (https://github.com/PyCQA/prospector project's docs build).

Solutions

To prevent this problem, we can add a workaround specifying astroid<2.0 to this requirements-detector project's requirements in its setup.py.