suutari / prequ Goto Github PK

View Code? Open in Web Editor NEW

This project forked from jazzband/pip-tools

16.0 16.0 5.0 2.64 MB

Tools for Python requirement handling

Home Page: https://pypi.python.org/pypi/prequ

License: BSD 2-Clause "Simplified" License

Python 99.91% Shell 0.09%

python python-3 requirements-management tool wheel

prequ's People

Contributors

Stargazers

Watchers

Forkers

i2biz ehaivala jssel tulimaki wgarlock

prequ's Issues

Sync doesn't use find-links from the txt files

When requirements.txt has --find-links some_directory and a package some-package, which is only available from that directory, is added/updated, then prequ sync fails to an error that it can't find some-package.

Invalid version syntax seems to be ignored

Had a case where a package version was set as >== instead of >= by accident, and prequ didn't raise any messages about this, nor update the requirement files for this package.

sync: Parse errors in requirements.txt files are not reported correctly

The sync command should report parse errors of the requirement files rather than just fail with a traceback.

E.g. when doing prequ sync -n *.txt in a directory where there is a non-requirement txt file too, the output will look like this:

$ prequ sync -n *.txt
Traceback (most recent call last):
  File "venv/bin/prequ", line 11, in <module>
    sys.exit(main())
  File "venv/lib/python3.4/site-packages/click/core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "venv/lib/python3.4/site-packages/click/core.py", line 697, in main
    rv = self.invoke(ctx)
  File "venv/lib/python3.4/site-packages/click/core.py", line 1066, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "venv/lib/python3.4/site-packages/click/core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "venv/lib/python3.4/site-packages/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "venv/lib/python3.4/site-packages/prequ/scripts/sync.py", line 71, in cli
    to_install, to_uninstall = sync.diff(requirements, installed_dists)
  File "venv/lib/python3.4/site-packages/prequ/sync.py", line 101, in diff
    requirements_lut = {r.link if r.editable else key_from_req(r.req): r for r in compiled_requirements}
  File "venv/lib/python3.4/site-packages/prequ/sync.py", line 101, in <dictcomp>
    requirements_lut = {r.link if r.editable else key_from_req(r.req): r for r in compiled_requirements}
  File "venv/lib/python3.4/site-packages/prequ/utils.py", line 55, in key_from_req
    key = req.name
AttributeError: 'NoneType' object has no attribute 'name'

It should look like this (e.g.):

$ prequ sync -n *.txt
Cannot parse as a requirement file: README.txt

Use setup.cfg as default configuration file

It was a mistake to create a new file format and naming practice for the pre-requirements file. Python projects usually have a setup.cfg and it's customary to have configuration of various Python tools in it. It would be better to have a [prequ] section in setup.cfg to configure Prequ rather than using the requirements.pre.

Implement parsing configuration from [prequ] section in setup.cfg and prefer it over the requirements.pre file. Also drop the pyyaml dependency, since it is not needed anymore.

sync: Hashes are not checked when installing

When installing new packages with prequ sync and there is some hashes specified in the generated requirements.txt, the hashes should be checked before installing, but that's currently not done.

See jazzband#619 for more details.

The "#egg" part in source requirements isn't always handled correctly

When adding an URL entry with #egg part to source requirements the resulting generated requirements will contain two eggs.

E.g. following requirements.in (l:

# Following is really just a one line, but is wrapped here for readability
git+git://github.com/City-of-Helsinki/django-oidc-provider.git
@v0.5.0.post1+uc
#egg=django-oidc-provider==0.5.0.post1+uc

results in

# Following is really just a one line, but is wrapped here for readability
git+git://github.com/City-of-Helsinki/django-oidc-provider.git
@v0.5.0.post1+uc
#egg=django-oidc-provider==0.5.0.post1+uc
#egg=django-oidc-provider==0.5.0.post1+uc

pip 19+ seems to cause error while compiling requirements

AttributeError: 'Values' object has no attribute 'process_dependency_links'

"prequ upgrade" command

Implement a command for upgrading the requirements to newer versions, either all or just specified package(s).

"prequ add"

Add a package to setup.cfg.

Would work like yarn in the sense that when prequ add is being ran the package is added to setup.cfg/requirements files and be installed. By default the packages should be added under the requirements section of the settings, but there should be a flag that can be set for selecting a specific requirements label.

Example:

$ cat setup.cfg
[prequ]
requirements =
    Django

$ prequ add pytest-django

$ cat setup.cfg
[prequ]
requirements =
    Django
    pytest-django

Arguments:
-l --label : Specifies the requirement file label.
Example: prequ add -l dev pytest-django would add pytest-django under requirements-dev

-a --only-add : Only adds the package to setup.cfg and updates the requirements files

Edit: Added arguments and better explanation of how the adding should work :)

Support for editable packages in relative directory

When there is a local directory dir-x which contains a Python package (i.e. has setup.py), Prequ should be able to have a source line -e ./dir-x which will be relative in the compiled requirements.txt too.

Currently having -e ./dir-x in the source requirements will generate an entry as -e file:///absolute/path/to/dir-x, which is usually not very useful.

Supporting -e . is a noteworthy special case of this.

Add pip 20.0.2 support

Latest pip causes this error:

File "/Users/panu/shuup-dev/owl/venv/lib/python3.7/site-packages/prequ/_pip_compat.py", line 35, in
from pip._internal.download import is_file_url, path_to_url, url_to_path
ModuleNotFoundError: No module named 'pip._internal.download'

"prequ sync -a"

Implement an "-a" or "--all" option to "prequ sync". The option would sync all known requirements*.txt files.

The lack of this feature can be worked around by doing prequ sync *.txt, but that might cause issues like in issue #17.

TypeError: init() missing 1 required positional argument: 'req_tracker'

Running prequ update yields the following result:

root@5b55f2c59a74:/app# prequ update
*** Compiling requirements.txt
Traceback (most recent call last):
  File "/usr/local/bin/prequ", line 11, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 697, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 1066, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/click/decorators.py", line 17, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/prequ/scripts/update.py", line 17, in main
    ctx.invoke(compile.main, verbose=verbose, silent=silent)
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/click/decorators.py", line 17, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/prequ/scripts/compile.py", line 26, in main
    compile(ctx, verbose, silent, check)
  File "/usr/local/lib/python3.7/site-packages/prequ/scripts/compile.py", line 48, in compile
    do_one_file(ctx, conf, label, compile_opts)
  File "/usr/local/lib/python3.7/site-packages/prequ/scripts/compile.py", line 63, in do_one_file
    **compile_opts)
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/prequ/scripts/compile_in.py", line 174, in cli
    results = resolver.resolve(max_rounds=max_rounds)
  File "/usr/local/lib/python3.7/site-packages/prequ/resolver.py", line 120, in resolve
    has_changed, best_matches = self._resolve_one_round()
  File "/usr/local/lib/python3.7/site-packages/prequ/resolver.py", line 218, in _resolve_one_round
    for dep in self._iter_dependencies(best_match):
  File "/usr/local/lib/python3.7/site-packages/prequ/resolver.py", line 307, in _iter_dependencies
    dependencies = self.repository.get_dependencies(ireq)
  File "/usr/local/lib/python3.7/site-packages/prequ/repositories/base.py", line 39, in get_dependencies
    return self._get_dependencies(ireq)
  File "/usr/local/lib/python3.7/site-packages/prequ/repositories/pypi.py", line 126, in _get_dependencies
    ireq, wheel_cache)
  File "/usr/local/lib/python3.7/site-packages/prequ/repositories/pypi.py", line 179, in _get_dependencies_with_wheel_cache
    build_isolation=False
TypeError: __init__() missing 1 required positional argument: 'req_tracker'

pip freeze

root@5b55f2c59a74:/app# pip freeze
aioredis==1.1.0
asgiref==2.3.2
asn1crypto==0.24.0
async-timeout==3.0.0
attrs==18.1.0
autobahn==18.6.1
Automat==0.6.0
certifi==2018.4.16
cffi==1.11.5
channels==2.1.1
channels-redis==2.2.1
chardet==3.0.4
click==6.7
constantly==15.1.0
cryptography==2.2.2
daphne==2.1.2
Django==2.0.3
django-environ==0.4.4
djangorestframework==3.8.2
h2==3.0.1
hiredis==0.2.0
hpack==3.0.0
hyperframe==5.1.0
hyperlink==18.0.0
idna==2.6
incremental==17.5.0
msgpack==0.5.6
Pillow==5.0.0
prequ==1.4.1
priority==1.3.0
psycopg2==2.7.4
psycopg2-binary==2.7.4
pyasn1==0.4.3
pyasn1-modules==0.2.1
pycparser==2.18
pyOpenSSL==18.0.0
pytz==2018.4
requests==2.18.4
service-identity==17.0.0
six==1.11.0
stripe==1.82.0
Twisted==18.4.0
txaio==2.10.0
urllib3==1.22
zope.interface==4.5.0

setup.cfg

[prequ]
annotate = yes
header = yes

requirements =
  django==2.0.7
  django-environ==0.4.5
  djangorestframework==3.8.2

requirements-dev =
  prequ==1.4.1
  pytest
  pytest-cov
  pytest-django
  model_mommy
  faker

ls

root@5b55f2c59a74:/app# ls
apps/      build.py       requirements-builder.txt      static/            static_src/
core/      manage.py      setup.cfg                     static_built/      var/

other details

no requirement files that are configured in setup.cfg are present in the folder

Extras are ignored for directory entries

If source requirements contains directory requirements with extras (e.g. .[extra1,extra2] or ./some-dir[extra]), the generated requirements will have the dependencies of the extras, but the extras are not present in the file for the directory lines.

Prequ 1.3.0 normalizes period in package names incorrectly

If there is a period (.) in the name of the distribution package, it will be written as dash (-) to the output file which causes some errors.

E.g.

$ virtualenv -q -p python2.7 venv-py27
$ . venv-py27/bin/activate
(venv-py27) $ pip install -q prequ==1.3.0

###########################################
# The "via" comments are not generated correctly:

(venv-py27) $ echo 'repoze-who' > requirements.in
(venv-py27) $ echo ' # via ' > requirements.txt
(venv-py27) $ prequ update
*** Compiling requirements.txt  
(venv-py27) $ cat requirements.txt
repoze-who==2.3
webob==1.7.4              # via repoze-who
zope.interface==4.4.3
(venv-py27) £ prequ update
*** Compiling requirements.txt  
(venv-py27) $ cat requirements.txt
repoze-who==2.3
webob==1.7.4              # via repoze-who
zope-interface==4.4.3     # via repoze-who

###########################################
# The "prequ check" fails:

(venv-py27) $ echo -e 'repoze-who\nzope.interface<=4.0.5\n' > requirements.in
(venv-py27) $ rm requirements.txt 
(venv-py27) $ prequ update
*** Compiling requirements.txt
(venv-py27) $ cat requirements.txt 
# This file is autogenerated by Prequ.  To update, run:
#
#   prequ update
#
repoze-who==2.3
webob==1.7.4
zope.interface==4.0.5
(venv-py27) $ prequ check -v
--- requirements.txt (current)
+++ requirements.txt (expected)
@@ -4,4 +4,4 @@
 #
 repoze-who==2.3
 webob==1.7.4
-zope.interface==4.0.5
+zope-interface==4.0.5
requirements.txt is outdated

###########################################
# Or even more fatal ones, like this:

(venv-py27) $ rm ~/.cache/prequ/depcache-py2.7.json 
(venv-py27) $ prequ check -v
No handlers could be found for logger "pip.index"
Traceback (most recent call last):
  ...
  File ".../venv-py27/local/lib/python2.7/site-packages/prequ/repositories/pypi.py", line 151, in _get_dependencies
    deps = reqset._prepare_file(self.finder, ireq)
  File ".../venv-py27/local/lib/python2.7/site-packages/pip/req/req_set.py", line 554, in _prepare_file
    require_hashes
  File ".../venv-py27/local/lib/python2.7/site-packages/pip/req/req_install.py", line 278, in populate_link
    self.link = finder.find_requirement(self, upgrade)
  File ".../venv-py27/local/lib/python2.7/site-packages/pip/index.py", line 514, in find_requirement
    'No matching distribution found for %s' % req
pip.exceptions.DistributionNotFound: No matching distribution found for zope-interface==4.0.5

"prequ edit" command

Implement prequ edit PACKAGE command which can be used to turn a requirement to editable.

The command should:

Fetch the sources of the package to a directory like deps/PACKAGE or to a user specified directory (with --dir DIRECTORY or via deps_dir configuration variable). If there already is sources in the directory, don't override them.
- Note: This tries to create a VCS checkout when possible, e.g. if there is a Git repository found via package info, it should be cloned and checked out to the currently pinned version.
Make the package temporarily editable in the current virtualenv
- If the package is currently installed in the current virtualenv (which should be active), reinstall it as editable from the directory specified in 1.
- Store the info that the package is editable into a .prequ.state file so that prequ sync knows to leave it so.

The editable status can be reverted back with prequ unedit PACKAGE. This won't remove the source directory, but it reinstalls the package to make it non-editable in the virtualenv again. It also updates .prequ.state.

Unnecessary changes on update

It seems that some package names are randomly either with or without capital letter.

For me this change is introduced pretty much every prequ update.

-markupsafe==1.0           # via jinja2
+markupsafe==1.0           # via Jinja2

Improve error message for "wheel not found"

Currently, if wheel build fails to produce the wheel file (e.g. because of some build error), a very confusing IndexError is generated. It should rather generate a proper error message with better details of what went wrong.

The IndexError is caused by get_wheels returning an empty list here:

prequ/prequ/scripts/build_wheels.py

Line 51 in ed3b798

built_wheel = get_wheels(conf, pkg, ver)[0]

Thanks to @RauliL and @inesjelovac for reporting this to me.