google / importlab Goto Github PK

A library that automatically infers dependencies for Python files

License: Apache License 2.0

Python 99.69% Shell 0.31%

importlab's Issues

port the ninja file generator from pytype

pytype can take the depgraph produced by importlab and generate a ninja file to process files in dependency order. it would be useful to have that as a more general library within importlab.

Can't find pip-installed google-auth in Python 2

In a Python 3.6 virtualenv:

$ pip install google-auth
$ cat foo.py
import google.auth
$ importlab --tree foo.py -V3.6
Reading 1 files
Source tree:
+ foo.py
    :: google/auth/__init__.py
    ...

But in a Python 2.7 virtualenv:

$ pip install google-auth
$ cat foo.py
import google.auth
$ importlab --tree foo.py -V2.7
Reading 1 files
Source tree:
+ foo.py

pyenv breaks unittests

Running on a pyenv-installed python seems to break several unittests. Running pyenv shell system makes the tests work again. Tried with both 2.7 and 3.6

Command-line tool needs extension?

pip install importlab

Expected to use the importlab command-line tool on Windows but:

'importlab' is not recognized as an internal or external command,
operable program or batch file.

I saw importlab was in fact sitting in C:\Users...\AppData\Local\Programs\Python\Python38-32\Scripts, but I guess it wouldn't do anything without an extension, so I just renamed it to runimportlab.py to avoid circular reference and works fine.

runimportlab.py --tree script.py

Run tests via python > 3.0 failed

Test TestParsePy uses python 2.7 syntax. And this tests failed if run on python3.

parsepy_test is broken

I executed python -m importlab.parsepy_test and got a bunch of errors of the following form:

ERROR: test_simple (\__main__.TestParsePy)
Traceback (most recent call last):
  File "/usr/local/google/home/rechen/importlab/importlab/parsepy_test.py", line 36, in test_simple
    """), [parsepy.ImportStatement(name="a")])
  File "/usr/local/google/home/rechen/importlab/importlab/parsepy_test.py", line 31, in parse
    return parsepy.scan_string(textwrap.dedent(src))
AttributeError: 'module' object has no attribute 'scan_string'

Add --version to bin importlab for auto generate man page via help2man

Hello. I'm deb packaging this tools.

For use help2man, we need add --version param to bin
PYTHONPATH=. help2man 'python3 ./bin/importlab '

Importlab running under Python 3.6 and analyzing Python 2.7 can't find networkx

Steps I took:

sudo apt-get install python-pip
pip install networkx # puts networkx in ~/.local/lib/python2.7/site-packages/
virtualenv --python=python3.6 .venv3
source .venv3/bin/activate
pip install importlab
echo "import networkx" > foo.py
importlab -V2.7 foo.py --tree

foo.py shows up as the only file in the tree. On the other hand, if I install importlab under Python 2.7 and analyze Python 3.5 code (I didn't try 3.6 because pip3 is 3.5 on my machine), the last command (correctly) causes a bunch of networkx files to be printed as part of the tree.

version 0.8 seems to miss some imports

Running with --tree --trim on this file https://github.com/deepmind/tf2jax/blob/main/tf2jax/_src/ops.py

0.7 gives the correct tree

+ tf2jax/_src/ops.py
    :: dataclasses.py
    :: functools.py
    :: typing.py
    :: absl/logging/__init__.py
    :: jax/__init__.py
    :: jax/_src/lax/control_flow/__init__.py
    :: jax/numpy/__init__.py
    :: numpy/__init__.py
    :: tensorflow/__init__.py
      tf2jax/_src/config.py
        :: contextlib.py
      tf2jax/_src/numpy_compat.py
      tf2jax/_src/xla_utils.py

but 0.8 misses tf2jax/_src/config.py

+ tf2jax/_src/ops.py
    :: dataclasses.py
    :: functools.py
    :: typing.py
    :: absl/logging/__init__.py
    :: jax/__init__.py
    :: jax/_src/lax/control_flow/__init__.py
    :: jax/numpy/__init__.py
    :: numpy/__init__.py
    :: tensorflow/__init__.py
      tf2jax/_src/__init__.py
      tf2jax/_src/numpy_compat.py
      tf2jax/_src/xla_utils.py

Any idea what is causing this?

Incorrect inferred module name for init.py files

See google/pytype#154 for more detail.

resolve.infer_module_name calculates the wrong name for __init__.py files. For example, for foo/bar/__init__.py, it will return foo.bar.__init__. The module name should be foo.bar.

importlab interacts badly with `pyenv local`

$ virtualenv testbug
New python executable in testbug/bin/python
Installing setuptools, pip, wheel...done.
$ source testbug/bin/activate
$ echo "import rsa" > /tmp/test_rsa.py
$ pip install pytype
Collecting pytype
Collecting six (from pytype)
  Using cached https://files.pythonhosted.org/packages/67/4b/141a581104b1f6397bfa78ac9d43d8ad29a7ca43ea90a2d863fe3056e86a/six-1.11.0-py2.py3-none-any.whl
Collecting importlab (from pytype)
Collecting pyyaml>=3.11 (from pytype)
Collecting networkx (from importlab->pytype)
Collecting decorator>=4.1.0 (from networkx->importlab->pytype)
  Using cached https://files.pythonhosted.org/packages/bc/bb/a24838832ba35baf52f32ab1a49b906b5f82fb7c76b2f6a7e35e140bac30/decorator-4.3.0-py2.py3-none-any.whl
Installing collected packages: six, decorator, networkx, importlab, pyyaml, pytype
Successfully installed decorator-4.3.0 importlab-0.2.1 networkx-2.1 pytype-2018.8.10 pyyaml-3.13 six-1.11.0
$ pytype -P /tmp /tmp/test_rsa.py
Analyzing 1 sources with 0 dependencies
File "/tmp/test_rsa.py", line 1, in <module>: Can't find module 'rsa'. [import-error]

For more details, see https://github.com/google/pytype/blob/master/docs/errors.md#import-error.
$ pip install rsa
Collecting rsa
  Using cached https://files.pythonhosted.org/packages/e1/ae/baedc9cb175552e95f3395c43055a6a5e125ae4d48a1d7a924baca83e92e/rsa-3.4.2-py2.py3-none-any.whl
Collecting pyasn1>=0.1.3 (from rsa)
  Using cached https://files.pythonhosted.org/packages/d1/a1/7790cc85db38daa874f6a2e6308131b9953feb1367f2ae2d1123bb93a9f5/pyasn1-0.4.4-py2.py3-none-any.whl
Installing collected packages: pyasn1, rsa
Successfully installed pyasn1-0.4.4 rsa-3.4.2
$ pytype -P /tmp /tmp/test_rsa.py
Analyzing 1 sources with 0 dependencies
File "/tmp/test_rsa.py", line 1, in <module>: Can't find module 'rsa'. [import-error]

For more details, see https://github.com/google/pytype/blob/master/docs/errors.md#import-error.
$ python /tmp/test_rsa.py
$ importlab /tmp/test_rsa.py --unresolved
Unresolved dependencies:
  rsa

Name clash between input file and system file leads to incorrect dependency tree.

Example:

$ cat abc.py
import abc
$ importlab --tree abc.py 
Reading 1 files
Source tree:
cycle {
    abc.py
}

If I rename the file to anything else, importlab generates the correct source tree:

Source tree:
+ foo.py
    :: abc.py
        :: _weakrefset.py

I'm running importlab-0.3.0 under Python 3.6

Tags are not up to date with pypi

pypi has v0.5, but tags has only v0.1.1

Include tests in sdist

Please include the tests in each sdist.

The lack of tags means I cant even use a github archive as a substitute.

c.f. https://build.opensuse.org/request/show/679730 which needs them, so this library is needed for pytype.

importlab crashes on file with tensorflow import

importlab version: 0.3.0

When running importlab my_file.py --tree on a file with contents import tensorflow, the result is the following exception:

Reading 1 files
/home/me/.local/lib/python3.6/site-packages/IPython/kernel/__init__.py:13: ShimWarning: The `IPython.kernel` package has been deprecated since IPython 4.0.You should import from ipykernel or jupyter_client instead.
  "You should import from ipykernel or jupyter_client instead.", ShimWarning)
Traceback (most recent call last):
  File "/usr/local/bin/importlab", line 79, in <module>
    sys.exit(main())
  File "/usr/local/bin/importlab", line 63, in main
    import_graph = graph.ImportGraph.create(env, args.inputs)
  File "/usr/local/lib/python3.6/dist-packages/importlab/graph.py", line 258, in create
    import_graph.add_file_recursive(os.path.abspath(filename))
  File "/usr/local/lib/python3.6/dist-packages/importlab/graph.py", line 128, in add_file_recursive
    deps, broken = self.get_file_deps(filename)
  File "/usr/local/lib/python3.6/dist-packages/importlab/graph.py", line 272, in get_file_deps
    for imp in parsepy.get_imports(filename, self.env.python_version):
  File "/usr/local/lib/python3.6/dist-packages/importlab/parsepy.py", line 79, in get_imports
    imports = import_finder.get_imports(filename)
  File "/usr/local/lib/python3.6/dist-packages/importlab/import_finder.py", line 136, in get_imports
    imports.append(i + (resolve_import(name, is_from, is_star),))
  File "/usr/local/lib/python3.6/dist-packages/importlab/import_finder.py", line 116, in resolve_import
    ret = _resolve_import(name)
  File "/usr/local/lib/python3.6/dist-packages/importlab/import_finder.py", line 98, in _resolve_import
    return _resolve_import_3(name)
  File "/usr/local/lib/python3.6/dist-packages/importlab/import_finder.py", line 80, in _resolve_import_3
    spec = importlib.util.find_spec(name)  # pytype: disable=module-attr
  File "/usr/lib/python3.6/importlib/util.py", line 88, in find_spec
    parent = __import__(parent_name, fromlist=['__path__'])
  File "/home/me/.local/lib/python3.6/site-packages/IPython/utils/_process_win32.py", line 23, in <module>
    from ctypes.wintypes import LPCWSTR, HLOCAL
  File "/usr/lib/python3.6/ctypes/wintypes.py", line 20, in <module>
    class VARIANT_BOOL(ctypes._SimpleCData):
ValueError: _type_ 'v' not supported

module recorded as "unreadable" when one of its direct dependencies is unloadable

Reproduce with:

git clone [email protected]:GoogleCloudPlatform/django-cloud-deploy.git
cd django-cloud-deploy
pip install .
importlab --tree django-cloud-deploy/cloudlib/database.py

importlab reports:

Reading 1 files
Source tree:

Unreadable files:
  /usr/local/google/home/rechen/django-cloud-deploy/django_cloud_deploy/cloudlib/database.py

In fact, database.py is readable, but a file in one of its pip-installed dependencies (django/contrib/auth/models.py) throws an error at import time.

importlab should mark models.py instead of database.py as unreadable.
It would be useful to surface the actual error that occurred, to give the user a better idea of why the file is unreadable.

# file a.py
import b

# file b.py
raise 1, 2, 3  # Python 2 syntax

Output of -V2.7:

$ importlab -V2.7 -P. *.py --tree
Source tree:
+ a.py
    + b.py

Output of -V3.6:

$ importlab -V3.6 -P. *.py --tree
Traceback (most recent call last):
  File "/usr/bin/importlab", line 69, in <module>
    sys.exit(main())
  File "/usr/bin/importlab", line 53, in main
    import_graph = graph.ImportGraph.create(env, args.filenames)
  File "/usr/local/lib/python2.7/site-packages/importlab/graph.py", line 248, in create
    import_graph.add_file_recursive(os.path.abspath(filename))
  File "/usr/local/lib/python2.7/site-packages/importlab/graph.py", line 123, in add_file_recursive
    deps, broken = self.get_file_deps(filename)
  File "/usr/local/lib/python2.7/site-packages/importlab/graph.py", line 262, in get_file_deps
    for imp in parsepy.get_imports(filename, self.env.python_version):
  File "/usr/local/lib/python2.7/site-packages/importlab/parsepy.py", line 90, in get_imports
    raise Exception('parse error for ' + filename)
Exception: parse error for /tmp/imptest/b.py

It would be nice if importlab didn't crash here. I think a --keep-going option would be useful for some users.

Don't require .py extension

As mentioned in 111, remove ".py" check.

Improve the default value of the --python-version option

EDIT: this post now demonstrates how easy it is to forget to set -V.

Repro:

echo "import django" > foo.py
echo "import jinja2" > bar.py
virtualenv --python=python3.5 .venv
source .venv/bin/activate
pip install importlab django jinja2
importlab --tree foo.py
importlab --tree bar.py

Neither dependency is found. Replace python3.5 with python3.6, and both are found.

Importlab doesn't work when passed a directory.

Command: importlab google --unresolved
(Running in the google-auth repository.)

Result:

Traceback (most recent call last):
  File "/usr/local/google/home/tsudol/pytype-venv/bin/importlab", line 69, in <module>
    sys.exit(main())
  File "/usr/local/google/home/tsudol/pytype-venv/bin/importlab", line 53, in main
    import_graph = graph.ImportGraph.create(env, args.filenames)
  File "/usr/local/google/home/tsudol/pytype-venv/local/lib/python2.7/site-packages/importlab/graph.py", line 248, in create
    import_graph.add_file_recursive(os.path.abspath(filename))
  File "/usr/local/google/home/tsudol/pytype-venv/local/lib/python2.7/site-packages/importlab/graph.py", line 123, in add_file_recursive
    deps, broken = self.get_file_deps(filename)
  File "/usr/local/google/home/tsudol/pytype-venv/local/lib/python2.7/site-packages/importlab/graph.py", line 262, in get_file_deps
    for imp in parsepy.get_imports(filename, self.env.python_version):
  File "/usr/local/google/home/tsudol/pytype-venv/local/lib/python2.7/site-packages/importlab/parsepy.py", line 90, in get_imports
    raise Exception('parse error for ' + filename)
Exception: parse error for /usr/local/google/home/tsudol/google-auth-library-python/google

Expected result:
A list of all unresolved dependencies in the directory.

Running with google/*.py worked.

google / importlab Goto Github PK

importlab's Issues

Recommend Projects

Recommend Topics

Recommend Org