Git Product home page Git Product logo

importlab's Introduction

importlab

Importlab is a library for Python that automatically infers dependencies and calculates a dependency graph. It can perform dependency ordering of a set of files, including cycle detection.

Importlab's main use case is to work with static analysis tools that process one file at a time, ensuring that a file's dependencies are analysed before it is.

(This is not an official Google product.)

License

Apache 2.0

Installation

Importlab can be installed from pip

pip install importlab

To check out and install the latest source code

git clone https://github.com/google/importlab.git
cd importlab
python setup.py install

Usage

Importlab is primarily intended to be used as a library. It takes one or more python files as arguments, and generates an import graph, typically used to process files in dependency order.

It is currently integrated into pytype

Command-line tool

Importlab ships with a small command-line tool, also called importlab, which can display some information about a project's import graph.

usage: importlab [-h] [--tree] [--unresolved] [filename [filename ...]]

positional arguments:
  filename              input file(s)

optional arguments:
  -h, --help            show this help message and exit
  --tree                Display import tree.
  --unresolved          Display unresolved dependencies.

Roadmap

  • Makefile generation, to take advantage of make's incremental update and parallel execution features
  • Integration with other static analysis tools

importlab's People

Contributors

cclauss avatar faunris avatar jayvdb avatar martindemello avatar matthiaskramm avatar mihi314 avatar msuozzo avatar rchen152 avatar solumin avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

importlab's Issues

Command-line tool needs extension?

pip install importlab

Expected to use the importlab command-line tool on Windows but:

'importlab' is not recognized as an internal or external command,
operable program or batch file.

I saw importlab was in fact sitting in C:\Users...\AppData\Local\Programs\Python\Python38-32\Scripts, but I guess it wouldn't do anything without an extension, so I just renamed it to runimportlab.py to avoid circular reference and works fine.

runimportlab.py --tree script.py 

Importlab doesn't work when passed a directory.

Command: importlab google --unresolved
(Running in the google-auth repository.)

Result:

Traceback (most recent call last):
  File "/usr/local/google/home/tsudol/pytype-venv/bin/importlab", line 69, in <module>
    sys.exit(main())
  File "/usr/local/google/home/tsudol/pytype-venv/bin/importlab", line 53, in main
    import_graph = graph.ImportGraph.create(env, args.filenames)
  File "/usr/local/google/home/tsudol/pytype-venv/local/lib/python2.7/site-packages/importlab/graph.py", line 248, in create
    import_graph.add_file_recursive(os.path.abspath(filename))
  File "/usr/local/google/home/tsudol/pytype-venv/local/lib/python2.7/site-packages/importlab/graph.py", line 123, in add_file_recursive
    deps, broken = self.get_file_deps(filename)
  File "/usr/local/google/home/tsudol/pytype-venv/local/lib/python2.7/site-packages/importlab/graph.py", line 262, in get_file_deps
    for imp in parsepy.get_imports(filename, self.env.python_version):
  File "/usr/local/google/home/tsudol/pytype-venv/local/lib/python2.7/site-packages/importlab/parsepy.py", line 90, in get_imports
    raise Exception('parse error for ' + filename)
Exception: parse error for /usr/local/google/home/tsudol/google-auth-library-python/google

Expected result:
A list of all unresolved dependencies in the directory.

Running with google/*.py worked.

importlab interacts badly with `pyenv local`

$ virtualenv testbug
New python executable in testbug/bin/python
Installing setuptools, pip, wheel...done.
$ source testbug/bin/activate
$ echo "import rsa" > /tmp/test_rsa.py
$ pip install pytype
Collecting pytype
Collecting six (from pytype)
  Using cached https://files.pythonhosted.org/packages/67/4b/141a581104b1f6397bfa78ac9d43d8ad29a7ca43ea90a2d863fe3056e86a/six-1.11.0-py2.py3-none-any.whl
Collecting importlab (from pytype)
Collecting pyyaml>=3.11 (from pytype)
Collecting networkx (from importlab->pytype)
Collecting decorator>=4.1.0 (from networkx->importlab->pytype)
  Using cached https://files.pythonhosted.org/packages/bc/bb/a24838832ba35baf52f32ab1a49b906b5f82fb7c76b2f6a7e35e140bac30/decorator-4.3.0-py2.py3-none-any.whl
Installing collected packages: six, decorator, networkx, importlab, pyyaml, pytype
Successfully installed decorator-4.3.0 importlab-0.2.1 networkx-2.1 pytype-2018.8.10 pyyaml-3.13 six-1.11.0
$ pytype -P /tmp /tmp/test_rsa.py
Analyzing 1 sources with 0 dependencies
File "/tmp/test_rsa.py", line 1, in <module>: Can't find module 'rsa'. [import-error]

For more details, see https://github.com/google/pytype/blob/master/docs/errors.md#import-error.
$ pip install rsa
Collecting rsa
  Using cached https://files.pythonhosted.org/packages/e1/ae/baedc9cb175552e95f3395c43055a6a5e125ae4d48a1d7a924baca83e92e/rsa-3.4.2-py2.py3-none-any.whl
Collecting pyasn1>=0.1.3 (from rsa)
  Using cached https://files.pythonhosted.org/packages/d1/a1/7790cc85db38daa874f6a2e6308131b9953feb1367f2ae2d1123bb93a9f5/pyasn1-0.4.4-py2.py3-none-any.whl
Installing collected packages: pyasn1, rsa
Successfully installed pyasn1-0.4.4 rsa-3.4.2
$ pytype -P /tmp /tmp/test_rsa.py
Analyzing 1 sources with 0 dependencies
File "/tmp/test_rsa.py", line 1, in <module>: Can't find module 'rsa'. [import-error]

For more details, see https://github.com/google/pytype/blob/master/docs/errors.md#import-error.
$ python /tmp/test_rsa.py
$ importlab /tmp/test_rsa.py --unresolved
Unresolved dependencies:
  rsa

pyenv breaks unittests

Running on a pyenv-installed python seems to break several unittests. Running pyenv shell system makes the tests work again. Tried with both 2.7 and 3.6

importlab crashes on file with tensorflow import

importlab version: 0.3.0

When running importlab my_file.py --tree on a file with contents import tensorflow, the result is the following exception:

Reading 1 files
/home/me/.local/lib/python3.6/site-packages/IPython/kernel/__init__.py:13: ShimWarning: The `IPython.kernel` package has been deprecated since IPython 4.0.You should import from ipykernel or jupyter_client instead.
  "You should import from ipykernel or jupyter_client instead.", ShimWarning)
Traceback (most recent call last):
  File "/usr/local/bin/importlab", line 79, in <module>
    sys.exit(main())
  File "/usr/local/bin/importlab", line 63, in main
    import_graph = graph.ImportGraph.create(env, args.inputs)
  File "/usr/local/lib/python3.6/dist-packages/importlab/graph.py", line 258, in create
    import_graph.add_file_recursive(os.path.abspath(filename))
  File "/usr/local/lib/python3.6/dist-packages/importlab/graph.py", line 128, in add_file_recursive
    deps, broken = self.get_file_deps(filename)
  File "/usr/local/lib/python3.6/dist-packages/importlab/graph.py", line 272, in get_file_deps
    for imp in parsepy.get_imports(filename, self.env.python_version):
  File "/usr/local/lib/python3.6/dist-packages/importlab/parsepy.py", line 79, in get_imports
    imports = import_finder.get_imports(filename)
  File "/usr/local/lib/python3.6/dist-packages/importlab/import_finder.py", line 136, in get_imports
    imports.append(i + (resolve_import(name, is_from, is_star),))
  File "/usr/local/lib/python3.6/dist-packages/importlab/import_finder.py", line 116, in resolve_import
    ret = _resolve_import(name)
  File "/usr/local/lib/python3.6/dist-packages/importlab/import_finder.py", line 98, in _resolve_import
    return _resolve_import_3(name)
  File "/usr/local/lib/python3.6/dist-packages/importlab/import_finder.py", line 80, in _resolve_import_3
    spec = importlib.util.find_spec(name)  # pytype: disable=module-attr
  File "/usr/lib/python3.6/importlib/util.py", line 88, in find_spec
    parent = __import__(parent_name, fromlist=['__path__'])
  File "/home/me/.local/lib/python3.6/site-packages/IPython/utils/_process_win32.py", line 23, in <module>
    from ctypes.wintypes import LPCWSTR, HLOCAL
  File "/usr/lib/python3.6/ctypes/wintypes.py", line 20, in <module>
    class VARIANT_BOOL(ctypes._SimpleCData):
ValueError: _type_ 'v' not supported

module recorded as "unreadable" when one of its direct dependencies is unloadable

Reproduce with:

git clone [email protected]:GoogleCloudPlatform/django-cloud-deploy.git
cd django-cloud-deploy
pip install .
importlab --tree django-cloud-deploy/cloudlib/database.py

importlab reports:

Reading 1 files
Source tree:

Unreadable files:
  /usr/local/google/home/rechen/django-cloud-deploy/django_cloud_deploy/cloudlib/database.py

In fact, database.py is readable, but a file in one of its pip-installed dependencies (django/contrib/auth/models.py) throws an error at import time.

  • importlab should mark models.py instead of database.py as unreadable.
  • It would be useful to surface the actual error that occurred, to give the user a better idea of why the file is unreadable.

parsepy_test is broken

I executed python -m importlab.parsepy_test and got a bunch of errors of the following form:

ERROR: test_simple (\__main__.TestParsePy)
Traceback (most recent call last):
  File "/usr/local/google/home/rechen/importlab/importlab/parsepy_test.py", line 36, in test_simple
    """), [parsepy.ImportStatement(name="a")])
  File "/usr/local/google/home/rechen/importlab/importlab/parsepy_test.py", line 31, in parse
    return parsepy.scan_string(textwrap.dedent(src))
AttributeError: 'module' object has no attribute 'scan_string'

port the ninja file generator from pytype

pytype can take the depgraph produced by importlab and generate a ninja file to process files in dependency order. it would be useful to have that as a more general library within importlab.

Improve the default value of the --python-version option

EDIT: this post now demonstrates how easy it is to forget to set -V.

Repro:

echo "import django" > foo.py
echo "import jinja2" > bar.py
virtualenv --python=python3.5 .venv
source .venv/bin/activate
pip install importlab django jinja2
importlab --tree foo.py
importlab --tree bar.py

Neither dependency is found. Replace python3.5 with python3.6, and both are found.

Crash when analyzing files of different Python versions

Input:

# file a.py
import b

# file b.py
raise 1, 2, 3  # Python 2 syntax

Output of -V2.7:

$ importlab -V2.7 -P. *.py --tree
Source tree:
+ a.py
    + b.py

Output of -V3.6:

$ importlab -V3.6 -P. *.py --tree
Traceback (most recent call last):
  File "/usr/bin/importlab", line 69, in <module>
    sys.exit(main())
  File "/usr/bin/importlab", line 53, in main
    import_graph = graph.ImportGraph.create(env, args.filenames)
  File "/usr/local/lib/python2.7/site-packages/importlab/graph.py", line 248, in create
    import_graph.add_file_recursive(os.path.abspath(filename))
  File "/usr/local/lib/python2.7/site-packages/importlab/graph.py", line 123, in add_file_recursive
    deps, broken = self.get_file_deps(filename)
  File "/usr/local/lib/python2.7/site-packages/importlab/graph.py", line 262, in get_file_deps
    for imp in parsepy.get_imports(filename, self.env.python_version):
  File "/usr/local/lib/python2.7/site-packages/importlab/parsepy.py", line 90, in get_imports
    raise Exception('parse error for ' + filename)
Exception: parse error for /tmp/imptest/b.py

It would be nice if importlab didn't crash here. I think a --keep-going option would be useful for some users.

version 0.8 seems to miss some imports

Running with --tree --trim on this file https://github.com/deepmind/tf2jax/blob/main/tf2jax/_src/ops.py

0.7 gives the correct tree

+ tf2jax/_src/ops.py
    :: dataclasses.py
    :: functools.py
    :: typing.py
    :: absl/logging/__init__.py
    :: jax/__init__.py
    :: jax/_src/lax/control_flow/__init__.py
    :: jax/numpy/__init__.py
    :: numpy/__init__.py
    :: tensorflow/__init__.py
      tf2jax/_src/config.py
        :: contextlib.py
      tf2jax/_src/numpy_compat.py
      tf2jax/_src/xla_utils.py

but 0.8 misses tf2jax/_src/config.py

+ tf2jax/_src/ops.py
    :: dataclasses.py
    :: functools.py
    :: typing.py
    :: absl/logging/__init__.py
    :: jax/__init__.py
    :: jax/_src/lax/control_flow/__init__.py
    :: jax/numpy/__init__.py
    :: numpy/__init__.py
    :: tensorflow/__init__.py
      tf2jax/_src/__init__.py
      tf2jax/_src/numpy_compat.py
      tf2jax/_src/xla_utils.py

Any idea what is causing this?

Importlab running under Python 3.6 and analyzing Python 2.7 can't find networkx

Steps I took:

sudo apt-get install python-pip
pip install networkx # puts networkx in ~/.local/lib/python2.7/site-packages/
virtualenv --python=python3.6 .venv3
source .venv3/bin/activate
pip install importlab
echo "import networkx" > foo.py
importlab -V2.7 foo.py --tree

foo.py shows up as the only file in the tree. On the other hand, if I install importlab under Python 2.7 and analyze Python 3.5 code (I didn't try 3.6 because pip3 is 3.5 on my machine), the last command (correctly) causes a bunch of networkx files to be printed as part of the tree.

Can't find pip-installed google-auth in Python 2

In a Python 3.6 virtualenv:

$ pip install google-auth
$ cat foo.py
import google.auth
$ importlab --tree foo.py -V3.6
Reading 1 files
Source tree:
+ foo.py
    :: google/auth/__init__.py
    ...

But in a Python 2.7 virtualenv:

$ pip install google-auth
$ cat foo.py
import google.auth
$ importlab --tree foo.py -V2.7
Reading 1 files
Source tree:
+ foo.py

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.