The current formats of stdlib and mapping aren't very flexible, changing them to something more suitable like JSON or python dictionary would make them more flexible. This change would make it easier to include and exclude certain modules based on python version, and provide simple fixes to issues such as #61, #62 and #74.

For example, to import beautifulsoup4 package, it should be import bs4.
So pipreqs cannot find the beautifulsoup4 package from pypi server.
I don't have an idea to resolve this problem... :(

Before looking up package on PyPi check if package is present locally and if yes then use the version of the local package.

When I try to run the pipreqs from cli, I got ther is no command found.
I miss something? The installation process and the usage is jus this:

$ pip install pipreqs
$ pipreqs [path/to/project]

Perhaps this is an edge case, but I found that because I have an IPython build in an unignored subdirectory of my package, I am getting a weird dependency output. (specifically,
A == 1.0
B == 1.0.0
)

The culprit file: test_deepreload.py: https://github.com/ipython/ipython/blob/40c34d3369c3d271149e13ac07f1bd2f5a716635/IPython/lib/tests/test_deepreload.py)
Relevant snippet included:

with open(os.path.join(tmpdir, 'A.py'), 'w') as f: f.write("class Object(object):\n pass\n") with open(os.path.join(tmpdir, 'B.py'), 'w') as f: f.write("import A\n") import A import B

This is less pipreqs fault and more IPythons fault, but it's an annoying edge case nonetheless

Please include your flake8 configuration file in the repo and update CONTRIBUTING.rst to use it.

Before making any changes, I followed the instructions in CONTRIBUTING.rst, and flake8 complained much. I used:

Centos 7(.2)
Python 2.7.5
virtualenvwrapper 4.7.2
virtualenv 15.1.0
pip 9.0.1
flake8 3.3.0

flake8 output:

(pipreqs) [doj@pan pipreqs]$ flake8 pipreqs tests
pipreqs/pipreqs.py:11:80: E501 line too long (108 > 79 characters)
pipreqs/pipreqs.py:19:80: E501 line too long (80 > 79 characters)
pipreqs/pipreqs.py:69:80: E501 line too long (87 > 79 characters)
pipreqs/pipreqs.py:82:80: E501 line too long (90 > 79 characters)
pipreqs/pipreqs.py:85:80: E501 line too long (91 > 79 characters)
pipreqs/pipreqs.py:90:5: E303 too many blank lines (3)
pipreqs/pipreqs.py:92:80: E501 line too long (96 > 79 characters)
pipreqs/pipreqs.py:94:80: E501 line too long (84 > 79 characters)
pipreqs/pipreqs.py:118:80: E501 line too long (99 > 79 characters)
pipreqs/pipreqs.py:121:1: E302 expected 2 blank lines, found 1
pipreqs/pipreqs.py:127:80: E501 line too long (86 > 79 characters)
pipreqs/pipreqs.py:128:30: E127 continuation line over-indented for visual indent
pipreqs/pipreqs.py:130:1: E302 expected 2 blank lines, found 1
pipreqs/pipreqs.py:130:80: E501 line too long (87 > 79 characters)
pipreqs/pipreqs.py:135:80: E501 line too long (91 > 79 characters)
pipreqs/pipreqs.py:159:80: E501 line too long (90 > 79 characters)
pipreqs/pipreqs.py:223:49: W291 trailing whitespace
pipreqs/pipreqs.py:225:51: E251 unexpected spaces around keyword / parameter equals
pipreqs/pipreqs.py:225:53: E251 unexpected spaces around keyword / parameter equals
pipreqs/pipreqs.py:253:80: E501 line too long (103 > 79 characters)
tests/test_pipreqs.py:23:80: E501 line too long (113 > 79 characters)
tests/test_pipreqs.py:23:112: E202 whitespace before ']'
tests/test_pipreqs.py:27:80: E501 line too long (87 > 79 characters)
tests/test_pipreqs.py:28:80: E501 line too long (100 > 79 characters)
tests/test_pipreqs.py:29:80: E501 line too long (108 > 79 characters)
tests/test_pipreqs.py:57:80: E501 line too long (85 > 79 characters)
tests/test_pipreqs.py:65:80: E501 line too long (110 > 79 characters)
tests/test_pipreqs.py:74:80: E501 line too long (97 > 79 characters)
tests/test_pipreqs.py:88:80: E501 line too long (83 > 79 characters)
tests/test_pipreqs.py:89:70: E231 missing whitespace after ':'
tests/test_pipreqs.py:89:80: E501 line too long (98 > 79 characters)
tests/test_pipreqs.py:89:92: E231 missing whitespace after ':'
tests/test_pipreqs.py:98:80: E501 line too long (83 > 79 characters)
tests/test_pipreqs.py:100:80: E501 line too long (83 > 79 characters)
tests/test_pipreqs.py:101:70: E231 missing whitespace after ':'
tests/test_pipreqs.py:101:80: E501 line too long (98 > 79 characters)
tests/test_pipreqs.py:101:92: E231 missing whitespace after ':'
tests/test_pipreqs.py:114:80: E501 line too long (127 > 79 characters)
tests/test_pipreqs.py:114:80: E231 missing whitespace after ':'
tests/test_pipreqs.py:114:102: E231 missing whitespace after ':'
tests/test_pipreqs.py:125:80: E501 line too long (85 > 79 characters)
tests/test_pipreqs.py:130:70: E231 missing whitespace after ':'
tests/test_pipreqs.py:130:80: E501 line too long (116 > 79 characters)
tests/test_pipreqs.py:130:92: E231 missing whitespace after ':'
tests/test_pipreqs.py:139:80: E501 line too long (87 > 79 characters)
tests/test_pipreqs.py:152:80: E501 line too long (137 > 79 characters)
tests/test_pipreqs.py:153:80: E501 line too long (113 > 79 characters)
tests/test_pipreqs.py:160:80: E501 line too long (96 > 79 characters)
tests/test_pipreqs.py:161:23: E127 continuation line over-indented for visual indent
tests/test_pipreqs.py:162:32: E231 missing whitespace after ':'
tests/test_pipreqs.py:163:38: E231 missing whitespace after ':'
tests/test_pipreqs.py:164:33: E231 missing whitespace after ':'
tests/test_pipreqs.py:165:13: E124 closing bracket does not match visual indentation
tests/test_pipreqs.py:167:80: E501 line too long (98 > 79 characters)
tests/test_pipreqs.py:173:5: E303 too many blank lines (2)
tests/_invalid_data/invalid.py:1:17: E401 multiple imports on one line
tests/_invalid_data/invalid.py:1:24: E999 SyntaxError: invalid syntax
tests/_data/test.py:2:80: E501 line too long (117 > 79 characters)
tests/_data/test.py:4:1: F401 'xml.sax' imported but unused
tests/_data/test.py:5:1: F401 'os.path as test' imported but unused
tests/_data/test.py:6:1: F401 'sys.argv as test2' imported but unused
tests/_data/test.py:7:1: F401 'sys.flags' imported but unused
tests/_data/test.py:9:1: F401 'collections.OrderedDict' imported but unused
tests/_data/test.py:9:1: F401 'collections.deque' imported but unused
tests/_data/test.py:9:1: F401 'collections.Counter' imported but unused
tests/_data/test.py:11:1: F401 'requests' imported but unused
tests/_data/test.py:17:1: F401 'importlib.invalidate_caches' imported but unused
tests/_data/test.py:20:1: F811 redefinition of unused 'zipimport' from line 14
tests/_data/test.py:20:1: F401 'zipimport' imported but unused
tests/_data/test.py:22:1: F401 'importlib.machinery.all_suffixes' imported but unused
tests/_data/test.py:23:1: F401 'importlib' imported but unused
tests/_data/test.py:25:1: F401 'IPython.utils.importstring.import_item' imported but unused
tests/_data/test.py:29:1: F401 'pyflakes.test.test_imports.Test as TestImports' imported but unused
tests/_data/test.py:32:1: F401 'nose.importer.add_path' imported but unused
tests/_data/test.py:32:1: F401 'nose.importer.Importer' imported but unused
tests/_data/test.py:32:1: F401 'nose.importer.remove_path' imported but unused
tests/_data/test.py:34:1: F401 'atexit' imported but unused
tests/_data/test.py:35:1: F404 from future imports must occur at the beginning of the file
tests/_data/test.py:36:1: F401 'docopt.docopt' imported but unused
tests/_data/test.py:37:1: F401 'curses' imported but unused
tests/_data/test.py:37:14: E401 multiple imports on one line
tests/_data/test.py:38:1: F811 redefinition of unused 'logging' from line 37
tests/_data/test.py:38:1: F401 'logging' imported but unused
tests/_data/test.py:39:1: F401 'os' imported but unused
tests/_data/test.py:40:1: F811 redefinition of unused 'sqlite3' from line 37
tests/_data/test.py:40:1: F401 'sqlite3' imported but unused
tests/_data/test.py:41:1: F401 'time' imported but unused
tests/_data/test.py:42:1: F401 'sys' imported but unused
tests/_data/test.py:43:1: F401 'signal' imported but unused
tests/_data/test.py:44:1: F401 'bs4' imported but unused
tests/_data/test.py:45:1: F401 'nonexistendmodule' imported but unused
tests/_data/test.py:46:1: F401 'boto as b' imported but unused
tests/_data/test.py:46:1: F401 'peewee as p' imported but unused
tests/_data/test.py:46:17: E401 multiple imports on one line
tests/_data/test.py:48:1: F401 'flask.ext.somext' imported but unused
tests/_data/test.py:49:1: F401 'sqlalchemy.model' imported but unused
tests/_data/test.py:53:5: F401 'json' imported but unused
tests/_data/test.py:55:1: F401 'models' imported but unused
tests/_data/test.py:61:1: E305 expected 2 blank lines after class or function definition, found 1
tests/_data/test.py:61:1: E402 module level import not at top of file
tests/_data/test.py:61:1: F401 'after_method_is_valid_even_if_not_pep8' imported but unused
tests/_data_duplicated_deps/db.py:1:1: F401 'pymongo' imported but unused
tests/_data_duplicated_deps/db.py:2:1: F401 'bson.objectid.ObjectId' imported but unused
tests/_data_ignore/test.py:2:80: E501 line too long (117 > 79 characters)
tests/_data_ignore/test.py:4:1: F401 'xml.sax' imported but unused
tests/_data_ignore/test.py:5:1: F401 'os.path as test' imported but unused
tests/_data_ignore/test.py:6:1: F401 'sys.argv as test2' imported but unused
tests/_data_ignore/test.py:7:1: F401 'sys.flags' imported but unused
tests/_data_ignore/test.py:9:1: F401 'collections.OrderedDict' imported but unused
tests/_data_ignore/test.py:9:1: F401 'collections.deque' imported but unused
tests/_data_ignore/test.py:9:1: F401 'collections.Counter' imported but unused
tests/_data_ignore/test.py:11:1: F401 'requests' imported but unused
tests/_data_ignore/test.py:17:1: F401 'importlib.invalidate_caches' imported but unused
tests/_data_ignore/test.py:20:1: F811 redefinition of unused 'zipimport' from line 14
tests/_data_ignore/test.py:20:1: F401 'zipimport' imported but unused
tests/_data_ignore/test.py:22:1: F401 'importlib.machinery.all_suffixes' imported but unused
tests/_data_ignore/test.py:23:1: F401 'importlib' imported but unused
tests/_data_ignore/test.py:25:1: F401 'IPython.utils.importstring.import_item' imported but unused
tests/_data_ignore/test.py:29:1: F401 'pyflakes.test.test_imports.Test as TestImports' imported but unused
tests/_data_ignore/test.py:32:1: F401 'nose.importer.add_path' imported but unused
tests/_data_ignore/test.py:32:1: F401 'nose.importer.Importer' imported but unused
tests/_data_ignore/test.py:32:1: F401 'nose.importer.remove_path' imported but unused
tests/_data_ignore/test.py:34:1: F401 'atexit' imported but unused
tests/_data_ignore/test.py:35:1: F404 from future imports must occur at the beginning of the file
tests/_data_ignore/test.py:36:1: F401 'docopt.docopt' imported but unused
tests/_data_ignore/test.py:37:1: F401 'curses' imported but unused
tests/_data_ignore/test.py:37:14: E401 multiple imports on one line
tests/_data_ignore/test.py:38:1: F811 redefinition of unused 'logging' from line 37
tests/_data_ignore/test.py:38:1: F401 'logging' imported but unused
tests/_data_ignore/test.py:39:1: F401 'os' imported but unused
tests/_data_ignore/test.py:40:1: F811 redefinition of unused 'sqlite3' from line 37
tests/_data_ignore/test.py:40:1: F401 'sqlite3' imported but unused
tests/_data_ignore/test.py:41:1: F401 'time' imported but unused
tests/_data_ignore/test.py:42:1: F401 'sys' imported but unused
tests/_data_ignore/test.py:43:1: F401 'signal' imported but unused
tests/_data_ignore/test.py:44:1: F401 'bs4' imported but unused
tests/_data_ignore/test.py:45:1: F401 'nonexistendmodule' imported but unused
tests/_data_ignore/test.py:46:1: F401 'boto as b' imported but unused
tests/_data_ignore/test.py:46:1: F401 'peewee as p' imported but unused
tests/_data_ignore/test.py:46:17: E401 multiple imports on one line
tests/_data_ignore/test.py:48:1: F401 'flask.ext.somext' imported but unused
tests/_data_ignore/test.py:49:1: F401 'sqlalchemy.model' imported but unused
tests/_data_ignore/test.py:53:5: F401 'json' imported but unused
tests/_data_ignore/test.py:55:1: F401 'models' imported but unused
tests/_data_ignore/test.py:61:1: E305 expected 2 blank lines after class or function definition, found 1
tests/_data_ignore/test.py:61:1: E402 module level import not at top of file
tests/_data_ignore/test.py:61:1: F401 'after_method_is_valid_even_if_not_pep8' imported but unused
tests/_data_ignore/.ignore_second/ignored.py:2:1: F401 'pattern.web.plaintext' imported but unused
tests/_data_ignore/.ignore_second/ignored.py:2:1: F401 'pattern.web.Twitter' imported but unused
tests/_data_ignore/.ignore_second/ignored.py:2:43: W292 no newline at end of file
tests/_data_ignore/.ignored_dir/ignored.py:2:1: F401 'click' imported but unused
tests/_data_ignore/.ignored_dir/ignored.py:2:13: W292 no newline at end of file
(pipreqs) [doj@pan pipreqs]$ python setup.py test
running test
running egg_info
writing requirements to pipreqs.egg-info/requires.txt
writing pipreqs.egg-info/PKG-INFO
writing top-level names to pipreqs.egg-info/top_level.txt
writing dependency_links to pipreqs.egg-info/dependency_links.txt
writing entry points to pipreqs.egg-info/entry_points.txt
reading manifest file 'pipreqs.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
warning: no previously-included files matching 'pycache' found under directory ''
warning: no previously-included files matching '.py[co]' found under directory '*'
warning: no files found matching 'stdlib' under directory 'docs'
warning: no files found matching 'mapping' under directory 'docs'
writing manifest file 'pipreqs.egg-info/SOURCES.txt'
running build_ext
test_custom_pypi_server (tests.test_pipreqs.TestPipreqs) ... ok
test_deduplicate_dependencies (tests.test_pipreqs.TestPipreqs) ... ok
test_get_all_imports (tests.test_pipreqs.TestPipreqs) ... ok
test_get_import_name_without_alias (tests.test_pipreqs.TestPipreqs) ... ok
test_get_imports_info (tests.test_pipreqs.TestPipreqs) ... ok
test_get_use_local_only (tests.test_pipreqs.TestPipreqs) ... ok
test_ignored_directory (tests.test_pipreqs.TestPipreqs) ... ok
test_init (tests.test_pipreqs.TestPipreqs) ... ok
test_init_local_only (tests.test_pipreqs.TestPipreqs) ... ok
test_init_overwrite (tests.test_pipreqs.TestPipreqs) ... WARNING:root:Requirements.txt already exists, use --force to overwrite it
ok
test_init_savepath (tests.test_pipreqs.TestPipreqs) ... ok
test_invalid_python (tests.test_pipreqs.TestPipreqs) ... ERROR:root:Failed on file: /home/doj/pipreqs/tests/_invalid_data/invalid.py
ok

Ran 12 tests in 15.563s

OK
(pipreqs) [doj@pan pipreqs]$ pip freeze
appdirs==1.4.3
configparser==3.5.0
docopt==0.6.2
enum34==1.1.6
flake8==3.3.0
mccabe==0.6.1
packaging==16.8
-e git+https://github.com/bndr/pipreqs.git@714393fcea65fa91cc593d8967ddd408bb5ad9e8#egg=pipreqs
pluggy==0.4.0
py==1.4.32
pycodestyle==2.3.1
pyflakes==1.5.0
pyparsing==2.2.0
requests==2.13.0
six==1.10.0
tox==2.6.0
virtualenv==15.1.0
yarg==0.1.9
(pipreqs) [doj@pan pipreqs]$

Pipreqs 0.3.8, Windows 10, my file example/main.py imports "bottle".

"""
Evil docstring
"""

print "Not a comment"
import bottle

"""
print "A comment"
import peewee
"""

Running:

> python example/main.py
Not a comment

> pipreqs --debug example
DEBUG: Found packages: set([])
DEBUG: Found imports:
DEBUG: Getting packages information from Local/PyPI
DEBUG: Writing 0 requirements:  to example\requirements.txt
INFO: Successfully saved requirements file in example\requirements.txt

Removal of the starting comment fixes the problem (bottle found).

Windows 7, pipreqs 0.3.9, file to process (example/main.py):

import logging
logging.basicConfig()
from pydal import DAL, Field # some comment
from transitions import Machine # some comment

print "Hello"

Running:

> pipreqs --savepath req-pipreqs.txt --debug example

DEBUG: Found packages: set(['logging'])
DEBUG: Found imports:
DEBUG: Getting packages information from Local/PyPI
DEBUG: Writing 0 requirements:  to req-pipreqs.txt
INFO: Successfully saved requirements file in req-pipreqs.txt

Logging is not saved into req-pipreqs.txt which is fine because it's a standard library. Imports of pydal and transitions are not taken into account which is an error.

Removing the second line (or commenting out), so "logging.basicConfig()" causes pydal and transitions to be recognised correctly so it's a source parsing issue.

Seems to be related to the fix of #35 but also may be a slightly different scenario.

Really useful tool thanks.

One slight issue. I am using fabric3 and importing fabric and fabric.api. When I run pipreqs it correctly identifies fabric3 as required but also incorrectly identifies fabric which is the python2 version. If I then run pip install -r requirements.txt it installs both fabric3 and the python2 fabric which then creates errors.

When we need a compatibility for legacy codes, we cannot always uses latest version for package.
I think utilizing these information also can help to solve #5.

When importing certain libraries that include other recursive dependencies, these are not imported.

For example, if my script is:

import sklearn

print("Hello")

requirements.txt will include:
sklearn = 0.18

Which is insufficient since, in this case, numpy and Scipy are also required.

Would be possible to amend these kind of issues?

Will this ever support Jupyter notebooks?

exception:

Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/2.7/bin/pipreqs", line 11, in <module>
    sys.exit(main())
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/pipreqs/pipreqs.py", line 396, in main
    init(args)
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/pipreqs/pipreqs.py", line 341, in init
    extra_ignore_dirs=extra_ignore_dirs)
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/pipreqs/pipreqs.py", line 91, in get_all_imports
    raise exc
  File "<unknown>", line 31
    return {
         ^
SyntaxError: invalid syntax

file:

# coding: utf-8

import os
import pandas as pd

def main():
	projectdir = 'a/path'
	datadir = os.path.join(os.getenv('asdev'),'projects/artifacts/', projectdir)
	datafile = os.path.join(datadir, 'a/file/name')
	data = pd.read_csv(datafile)
	data = data.values.tolist()
	indexed_data = {}
	
	for index in range(len(data[0])):
		if pd.isnull(data[0][index]):
			c1 = None
		else:
			c1 = data[0][index]
		indexed_data[(c1, data[1][index])] = index
	unique_0s = {}
	unique_1s = {}
	for k in indexed_data.iterkeys():
		if k[0] not in unique_0s: unique_0s[k[0]] = 0
		if k[1] not in unique_1s: unique_1s[k[1]] = 0
		unique_0s[k[0]] += 1
		unique_1s[k[1]] += 1
	meta_fields = [k[1] for k in indexed_data.iterkeys() if pd.isnull(k[0])]
	field_not_meta = [k for k in unique_1s.iterkeys() if k not in meta_fields]
	component_ids = {k[0] for k in indexed_data.iterkeys() if pd.isnull(k[0])==False}
	
	def create_meta(row):
		sn = row[indexed_data[(None, 'Serial Number')]
		return {
			'gbx_sn*': row[indexed_data[(None, 'Serial Number')]]
		}
	
	win_code_map = {}
	win_code_map['150'] = 'HSS-BRG-GS-IB'
	win_code_map['151'] = 'HSS-BRG-RS'
	win_code_map['152'] = 'HSS-BRG-GS-OB'
	win_code_map['255'] = 'LSS-BRG-GS'
	win_code_map['256'] = 'LSS-BRG-RS'
	win_code_map['350'] = 'LSIS-BRG-GS'
	win_code_map['351'] = 'LSIS-BRG-RS'
	win_code_map['357'] = ''
	win_code_map['450'] = 'HSIS-BRG-GS-IB'
	win_code_map['451'] = 'HSIS-BRG-RS'
	win_code_map['452'] = 'HSIS-BRG-GS-OB'
	win_code_map['465'] = ''
	win_code_map['248-1 '] = 'LSPS-BRG'
	win_code_map['248-2'] = 'LSPS-BRG'
	win_code_map['248-3'] = 'LSPS-BRG'
	
	clean = []
	for row in data[2:]:
		for idx in component_ids:
			if win_code_map[idx] == '': continue
			seed = create_meta(row)
			if pd.isnull(seed['gbx_sn*']): continue
			seed['position_id'] = win_code_map[idx]
			for field in field_not_meta:
				seed[field] = row[indexed_data[idx, field]]
			clean.append(seed)
	
	clean_out = pd.DataFrame(clean)
	clean_out.to_csv('../datasets/another/file/name.ext')

if __name__ == '__main__':
	main()

The Unix philosphy is to hide interpreter choice inside an executable file (ie magic or #!). It needn't have a file extension. If you do this, you can swap implementations say, between sh and py and C and not change the user's view.

So if you have a directory containing a perfectly cromulent py program,

#!/usr/bin/python
import requests
print requests.get("http://ifconfig.co/ip").text

pipreqs will not find requests. Is this intentional? Why not have a "--examine-all" switch or make such the default?

after installing pipreqs from source onto a machine, I noticed:

Traceback (most recent call last):
File "/usr/lib/app/bin/pipreqs", line 9, in
load_entry_point('pipreqs==0.4.2', 'console_scripts', 'pipreqs')()
File "/usr/lib/app/lib/python2.7/site-packages/pipreqs-0.4.2-py2.7.egg/pipreqs/pipreqs.py", line 253, in main
init(args)
File "/usr/lib/app/lib/python2.7/site-packages/pipreqs-0.4.2-py2.7.egg/pipreqs/pipreqs.py", line 211, in init
extra_ignore_dirs = extra_ignore_dirs)
File "/usr/lib/app/lib/python2.7/site-packages/pipreqs-0.4.2-py2.7.egg/pipreqs/pipreqs.py", line 100, in get_all_imports
with open(join("stdlib"), "r") as f:
IOError: [Errno 2] No such file or directory: '/usr/lib/app/lib/python2.7/site-packages/pipreqs-0.4.2-py2.7.egg/pipreqs/stdlib'

I didn't get this error when using the wheel file, and couldn't figure out how stdlib and mapping were created. After hunting around, I noticed that earlier releases of pipreqs include the files, as well as the autogenerated tarballs from github. It appears that the 0.4.2 tarball on PyPI with md5=a5c865b9400645abc0330b64ef2f0f16 is the only archive missing the files.

pipreqs excludes the names of all directories below the target path and all .py files (without the .py) from the list of imports it reports. Sometimes this can exclude actual external imports. For example, you might have a package hierarchy like:

my_awesome_parser/
  __init__.py
  module.py
  util/
    __init__.py
    yaml.py

and my_awesome_parser.module might import yaml as provided by PyYaml but the yaml.py in my_awesome_parser/util will cause pipreqs to exclude yaml from its report.

It would require added complexity to handle this case, and I think you could make an argument that a project where this happens is poorly structured, so the best way to address this might be to put a note in the documentation explaining the current behavior. It took me a bit of debugging to figure out why PyYaml was not being reported as a requirement for a project I was working with (in my case, it was actually a directory of config files named "yaml" inside of the package, not something intended to be imported). Otherwise there would need to be a mechanism for telling pipreqs which directories and .py files were valid top-level imports.

Hey,

does pipreqs somehow support cleaning up a requirements.txt, or at least telling you which dependencies can be deleted? That would be very useful for any longer-running project. And given what pipres currently does, it may already do that?

Thanks for the work!

Cheers,
murat

When I'm trying to run pipreqs I receive this error.

Mac OS X, Python 2.7.11

MacBook-Pro-User-2:edwin-bot igor$ pipreqs ~/work/edwin/repos/edwin-bot/
ERROR: Failed on file: /Users/igor/work/edwin/repos/edwin-bot/core/event_log.py
Traceback (most recent call last):
File "/usr/local/bin/pipreqs", line 11, in
sys.exit(main())
File "/Library/Python/2.7/site-packages/pipreqs/pipreqs.py", line 258, in main
init(args)
File "/Library/Python/2.7/site-packages/pipreqs/pipreqs.py", line 216, in init
extra_ignore_dirs = extra_ignore_dirs)
File "/Library/Python/2.7/site-packages/pipreqs/pipreqs.py", line 85, in get_all_imports
raise exc
File "", line 6
class AbstractLogDriver(object, metaclass=abc.ABCMeta):
^
SyntaxError: invalid syntax

Usually you want such a tool to just print, then you redirect it to the file you want. Since we can't do that without breaking compatibilities without previous versions, how about adding a --print option ?

This will allow you to easily pipe the output, grep it, etc. Or just inspect the content before saving it somewhere.

First of all, thank you to the authors for writing pipreqs! I used this successfully for a project.

Unfortunately, it had one problem. For some reason, despite the fact that my project has never used "jenkins 1.0.2", it listed this as being apart of my project. I do use 'python_jenkins' 0.4.15. I think this is caused by both of them being imported with "import jenkins" and pipreqs not knowing how to handle that situation

It's a small bug, but one that can seriously mess up the output.

Currently I cannot use pipreqs other than with locals settings as I am behind a poxy server, spelling intentional.

But pip allows working with --proxy= when using from the command line.

If this was exposed as a command line option then the flexibility would be there for pipreqs.

I usually install multiple packages under development in a virtualenv using python setup.py develop. As a result, I get a <package>.egg-info directory (which contains top_level.txt etc.).

In presence of such packages, the requirements.txt file created by pipreqs contains entries like this:

mypackage.egg==info
otherpackage.egg==info

Instead, either the version from setup.py should be used, or maybe the version should be omitted altogether.

(see also #33 which was about checking local versions of packages)

I pip install a package directly from github (it's instructions state so):
https://github.com/perrygeo/jenks

when I run pipreqs on the resulting project, the resulting requirements.txt file has a line
jenks.egg==info

Shouldn't it be:
-e git+https://github.com/perrygeo/jenks.git@197130a220b5b2e1b8cbc3d1ad70211e6441030f#egg=jenks

I see in #59 that it was stated that concurrent.futures should not be reported because it is in the standard library. For Python 2, this is not the case. It has to be installed as futures. Whether this special case should be addressed depends on what level of support for Python 2 is wanted now.

get_locally_installed_packages() ingnores the packages:

• a package zipped
• a package has the top_level.txt in EGG-INFO folder

Is there any reason for this? it causes test_pipreqs.py failures. below is my environment

• virtualenv site-packages

Directory of D:\projects\pipreqs\venv\Lib\site-packages

2015-10-20  오전 11:12    <DIR>          .
2015-10-20  오전 11:12    <DIR>          ..
2015-10-20  오전 11:00            22,394 docopt-0.6.2-py3.5.egg
2015-10-20  오전 11:01               286 easy-install.pth
2015-10-20  오전 11:06               126 easy_install.py
2015-10-20  오전 11:00    <DIR>          pip
2015-10-20  오전 11:00    <DIR>          pip-7.1.2.dist-info
2015-10-20  오전 11:00                22 pipreqs.egg-link
2015-10-20  오전 11:06    <DIR>          pkg_resources
2015-10-20  오전 11:01    <DIR>          requests-2.8.1-py3.5.egg
2015-10-20  오전 11:06    <DIR>          setuptools
2015-10-20  오전 11:06    <DIR>          setuptools-18.4.dist-info
2015-10-20  오전 11:12    <DIR>          wheel
2015-10-20  오전 11:12    <DIR>          wheel-0.23.0.dist-info
2015-10-20  오전 11:00            35,026 yarg-0.1.9-py3.5.egg
2015-10-20  오전 11:06    <DIR>          _markerlib
2015-10-20  오전 11:06    <DIR>          __pycache__

• localy installed packages

Python 3.5.0 (v3.5.0:374f501f4567, Sep 13 2015, 02:27:37) [MSC v.1900 64 bit (AMD64)] on win32
>>> import wheel
>>> import _markerlib
>>> import easy_install
>>> import pkg_resources
>>> import setuptools
>>> import requests
>>> import pip
>>> import docopt
>>> import yarg

• get_locally_installed_packages() returns (without requests, docopt, yarg)

{
'pip': {'name': 'pip', 'version': '7.1.2'}, 
'pipreqs': {'name': 'pipreqs.egg', 'version': 'info'}, 
'pkg_resources': {'name': 'setuptools', 'version': '18.4'}, 
'easy_install': {'name': 'setuptools', 'version': '18.4'}, 
'_markerlib': {'name': 'setuptools', 'version': '18.4'}, 
'setuptools': {'name': 'setuptools', 'version': '18.4'}, 
'numpy': {'name': 'numpy', 'version': '1.10.0b1'}, 
'wheel': {'name': 'wheel', 'version': '0.23.0'}
}

I have another project where I use

from sklearn import preprocessing

from sklearn import cross_validation

from sklearn.cross_validation import StratifiedShuffleSplit

However, pipreqs does not include scikit-learn in the requirements file.

Hi,
I have tried various [options] in the pipreqs command, but haven't yet properly able to create a requirements.txt file, if the Libraries for the project are placed in some local directory, instead of Python Package directory.

For example, while working on Google Appengine projects, it's necessary to put all required third-party libraries in a local folder inside project root, and then change the Python env during runtime.

I wanted to create requirements.txt for such kind of project environments.
Is it possible using pipreqs?

Here's the error

Traceback (most recent call last):
  File "/usr/local/bin/pipreqs", line 11, in <module>
    sys.exit(main())
  File "/usr/local/lib/python2.7/dist-packages/pipreqs/pipreqs.py", line 281, in main
    init(args)
  File "/usr/local/lib/python2.7/dist-packages/pipreqs/pipreqs.py", line 235, in init
    extra_ignore_dirs=extra_ignore_dirs)
  File "/usr/local/lib/python2.7/dist-packages/pipreqs/pipreqs.py", line 86, in get_all_imports
    raise exc
  File "<unknown>", line 22
    async def concat_async(async_gen):
            ^
SyntaxError: invalid syntax

I just tried using pipreqs, but it seems useless for my workflow heavily relying on Jupyter notebooks.

Will Jupyter notebooks be supported in the future?

Hi, thanks for working on this.
pipreqs 0.4.6 appears to miss some imports, including but not limited to the following:

(ve) [tdonahue@timdev backend]$pipreqs --print .
easysnmp==0.2.4
portal==0.3.1
psycopg2==2.6.2
pymodbus==1.2.0
pyodbc==3.1.1
pytz==2016.10
redis==2.10.5
urllib3==1.19.1
INFO: Successfully output requirements

(ve) [tdonahue@timdev backend]$!find
find . -type f -name "*.py" | xargs grep futures
./modbus_collector.py:from concurrent.futures import as_completed
./meter_collector.py:from concurrent.futures import as_completed
./rest_collector.py:from concurrent.futures import as_completed
./vfd_collector.py:from concurrent.futures import as_completed
./collection_main.py:from concurrent.futures import ThreadPoolExecutor

(ve) [tdonahue@timdev backend]$pipreqs --version
0.4.6

(ve) [tdonahue@timdev backend]$python --version
Python 2.7.9

I understand concurrent.futures is a back-ported Python3 feature. If pipreqs is Python version-aware that might be good to know. The overall goal is to split requirements files, which 'pip freeze' doesn't seem to support.

Thanks for any help.

Hello!

C:\Users\ipetrash\Desktop\PyScripts\hideme.ru-proxy-list>pipreqs C:\Users\ipetrash\Desktop\PyScripts\hideme.ru-proxy-list
Traceback (most recent call last):
  File "C:\Python34\lib\runpy.py", line 170, in _run_module_as_main
    "__main__", mod_spec)
  File "C:\Python34\lib\runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "C:\Python34\Scripts\pipreqs.exe\__main__.py", line 9, in <module>
  File "C:\Python34\lib\site-packages\pipreqs\pipreqs.py", line 178, in main
    init(args)
  File "C:\Python34\lib\site-packages\pipreqs\pipreqs.py", line 157, in init
    local = get_import_local(candidates)
  File "C:\Python34\lib\site-packages\pipreqs\pipreqs.py", line 112, in get_import_local
    local = get_locally_installed_packages()
  File "C:\Python34\lib\site-packages\pipreqs\pipreqs.py", line 105, in get_locally_installed_packages
    'version': package[1].replace(".dist", ""),
IndexError: list index out of range

Used for: https://github.com/gil9red/hideme.ru-proxy-list

Sincerely yours,
gil9red

I just came across a problem when trying to use multiple imports in the same line with defined aliases, like import requests as R, bs4 as BS, dns.

As I see it the problem lies in the fact that you're only stripping the aliases from the import name when you don't find a , on the line.

First of all, this is really useful tool, and thanks for writing it!

In the README, the example output looks like

wheel==0.23.0
Yarg==0.1.9
docopt==0.6.2

whereas the actual output would be

wheel == 0.23.0
Yarg == 0.1.9
docopt == 0.6.2

(with spaces around the ==). Are the extra spaces intentional? Should the README be updated? All of my existing requirements.txt files use the format without spaces, so it would be great if there was a way to omit the spaces.

I installed pipreqs using pip install pipreqs within my local virtualenv and then executed it, got this error:

~/code % pipreqs web
Looking for imports
Traceback (most recent call last):
  File "/home/satiani/code/web/venv/bin/pipreqs", line 8, in <module>
    load_entry_point('pipreqs==0.1.8', 'console_scripts', 'pipreqs')()
  File "/home/satiani/code/web/venv/local/lib/python2.7/site-packages/pipreqs/pipreqs.py", line 109, in main
    init(args)
  File "/home/satiani/code/web/venv/local/lib/python2.7/site-packages/pipreqs/pipreqs.py", line 92, in init
    imports = get_all_imports(args['<path>'])
  File "/home/satiani/code/web/venv/local/lib/python2.7/site-packages/pipreqs/pipreqs.py", line 62, in get_all_imports
    with open(os.path.join(os.path.dirname(__file__), "stdlib"), "r") as f:
IOError: [Errno 2] No such file or directory: '/home/satiani/code/web/venv/local/lib/python2.7/site-packages/pipreqs/stdlib'
zsh: exit 1     pipreqs web

Using the lastest pipreqs:

$ pipreqs --version
xstat 0.1

which should be 0.2.8.

In this line, you write down that the version info is xstat 0.1, while you bump pipreqs version only in setup.py

Here is my pip freeze (including pipreqs)

Django==1.9.2
django-environ==0.4.0
docopt==0.6.2
pipreqs==0.4.2
psycopg2==2.6.1
requests==2.9.1
six==1.10.0
wheel==0.29.0
yarg==0.1.9

Here is the generated requirements.txt

Django == 1.9.2
django_environ == 0.4.0
environ == 1.0

Seems to have added an environ package and not included psycopg2.

As a separate note, if I uninstall pipreqs it does not remove the other dependency packages that it installed on it's own. Perhaps a different method of dependency management?

A freshly installed pipreqs (in a venv) crashes on pylint source (in a venv in my case)

Offending line is: https://bitbucket.org/logilab/pylint/src/cc5990f6a12a467d09567bfc1b210ba999f6cb13/pylint/test/functional/unused_import.py?at=default#cl-3

~/src/c3 $ pipreqs .
Traceback (most recent call last):
  File "/home/koert/src/c3/.ve2/bin/pipreqs", line 11, in <module>
    sys.exit(main())
  File "/home/koert/src/c3/.ve2/local/lib/python2.7/site-packages/pipreqs/pipreqs.py", line 167, in main
    init(args)
  File "/home/koert/src/c3/.ve2/local/lib/python2.7/site-packages/pipreqs/pipreqs.py", line 139, in init
    candidates = get_all_imports(args['<path>'])
  File "/home/koert/src/c3/.ve2/local/lib/python2.7/site-packages/pipreqs/pipreqs.py", line 48, in get_all_imports
    res = map(get_name_without_alias, item.split(","))
  File "/home/koert/src/c3/.ve2/local/lib/python2.7/site-packages/pipreqs/pipreqs.py", line 130, in get_name_without_alias
    name = REGEXP[0].match(name.strip()).groups(0)[0]
AttributeError: 'NoneType' object has no attribute 'groups'
~/src/c3 $ pip freeze
docopt==0.6.2
pipreqs==0.2.4
requests==2.7.0
yarg==0.1.9

I develop this project: https://github.com/tuwien-musicir/rp_extract

As you can see from the imports it uses a lot of libraries (numpy, scipy, matplotlib, argparse, unicsv, cPickle, sklearn, ...) - many of which are missed by pipreqs:

pipreqs --debug --force .
DEBUG: Found packages: set(['argparse', 'cPickle', 'uuid', 'tempfile', 'numpy', 'pylab', 'matplotlib', 'wave', 'sys', 'subprocess', 'os', 'sklearn'])
DEBUG: Found imports: matplotlib, numpy, matplotlib, scikit_learn
DEBUG: Getting packages information from Local/PyPI
INFO: Starting new HTTPS connection (1): pypi.python.org
DEBUG: Setting read timeout to None
DEBUG: "GET /pypi/numpy/json HTTP/1.1" 200 65657
INFO: Starting new HTTPS connection (1): pypi.python.org
DEBUG: Setting read timeout to None
DEBUG: "GET /pypi/scikit_learn/json HTTP/1.1" 301 0
DEBUG: Setting read timeout to None
DEBUG: "GET /pypi/scikit-learn/json HTTP/1.1" 200 102410
DEBUG: Writing 4 requirements: matplotlib, matplotlib, numpy, scikit_learn to ./requirements.txt
INFO: Successfully saved requirements file in ./requirements.txt
musimap@asus:~/rp_extract$ cat requirements.txt
matplotlib == 1.3.1.egg
matplotlib == 1.3.1.egg
numpy == 1.10.1
scikit_learn == 0.17

Why does it miss many of the library requirements?
(some of them are even mentioned in the debug output, some are not, e.g. scipy, unicsv)
(and matplotlib on the other hand is put twice into the requirements.txt)

I'd like if there is input param that makes requirements not like:
aaa==0.0.1
bbb==1.1.0
ccc==2.0.0

but like:
aaa>=0.0.1,<1.0.0
bbb>=1.1.0,<2.0.0
ccc>=2.0.0,<3.0.0

Currently pipreqs.py has an ignore list of: line 35 of pipreqs.p
ignore_dirs = [".git", "__pycache__", "env"]
which is fine if you are running git but users of SVN & Mercurial will either end up with very slow performance or other problems.

Please consider at least:
ignore_dirs = [".hg", ".svn", ".git", "__pycache__", "env"]

Traceback (most recent call last):
File "/usr/bin/pipreqs", line 11, in
sys.exit(main())
File "/usr/lib/python2.7/site-packages/pipreqs/pipreqs.py", line 158, in main
init(args)
File "/usr/lib/python2.7/site-packages/pipreqs/pipreqs.py", line 139, in init
imports_local = get_import_local(imports)
File "/usr/lib/python2.7/site-packages/pipreqs/pipreqs.py", line 120, in get_import_local
local = get_locally_installed_packages()
File "/usr/lib/python2.7/site-packages/pipreqs/pipreqs.py", line 110, in get_locally_installed_packages
'Could not determine import name for package ' + package_import)
TypeError: cannot concatenate 'str' and 'list' objects

Ubuntu 16.10 x86_64. Python 2.7.13. pipreqs 0.4.6

$ mktmpenv -i pipreqs
$ git clone https://github.com/pallets/flask
$ pipreqs flask
$ cat flask/requirements.txt

setuptools==34.3.2
setuptools==34.3.2
blinker==1.4
click==6.7
greenlet==0.4.12
itsdangerous==0.24
Jinja2==2.9.5
Pygments==2.2.0
pytest==3.0.7
Sphinx==1.5.3
Werkzeug==0.12.1

same with --use-local

setuptools==34.3.2
setuptools==34.3.2

After installing pipreqs and running it in the current directory, an error warns that the file is mixing tabs and spaces:

$ pip install pipreqs

Collecting pipreqs
  Using cached pipreqs-0.4.6-py2.py3-none-any.whl
Collecting docopt (from pipreqs)
Collecting yarg (from pipreqs)
  Using cached yarg-0.1.9-py2.py3-none-any.whl
Requirement already satisfied (use --upgrade to upgrade): requests in /Applications/anaconda/envs/call-data/lib/python3.5/site-packages (from yarg->pipreqs)
Installing collected packages: docopt, yarg, pipreqs
Successfully installed docopt-0.6.2 pipreqs-0.4.6 yarg-0.1.9

You are using pip version 8.1.2, however version 9.0.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.


$ pipreqs .

ERROR: Failed on file: ./CallScore/classes/text_model.py
Traceback (most recent call last):
  File "/Applications/anaconda/envs/call-data/bin/pipreqs", line 11, in <module>
    sys.exit(main())
  File "/Applications/anaconda/envs/call-data/lib/python3.5/site-packages/pipreqs/pipreqs.py", line 271, in main
    init(args)
  File "/Applications/anaconda/envs/call-data/lib/python3.5/site-packages/pipreqs/pipreqs.py", line 225, in init
    extra_ignore_dirs = extra_ignore_dirs)
  File "/Applications/anaconda/envs/call-data/lib/python3.5/site-packages/pipreqs/pipreqs.py", line 86, in get_all_imports
    raise exc
  File "/Applications/anaconda/envs/call-data/lib/python3.5/site-packages/pipreqs/pipreqs.py", line 72, in get_all_imports
    tree = ast.parse(contents)
  File "/Applications/anaconda/envs/call-data/lib/python3.5/ast.py", line 35, in parse
    return compile(source, filename, mode, PyCF_ONLY_AST)
  File "<unknown>", line 431
    recall_divisor = positives[category]+false_negatives[category]
                                                                 ^
TabError: inconsistent use of tabs and spaces in indentation

And when running it in a different directory (another downloaded GitHub repo), it throws an error due to invalid syntax:

$ pipreqs .
ERROR: Failed on file: ./<filename>.py
Traceback (most recent call last):
  File "/Applications/anaconda/envs/get-requirements/bin/pipreqs", line 11, in <module>
    sys.exit(main())
  File "/Applications/anaconda/envs/get-requirements/lib/python3.5/site-packages/pipreqs/pipreqs.py", line 271, in main
    init(args)
  File "/Applications/anaconda/envs/get-requirements/lib/python3.5/site-packages/pipreqs/pipreqs.py", line 225, in init
    extra_ignore_dirs = extra_ignore_dirs)
  File "/Applications/anaconda/envs/get-requirements/lib/python3.5/site-packages/pipreqs/pipreqs.py", line 86, in get_all_imports
    raise exc
  File "/Applications/anaconda/envs/get-requirements/lib/python3.5/site-packages/pipreqs/pipreqs.py", line 72, in get_all_imports
    tree = ast.parse(contents)
  File "/Applications/anaconda/envs/get-requirements/lib/python3.5/ast.py", line 35, in parse
    return compile(source, filename, mode, PyCF_ONLY_AST)
  File "<unknown>", line 10
    print 'You must provide an host and index to backfill (ex: east_qa v5.cdr.201608). Exiting...'
                                                                                                 ^
SyntaxError: invalid syntax

This might be the fault of the Python scripts that pipreqs is analyzing, if those scripts are badly written, but I would like to troubleshoot and make sure I know what the issue is.

I have a project where I use pycryptodomex, but pipreqs don't identify it as a dependency

I usually have throwaway/experimental scripts I'm working on for my projects so I want to create a list of dependencies for files that are being tracked only. Something like this:

$ pipreqs --from-these-files-only <(git ls-files)

e.g. fails if you do this

 pipreqs . --ignore dist build

It seems pipreqs can't find correct package name for zope