sganis / minirepo Goto Github PK

Create a local pypi repository to use pip off-line

License: MIT License

Python 100.00%

minirepo's Introduction

Minirepo

Create a local pypi repository to use pip off-line. (Updated to mirror python 3 packages).

Minirepo is a command-line program that downloads Python packages from pypi.python.org, so you can use pip without internet. I needed to maintain a python repository in an isolated cluster environment, and after trying several tools to mirror pypi index, I dedided to make my own tool.

Some mirroring tools such us bandersnatch didn't meet my requirements, because I wanted to do a selective mirror, only downlowing all sources for python 2.7, for example. Bandersnatch gets the full content, about 140GB at the time of my first version of minirepo.

Then I was inspired by basket, which is almost what I wanted, but you need to specify the list of packages to download. I ended up using the json API to get the packages that I needed, and then calling basket to download or update the packages. In the end, that approach was slow and buggy, so I wrote this small program to do what I just needed. Now, my minirepo folder has about 12GB with only the latest packages, and it takes about 20 minutes to mirror.

Installation

Use pip

The easiest way to install it is to use pip:

$ pip install minirepo

Or download and install

Download the package file from https://pypi.python.org/pypi/minirepo, or the latest development version from https://github.com/sganis/minirepo, then:

$ tar xvzf minirepo-1.0.3.tar.gz
$ cd minirepo-1.0.3
$ python setup.py install

You can also use git:

$ git clone https://github.com/sganis/minirepo.git
$ cd minirepo
$ python setup.py install

Usage

# run it from the command line:
$ minirepo

# or run the python script if you didn't install it:
$ ./minirepo.py

The firt time it's executed, the program will ask you for the local repository path, which defaults to ~/minirepo in Linux. A json configuration file is created and saved as ~/.minirepo, that you can edit to your preferences. This configuration file looks like this:

{
    "processes"       : 20, 
    "repository"      : "/home/user/minirepo",
    "package_types"   : ["bdist_egg","bdist_wheel","sdist"], 
    "extensions"      : ["bz2","egg","gz","tgz","whl","zip"], 
    "python_versions" : ["2.7","any","cp27","py2","py27","source"]
        "platforms"   : ["linux"]
}

Minirepo uses packages_types, extensions, platforms and python_versions as filters. I was analysing the full list of packages available in pypi.python.org, and it looks that all the options are something like the list below, you can try any other option. For me, I was only interested in python 2.7 packages, sources, wheels and eegs distributions, and some extensions.

PYTHON_VERSIONS = [
    '2', '2.2', '2.3', '2.4', '2.5', '2.6', '2.7', '2.7.6', '3.0', '3.1', 
    '3.2', '3.3', '3.4', '3.5', 'any', 'cp25', 'cp26', 'cp27', 'cp31', 
    'cp32', 'cp33', 'cp34', 'cp35', 'py2', 'py2.py3', 'py26', 'py27', 
    'py3', 'py32, py33, py34', 'py33', 'py34', 'python', 'source'
]

PACKAGE_TYPES = [
    'bdist_dmg', 'bdist_dumb', 'bdist_egg', 'bdist_msi', 'bdist_rpm', 
    'bdist_wheel', 'bdist_wininst', 'sdist'
]

EXTENSIONS = [
    'bz2', 'deb', 'dmg', 'egg', 'exe', 'gz', 'msi', 'rpm', 'tgz', 'whl', 'zip'
]

PLATFORMS = [
    'linux', 'win32', 'win_amd64', 'macosx'
]

Use pip without internet

$ pip install --no-index --find-links=/home/user/minirepo <package-name>

I prefer to setup environment variables in my profile so I don't have to provide extra command line arguments.

# save these 2 variable in your profile 
$ export PIP_NO_INDEX=true
$ export PIP_FIND_LINKS=/home/user/minirepo

# then run pip as usual
$ pip install <package-name>

minirepo's People

Contributors

Stargazers

Watchers

Forkers

pythonpeixun shustinm greatbahram sy-base dm168168 nadeemnazeer3 niftymist abduhbm theonlynexus baas-hans bartbroere

minirepo's Issues

feature request: sync minirepo with only new/updated packages

if i re-run the minirepo command, it downloads again all pkgs. it would be nice to download only the new/updated packages (maybe keepiong mongodb/sqlite db of downloaded pkgs internally)

also to delete old/obsolete/old versioned pkgs

Skip if already in repo not skipping but downloading

Thanks for this great work, I've been running this script and the first time it downloads the packages however when running a 2nd and ongoing time it is actually downloading the packages again and not skipping them https://github.com/sganis/minirepo/blob/master/minirepo.py#L166

Typos in README

I've corrected some 2 typos in the json in a local branch. Would love to push it for review.

Fails when Installing on a Mac

I just installed the project, both times in a brand new VirtualEnv (Python 2.7) - on a Mac.

First time I used the PIP mechanism - the 2nd Time I pulled your GIT Repo

Alas - same error.... Not sure if it is not a MAC Issue... as it is occurring in the MultiProcessor calls....

The error is

/******** Minirepo ********/
extensions = [u'bz2', u'egg', u'gz', u'tgz', u'whl', u'zip']
package_types = [u'bdist_egg', u'bdist_wheel', u'sdist']
processes = 1
python_versions = [u'3.4', u'py34']
repository = /Users/tim/minirepo
Using config file /Users/tim/.minirepo
Traceback (most recent call last):
File "/Users/tim/pymirror/bin/minirepo", line 9, in
load_entry_point('minirepo==1.0.3', 'console_scripts', 'minirepo')()
File "build/bdist.macosx-10.11-x86_64/egg/minirepo.py", line 290, in main
File "/usr/local/Cellar/python/2.7.11/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/init.py", line 232, in Pool
return Pool(processes, initializer, initargs, maxtasksperchild)
File "/usr/local/Cellar/python/2.7.11/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/pool.py", line 159, in init
self._repopulate_pool()
File "/usr/local/Cellar/python/2.7.11/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/pool.py", line 214, in _repopulate_pool
for i in range(self._processes - len(self._pool)):
TypeError: unsupported operand type(s) for -: 'unicode' and 'int'

Repo does not have a license

Hi, I'd like to use minirepo in one of my projects, but I'm hesitant to do so without a license for the repo.
Would you please consider adding one? Perhaps MIT or BSD?
Thank you!

TypeError: unsupported operand type(s) for -: 'unicode' and 'int'

Running minirepo generates the following error with no human input taken.

minirepo
/******** Minirepo ********/
extensions      = [u'bz2', u'egg', u'gz', u'tgz', u'whl', u'zip']
package_types   = [u'bdist_egg', u'bdist_wheel', u'sdist']
processes       = ls -la
python_versions = [u'2.7', u'any', u'cp27', u'py2', u'py2.py3', u'py27', u'source']
repository      = pwd
Using config file /home/ec2-user/.minirepo
Traceback (most recent call last):
  File "/home/ec2-user/.local/share/virtualenvs/pypi_minirepo-r1IsK5pf/bin/minirepo", line 11, in <module>
    sys.exit(main())
  File "/home/ec2-user/.local/share/virtualenvs/pypi_minirepo-r1IsK5pf/lib/python2.7/site-packages/minirepo.py", line 290, in main
    pool = mp.Pool(PROCESSES)
  File "/usr/lib64/python2.7/multiprocessing/__init__.py", line 232, in Pool
    return Pool(processes, initializer, initargs, maxtasksperchild)
  File "/usr/lib64/python2.7/multiprocessing/pool.py", line 159, in __init__
    self._repopulate_pool()
  File "/usr/lib64/python2.7/multiprocessing/pool.py", line 213, in _repopulate_pool
    for i in range(self._processes - len(self._pool)):
TypeError: unsupported operand type(s) for -: 'unicode' and 'int'

OS Filters

When I ran minirepo it was downloading amd64 wheel files only. I expected that it would download wheel files and source files. The config I used was

{
  "processes"      : "20", 
  "package_types"  : ["bdist_wheel","sdist"], 
  "extensions"     : ["tgz","whl","gz"], 
  "python_versions": ["2.7","cp27","cp36","cp37","py2","py3","py2.py3","py27","py36","py37
  "repository"     : "/home/<username>/minirepo"
}

Now I saw in the code something about a platform filter.
"platforms" : ["linux","win32","win_amd64","macOS"],
It looks that it covers all platforms. But I wonder if there is some hardcoded filter to download amd64 only?

Because when I added to the config above the following line
"platforms" : ["linux"],
and restarted it, it took a while (i guess to process the huge packages.json file) and then it downloaded this:

2020-05-05 20:32:14,417:WARNING: Downloaded: cmappertools-1.0.24-cp27-cp27m-win_amd64.whl Ok pid:8919 1% [277/23193.0]

Before it had downloaded some 12GB of win_amd64 packages. Shouldn't if have downloaded only linux (...none-linux_arm.whl) and source files?
How can I make it download Linux and Source files?

Depency on request isn't installed automatically

Really minor issue... When installing via pip, the dependency for requests is not picked up, so an error occurs when running. Looks like the line in setup.py is commented out - assuming accidentally?

from setup.py (approx #66):

install_requires=['requests'],

dependency resolver

Hi
Do you use any dependency resolver for list of packages? I just skimmed your code and I didn't see anything that check package's dependencies.