Package: mysqlclient
Reason for failure: missing build dependencies libmysqlclient-dev (jessie) or libmariadbclient-dev (stretch)

Package: lambda_packages
Reason for failure: missing build dependency pandoc

The build output recommends:

sudo apt-get install pandoc

I haven't tested it yet but I will when I get chance, and see if building the wheel works after that.

I have seen a few users have issues with installations due to this package not being available.

I should aim to provide developer setup instructions so people can git clone this repo, run a set of commands and have a working copy of piwheels ready to use for testing, initially on a Pi, later on any platform.

It would be handy if we could manually add wheels for certain packages, so that we can provide wheels for packages without an sdist, but we have a wheel available, like @samjabrahams's tensorflow: https://github.com/samjabrahams/tensorflow-on-raspberry-pi

From #43:

Well ... sort of. A postgres interface compatible with SQLAlchemy is required but not specifically psycopg2. For example, I tend to test with pg8000 (which I rather like as its pure python and therefore works on basically everything). The default postgres driver for SQLAlchemy is indeed psycopg2 and therefore we should probably add it as to the deployment instructions just for the sake of simplicity for users, but I wouldn't add it as a direct dependency in the master option of setup.py because it's not mandatory and does have specific requirements which may not be met on certain platforms.

I'm not able to connect to the repository: www.piwheels.hostedpi.com. It was working fine about several minutes ago, but then I'm not able to connect to it anymore. Could anyone help with this? Thank you so much!!

As in #38 there's bound to be more instances where people require us to remove packages from piwheels (consider the scenario where someone deletes a release from PyPI for some legal reason only to discover we built it within a minute of it being registered / uploaded and are still serving it).

Setting a package (or a version of a package) to skip is trivial (just set the relevant skip field in the database ... although it should probably be a text field rather than bool so we can track a reason for skipping it). It's also easy to remove the builds (and files and build logs) from the database. However, removing the files and replacing them with a blank index (to avoid the issues in pypa/pip#4580) is more tricky. Might be easiest to add something to MrChase's protocol for this? (it's sort of an associated behaviour to manual import of packages).

On the human side of things we might need a policy for removal of packages (consider the case where someone requests removal of a competitor's package ... probably paranoid thinking on my part, but it's a thought nonetheless). In the case of #38 I simply verified that the reporter of the ticket was a major contributor to the project on GitHub which wasn't hard, but what do we do about non-github projects? I suppose it's not a huge deal if we do erroneously remove stuff (after all, we can rebuild them trivially). Still, might be worth a FAQ entry or something ("we'll only delete stuff if we can easily verify you're associated with the project").

Hi,

Big fan of this project, as someone who regularly needs to build pip packages in a Pi environment.

I took a look at your build logs for those packages, and confirmed (on a docker image - https://hub.docker.com/r/resin/rpi-raspbian/) that if you add the packages libffi-dev and zlib1g-dev, cffi and pynacl will be installed correctly, and if you further add libxml2-dev and libxslt1-dev, lxml will be installed correctly.

We should provide an HTML page for every package, intended for users to browse packages on the piwheels website, akin to the browsable package index on PyPI: e.g. https://pypi.python.org/pypi/numpy

We would provide package pages at /piwheels/<package> (unless anyone has a better suggestion).

I imagine the process to generate pages would be alongside where we currently generate the links page in /simple/<package>.

Based on a template (using the homepage style), this could include a list of versions, and maybe something like checkboxes for which builds we have (python version / etc). Bit tricky due to some being "none", others being py3, others being cp34m.

I have started work on the template and I will try to get a basic version working...

Related to #21, here's a new overview for the homepage:

Python Wheels for the Raspberry Pi

piwheels is a Python package repository providing ARM platform wheels (pre-compiled binary Python packages) specifically for the Raspberry Pi. Packages are natively compiled on Raspberry Pi 3 hardware using the Mythic Beasts Pi cloud.

Packages: 105,885
Versions: 766, 450

Configuration

Raspbian Stretch includes configuration for pip to use piwheels by default. If you're using an alternate distribution, you can use piwheels by placing the following lines in /etc/pip.conf:

[global]
extra-index-url=https://www.piwheels.hostedpi.com/simple

That's it! With this simple configuration change, pip should now search piwheels for built distributions of packages and use them in preference to building from source. You can test this out by creating a virtual environment and attempting to install numpy which usually involves a fairly lengthy compilation:

$ sudo apt install virtualenv python3-virtualenv -y
$ virtualenv -p /usr/bin/python3 testpip
$ source testpip/bin/activate
(testpip) $ pip install numpy
...
(testpip) $ deactivate
$ rm -rf testpip/

Support

Wheels provided support all Raspberry Pi models (Pi 3, Pi 2, Pi 1 and Pi Zero). Wheels are provided for Python 3.4 and 3.5. We plan to add support for Python 3.6 and 2.7.

About

piwheels is a community project by Ben Nuttall and Dave Jones. Hosting kindly provided by Mythic Beasts. See the project source code on GitHub and read more about the service in our FAQs.

This morning, through an error on my own behalf, I realised a project to pip... guizero 0.4.2 Immediately noticing my mistake I removed the release from pypi. Unfortunately the release is now on piwheels, so when people install they get 0.4.2 not 0.4.1!

www.piwheels.hostedpi.com seems to be down.
Confirmed with http://downforeveryoneorjustme.com/www.piwheels.hostedpi.com and Pingdom.

Package: psycopg2
Reason for failure: missing build dependencies libpq-dev.

We should log package searches from pip and manual (wget) file downloads as well as actual pip installs. If we log the user agent too we can determine how many downloads of each package there were, and how many were pip installs vs regular downloads.

Also, when you pip install gpiozero, pip looks at both pypi and piwheels, and as it finds wheels in both locations, it opts for downloading from pypi, so we don't register a download, and hence, packages like this don't feature in our package search on the website.

Not sure this is the best place for this, but it seems like the website http://www.piwheels.hostedpi.com/ is having either server or SSL issues.

I noticed this when trying to install some packages:

pi@raspberrypi:~ $ pip3 install requests
Collecting requests
Exception:
Traceback (most recent call last):
  File "/usr/share/python-wheels/urllib3-1.19.1-py2.py3-none-any.whl/urllib3/contrib/pyopenssl.py", line 417, in wrap_socket
    cnx.do_handshake()
  File "/usr/lib/python3/dist-packages/OpenSSL/SSL.py", line 1426, in do_handshake
    self._raise_ssl_error(self._ssl, result)
  File "/usr/lib/python3/dist-packages/OpenSSL/SSL.py", line 1149, in _raise_ssl_error
    raise WantReadError()
OpenSSL.SSL.WantReadError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/share/python-wheels/urllib3-1.19.1-py2.py3-none-any.whl/urllib3/connectionpool.py", line 350, in _make_request
    self._validate_conn(conn)
  File "/usr/share/python-wheels/urllib3-1.19.1-py2.py3-none-any.whl/urllib3/connectionpool.py", line 837, in _validate_conn
    conn.connect()
  File "/usr/share/python-wheels/urllib3-1.19.1-py2.py3-none-any.whl/urllib3/connection.py", line 323, in connect
    ssl_context=context)
  File "/usr/share/python-wheels/urllib3-1.19.1-py2.py3-none-any.whl/urllib3/util/ssl_.py", line 324, in ssl_wrap_socket
    return context.wrap_socket(sock, server_hostname=server_hostname)
  File "/usr/share/python-wheels/urllib3-1.19.1-py2.py3-none-any.whl/urllib3/contrib/pyopenssl.py", line 421, in wrap_socket
    raise timeout('select timed out')
socket.timeout: select timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/share/python-wheels/urllib3-1.19.1-py2.py3-none-any.whl/urllib3/connectionpool.py", line 594, in urlopen
    chunked=chunked)
  File "/usr/share/python-wheels/urllib3-1.19.1-py2.py3-none-any.whl/urllib3/connectionpool.py", line 353, in _make_request
    self._raise_timeout(err=e, url=url, timeout_value=conn.timeout)
  File "/usr/share/python-wheels/urllib3-1.19.1-py2.py3-none-any.whl/urllib3/connectionpool.py", line 313, in _raise_timeout
    raise ReadTimeoutError(self, url, "Read timed out. (read timeout=%s)" % timeout_value)
requests.packages.urllib3.exceptions.ReadTimeoutError: HTTPSConnectionPool(host='www.piwheels.hostedpi.com', port=443): Read timed out. (read timeout=15)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/pip/basecommand.py", line 215, in main
    status = self.run(options, args)
  File "/usr/lib/python3/dist-packages/pip/commands/install.py", line 353, in run
    wb.build(autobuilding=True)
  File "/usr/lib/python3/dist-packages/pip/wheel.py", line 749, in build
    self.requirement_set.prepare_files(self.finder)
  File "/usr/lib/python3/dist-packages/pip/req/req_set.py", line 380, in prepare_files
    ignore_dependencies=self.ignore_dependencies))
  File "/usr/lib/python3/dist-packages/pip/req/req_set.py", line 554, in _prepare_file
    require_hashes
  File "/usr/lib/python3/dist-packages/pip/req/req_install.py", line 278, in populate_link
    self.link = finder.find_requirement(self, upgrade)
  File "/usr/lib/python3/dist-packages/pip/index.py", line 465, in find_requirement
    all_candidates = self.find_all_candidates(req.name)
  File "/usr/lib/python3/dist-packages/pip/index.py", line 423, in find_all_candidates
    for page in self._get_pages(url_locations, project_name):
  File "/usr/lib/python3/dist-packages/pip/index.py", line 568, in _get_pages
    page = self._get_page(location)
  File "/usr/lib/python3/dist-packages/pip/index.py", line 683, in _get_page
    return HTMLPage.get_page(link, session=self.session)
  File "/usr/lib/python3/dist-packages/pip/index.py", line 792, in get_page
    "Cache-Control": "max-age=600",
  File "/usr/share/python-wheels/requests-2.12.4-py2.py3-none-any.whl/requests/sessions.py", line 501, in get
    return self.request('GET', url, **kwargs)
  File "/usr/lib/python3/dist-packages/pip/download.py", line 386, in request
    return super(PipSession, self).request(method, url, *args, **kwargs)
  File "/usr/share/python-wheels/requests-2.12.4-py2.py3-none-any.whl/requests/sessions.py", line 488, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/share/python-wheels/requests-2.12.4-py2.py3-none-any.whl/requests/sessions.py", line 609, in send
    r = adapter.send(request, **kwargs)
  File "/usr/share/python-wheels/CacheControl-0.11.7-py2.py3-none-any.whl/cachecontrol/adapter.py", line 47, in send
    resp = super(CacheControlAdapter, self).send(request, **kw)
  File "/usr/share/python-wheels/requests-2.12.4-py2.py3-none-any.whl/requests/adapters.py", line 423, in send
    timeout=timeout
  File "/usr/share/python-wheels/urllib3-1.19.1-py2.py3-none-any.whl/urllib3/connectionpool.py", line 643, in urlopen
    _stacktrace=sys.exc_info()[2])
  File "/usr/share/python-wheels/urllib3-1.19.1-py2.py3-none-any.whl/urllib3/util/retry.py", line 315, in increment
    total -= 1
TypeError: unsupported operand type(s) for -=: 'Retry' and 'int'

We need to find a way of providing instructions to configure pip to use piwheels as an additional index in a way that works for both use cases: globally and in a virtualenv.

The user guide suggests:

Per user:
On Unix the default configuration file is: $HOME/.config/pip/pip.conf

Inside a virtualenv:
On Unix and macOS the file is $VIRTUAL_ENV/pip.conf

Site-wide:
On Unix the file may be located in /etc/pip.conf

There is an issue with the numpy package while it's being imported in python, in a conda environment.
I have described the issue on conda's repository, so there's no need in repeating myself: conda/conda#6423

What I've noticed is that after I build the wheel from PyPi, the error described on the aforementioned issue disappears. I'm running all this on a Jessie Lite + Raspberry Pi 3.

It might help, so here's the wheel I've built which works on my machine:

numpy-1.13.3-cp34-cp34m-linux_armv7l.zip

Let's add the 3.6 ABI to the database and fire up some Stretch builders with Python 3.6 installed, once the 3.5 builds (#15) are complete.

Let's add the 2.7 ABI to the database and fire up some Stretch builders, once the 3.5 (#15) and 3.6 (#27) are complete, and rebuild everything for Python 2.7.

Package: pybluez
Reason for failure: missing build dependencies libbluetooth-dev .

I'd like to monitor usage of piwheels, either by post-processing Apache logs, or ideally, adding an Apache hook so the logs get sent into a database. It would be really cool if we could catch every web request and log it onto the main piwheels database.

The Apache logs include entries like this:

[31/Jul/2017:11:17:33 +0100] "GET /numpy/numpy-1.13.1-cp34-cp34m-linux_armv7l.whl HTTP/1.1" 200 8988260 "-" "pip/1.5.6 CPython/3.4.2 Linux/4.4.48-v7+"

Which indicates a pip install numpy. Note the user agent is pip.

It would be good to monitor stats on:

• Downloads (installs)
• Download attempts (can we see if a pip install attempts to browse for a package that we don't have)
• Regular web browser access to the homepage and package pages

If it was related to the piwheels database then we can run queries joining on the packages and package_versions tables. Could be handy!

I do love what you are doing with PiWheels but I would really like to blacklist one my packages from being built and provided by it.
For one, mine are pure python so there's absolutely no need.
Second, one of my packages BiblioPixelAnimations is a little weird in the way it builds. The version is just a timestamp of when my travis instance builds the package. But by PiWheels rebuilding that the version gets changed. And from what I can tell, the latest version it has is even a very old version of the code...the very old Python 2.7 only version. Can't tell you why but I know it is. All of my users on the Pi have suddenly begun complaining that their code is crashing and I've noticed that all the complaints are coming from those who have the package installed from PiWheels.
In the future maybe they can be added back but right now I have an immediate need for them to be removed ASAP so that my users can get back up and running and take the time to triage why there was a problem later.
Thanks

A few people have asked if we're using the compiler switches which turn on the neon optimisations, apparently rapidly increasing performance on Pi 2 and 3.

If it's possible, and seems to be worth doing, we would still build without the flags, to provide a Pi 1-compatible wheel, but build again with the flags on, and provide that as the true ARMv7 wheel. I guess we would only rebuild the packages with a not-none ABI tag.

Assuming there's no compatibility issue between Pi 2 and 3, this might be worthwhile. Though it does add further complexity to the system :)

Not a priority, but worth looking into at a later date.

In the mean time, if anyone can provide instructions on building wheels with neon, that would be useful. For example:

pip3 wheel numpy

outputs a wheel file (after 20 mins) on a Pi 3. If the compiler flags can be added to this command, and produce a faster version, that would be awesome.

I just noticed that the new numpy 1.14.0 failed to build on piwheels, and looking at the output saved, it was reported as "no distribution available". I looked at PyPI and realised the numpy team had published wheels 2 days before publishing the sdist, and looking at the dates, it's obvious we picked it up too quickly and there was no sdist there to build from.

We will need to get the cloud gazer to notice file type uploads and not attempt to build a package until sdist is published.

When installing something like opencv from piwheels, it doesn't work until the user dependencies are satisfied - not pip dependencies as these are obviously covered by pip itself, but apt dependencies.

It would be good if we were able to log these in the database (could just be a text field with html - or even just assume it's all commands and stick the whole thing in a <pre>) and render them on the project pages (coming in #33). We'd only do it for a select few, and allow additions via github issue, no need to do more than that.

• Database change
• Algorithm to calculate dependencies
• Enhance importer to permit manual dependency specification
• Inclusion in web pages

I noticed that piw-master running on the piwheels master Pi fell over at some point over Christmas, due to a 504 Gateway Timeout on pypi.python.org.

We should catch this ProtocolError exception and enter a retry loop rather than throwing a fatal error.

All connection attempts to https://www.piwheels.org or https://www.piwheels.hostedpi.com are timing out as of right now.

In the new separate-tasks branch (currently in production) shutdown sometimes works and sometimes fails (and the slave shutdown code is entirely missing). Ought to fix this if only to make maintenance a little easier.

Looks like warehouse tweaks the PyPI XML-RPC API, we'd better check and fix for it as they're moving from legacy to warehouse in the next month (and already started directing a proportion of traffic to it).

As the title says. Is there any way that would be possible, on your end? We'd like to be able to use that server as part of our build process (and in doing so, help this project troubleshoot packages that are not building), but not at the cost of having to pull from a non-https source.

Running pip3 -v install rpi.gpio searches the https://pypi.python.org/simple/rpi-gpio/ url (and subsequently the https://www.piwheels.hostedpi.com/simple/rpi-gpio/ url when this awesome project is used).

Looking at your index, rpi.gpio IS indexed, but it resides in https://www.piwheels.hostedpi.com/simple/rpi.gpio/ (with . instead of -).

The problem is magnified by this bug here, that makes pip sometimes fail completely if it's not in the provided index :(.

Firstly, thanks for the great work on piwheels! It's greatly appreciated. ❤️

pycurl is currently missing on piwheels. There is nothing at https://www.piwheels.hostedpi.com/simple/pycurl/

I noticed it because I tried to install something with a dependency on pycurl using pip and piwheels. Because there seems to be nothing there it doesn't even try to build from source.

I couldn't find a link or anything regarding logs from the piwheels cluster to determine what's going wrong, so I assume only you guys have access to that.

It would be nice if we implemented a heartbeat signal of some sort, so that we could monitor whether piwheels was running (without logging in and seeing the master output).

It could be something simple, like a timestamp in an HTML comment at the end of the index page, or actually visible in the footer, or even something more detailed, covering all the different parts, to show which bits were active.

A feature-complete piwheels should:

• be able to build ARM platform wheels which work on Raspberry Pi
• be able to host wheels on a web server which are installable with pip
• be able to log the status and output of each build
• maintain an updated complete list of packages on PyPI, along with version numbers
• be able to rebuild packages when new versions are released
• be able to build ARMv7 wheels
• be able to provide ARMv6 wheels
• be able to build wheels for multiple Python versions
• be able to publicly provide output from failed build attempts
• be autonomous
• be resilient to failure

Hey Ben,

Curious if you or anyone else has been able to run piwheels builds inside Docker containers on a Pi. No luck here:

$ docker run --rm -it python:3.5 bash

root@8cabc40c29db:/# pip3 install numpy -i https://www.piwheels.hostedpi.com/simple
Collecting numpy
  Downloading https://www.piwheels.hostedpi.com/simple/numpy/numpy-1.14.0-cp35-cp35m-linux_armv7l.whl (6.3MB)
    100% |████████████████████████████████| 6.3MB 17kB/s
Installing collected packages: numpy
Successfully installed numpy-1.14.0

root@8cabc40c29db:/# python
iPython 3.5.5 (default, Feb 16 2018, 00:40:25)
[GCC 4.9.2] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import numpy
Traceback (most recent call last):
  File "/usr/local/lib/python3.5/site-packages/numpy/core/__init__.py", line 16, in <module>
    from . import multiarray
ImportError: cannot import name 'multiarray'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.5/site-packages/numpy/__init__.py", line 142, in <module>
    from . import add_newdocs
  File "/usr/local/lib/python3.5/site-packages/numpy/add_newdocs.py", line 13, in <module>
    from numpy.lib import add_newdoc
  File "/usr/local/lib/python3.5/site-packages/numpy/lib/__init__.py", line 8, in <module>
    from .type_check import *
  File "/usr/local/lib/python3.5/site-packages/numpy/lib/type_check.py", line 11, in <module>
    import numpy.core.numeric as _nx
  File "/usr/local/lib/python3.5/site-packages/numpy/core/__init__.py", line 26, in <module>
    raise ImportError(msg)
ImportError:
Importing the multiarray numpy extension module failed.  Most
likely you are trying to import a failed build of numpy.
If you're working with a numpy git repo, try `git clean -xdf` (removes all
files not under version control).  Otherwise reinstall numpy.

Original error was: cannot import name 'multiarray'

Cheers!

Currently we only rebuild packages successfully built for 3.4 for 3.5, but that's excluding any packages which depend on Python 3.5 features. I can't think of another way around it, but I guess we'll have to re-attempt all failed builds for Python 3.5 (and 3.6).

This means the current build queue logic (multiply ABIs and successful builds) means we also need to include failed builds. We don't need to re-attempt successful builds with an ABI of none, but we need to rebuild those with a lower ABI (e.g. cp34m) and re-attempt failed builds, in case they are dependent on the current higher ABI (e.g. cp3.535m).

A non-trivial build. No sdist is available on PyPI.

For the time being, I have manually imported some @samjabrahams's builds from github https://github.com/samjabrahams/tensorflow-on-raspberry-pi/releases

Sam provides build instructions here: https://github.com/samjabrahams/tensorflow-on-raspberry-pi/blob/master/GUIDE.md

I'll have a go, see how I get on :)

isup.me/piwheels.org
isitdownrightnow.com/piwheels.org.html

Is there any mirror for piwheels.org ?

We should document which apt packages we definitely don't want installed on the slaves, like pyqt which makes subsequent matplotlib builds dependent on that also being installed for the user (but works fine if built without it, regardless of whether it's installed for the user).

It doesn't need to be part of the installation process, just documented somewhere. For example when I added those packages to fix pygame #52 and pycurl #49 I could have referred to the blacklist before adding them (risking breaking built packages in future).

Hi!

I am working on a project which we deploy to Raspberry Pis.
I develop it in Ubuntu and install it in a Raspberry Pi for production.
Recently I migrated it to pipenv, with its Pipfile and Pipfile.lock.

I ran into a problem when installing it in the Raspberry Pi:

• If I leave the Pipfile.lock that was created in Ubuntu, it will complain that the hashes don't match. I expected pipenv to know that I am not running pipenv in Ubuntu anymore, the packages should be different, hence the hashes are different, however for the pure Python packages that's not the case, I guess.

As a solution I remove the Pipfile.lock file, the pip cache folder, and the /etc/pip.conf file to use PyPI instead, but then I needed some apt libraries to compile a couple of pip dependencies. Which is my current solution.

Thanks!

It would be nice if we could provide some information via an API - nothing fancy, just some static JSON generated from the DB.

Either one big JSON blob containing everything, or perhaps a few more distinct sets of data.

I'm thinking:

• List of packages (could be limited to those we have available, or contain metadata including success / build output
• List of versions per package
• Build output

Publish the piwheels package on PyPI

We need to create a mechanism for re-queuing builds which are tagged cp34m to be re-build for Python 3.5. It's easy enough to build them using a Pi running Raspbian Stretch, but the DB needs to a) distinguish these builds from others, and b) be able to create a queue for packages which need rebuilding for different Python versions.

Now Pygame is available on PyPI, we are able to build it on piwheels. However, all builds have failed, and it boils down to:

Unable to run "sdl-config". Please make sure a development version of SDL is installed.

If anyone knows what the build requirements for Debian are for Pygame, please post here, we'll install and kick off the builds again.

So far I've been running through the complete set of packages on PyPI, building ARMv7 wheels for Python 3 on a Raspberry Pi 3. It's partly a test, partly a proof of concept, but the output will form the basis of the hosted service.

Before I progress to ARMv6, Python 2 and other varieties, I just wanted to see how long it takes to build every package, what proportion of them succeed, and what the common reasons for failure are.

Current status (last updated 23/4):

• Packages processed: 106,574 / 106,574 (100%)
• Successfully built: 81,180 (76%)
• Failed: 25, 394 (24%)
• Total time spent building: 237 hours
• Total disk usage from wheels: 17GB
• System disk usage: 67% (of 32GB card)

See live data at http://piwheels.bennuttall.com/

Hey,

It seems like, for some reason, the lxml package is missing. The folder of the package exists, but it's empty: https://www.piwheels.hostedpi.com/simple/lxml/

looking at the deploy_slave_{jessie,stretch}.sh files, you have apt dependencies on python{,3}-lxml as well as python{,3}-yaml, python{,3}-pil and a few more, which would make you think will bring all the required dependencies in order to build those packages, but in reality, they bring the non-dev packages, thus building might fail.

Thank you again for a wonderful project :)

Package: systemd-python

I'm having issues keeping track of whether or not I have the latest version of a package.

• I keep a record of what I consider the latest version number of a package in the database, and update this regularly (from PyPI's JSON API)
• I keep a record of the version of the package when I built it
• Therefore, I can deduce which packages require an update

This sounds trivial. However, the way I record the version at the point of build is to extract it from the output wheel filename. I've now come to realise that this is often inconsistent with the version recorded in PyPI.

For example:

1. Look up the version of doxybuild on PyPI (JSON). It's 0.1.1-alpha.
2. Run pip wheel doxybuild and extract the version from the filename. It's 0.1.1_alpha.

Again, this seems trivial, as the hyphen has been replaced with an underscore. Another example:

1. Look up the version of Hooke on PyPI (JSON). It's 1.0.0.alpha (Ninken).
2. Run pip wheel Hooke and extract the version from the filename. It's 1.0.0.alpha._Ninken_.

Again, seems trivial, but there are lots of similar examples, and some are more different than the above examples:

package | strict
version | 0.1alpha
version | 0.1a0
-[ RECORD 2 ]----------------------------
package | strippers.wikipedia
version | 0.3
version | 0.3a0
-[ RECORD 3 ]----------------------------
package | sudokulib
version | 0.6a
version | 0.6a0
-[ RECORD 4 ]----------------------------
package | sy85tools
version | 0.2.2a
version | 0.2.2a0
-[ RECORD 5 ]----------------------------
package | synchrophasor
version | 1.0.0-alpha
version | 1.0.0a0
-[ RECORD 6 ]----------------------------
package | testsuite-prettyprint-outcomes
version | 0.1.0-alpha.1
version | 0.1.0a1
-[ RECORD 7 ]----------------------------
package | testsuite-prettyprint-traceback
version | 0.1.0-alpha.2
version | 0.1.0a2
-[ RECORD 8 ]----------------------------
package | timepy
version | 0.1a
version | 0.1a0
-[ RECORD 9 ]----------------------------
package | tissu
version | 0.4-alpha
version | 0.4a0
-[ RECORD 10 ]---------------------------
package | tracticketpost
version | 0.2a6dev
version | 0.2a6.dev0

I need to be able to tell when a new version has been released by the version number changing. If it's expected to be different, then how can I tell?

Does anyone have any ideas for how I can/should record the version to be more consistent?

I've written some FAQs to go on piwheels master alongside the index file:

How do I install packages from piwheels?

Raspbian Stretch:

Simply add the following to /etc/pip.conf:

[global]
extra-index-url=https://www.piwheels.hostedpi.com/simple

Raspbian Jessie:

Upgrade pip to 9.0.1 using sudo pip3 install pip –upgrade and follow the Raspbian Stretch instructions.

Other operating systems:

Install pip 9.0.1, and follow the Raspbian Stretch instructions.

How do I install a package from piwheels within a virtual environment?

If you had to upgrade pip to 9.0.1 you may also need to upgrade pip within the virtual environment with pip install pip –upgrade. If not (e.g. in Raspbian Stretch), pip should honour your /etc/pip.config file and use piwheels as configured.

How do I install a package from piwheels without changing my pip configuration?

To explicitly use piwheels, simply use the -i or –index-url option, for example:

sudo pip3 install numpy -i https://www.piwheels.hostedpi.com/simple

or

sudo pip3 install numpy –index-url https://www.piwheels.hostedpi.com/simple

Alternatively, you can use piwheels as an additional index, which will allow pip to fall back to PyPI if the requested package (or one of its dependencies) is not available on piwheels, by using the –extra-index-url option, for example:

sudo pip3 install numpy -extra-index-url https://www.piwheels.hostedpi.com/simple

How do I install a package from PyPI (not from piwheels)?

If your pip is configured to use piwheels, but you want to use PyPI instead, you can either remove the extra-index-url configuration in /etc/pip.conf to avoid piwheels permanently, or you can explicitly request PyPI in the installation command, for example:

sudo pip3 install numpy -i https://pypi.python.org/simple

How much time does piwheels save?

Time saved depends on the package requested, and the Raspberry Pi model used. Some packages usually require significant build time, which is avoided by using piwheels. Many packages are usually quite quick to install anyway, and only a small amount of time is saved.

I maintain a package on PyPI. How do I publish it on piwheels?

You don’t need to do anything. All package releases on PyPI are automatically processed by the piwheels project.

Why is a package I maintain is not available on piwheels?

Some packages fail to build, for various reasons. If you would like to investigate, try building the package yourself on a Raspberry Pi 3 with a Raspbian Lite image by running pip3 wheel <package>. If you find you are able to successfully build the package (for example, by installing some build dependencies), please open an issue on github and we’ll try to resolve it.

Why are some versions of a package available on piwheels and other versions are not?

piwheels attempts to build every release of every package on PyPI. Some build attempts are unsuccessful, and these versions will not be available on piwheels. While some versions of a given package may be available, others may not have been built successfully. If you maintain the package or otherwise have a solution for the build, please open an issue on github and we’ll try to resolve it.

How long does it take for a new release to become available on piwheels?

???

Does piwheels use cross-compilation?

No. All wheels are built natively on the Raspbian operating system on Raspberry Pi 3 hardware.

Which Raspberry Pi models are supported by packages provided by piwheels?

All curent Raspberry Pi models (as of September 2017) are supported:

• Raspberry Pi 3
• Raspberry Pi 2
• Raspberry Pi 1
• Raspberry Pi Zero

Please also observe answers to questions relating to operating systems and Python versions.

I installed a package from piwheels and it doesn’t work as expected

Some packages build differently depending on the tools available. For example, if the package matplotlib is built with pyqt installed, the wheel will depend on the user also having pyqt installed. If it is built without pyqt, the user can use matplotlib without pyqt.

If you find an issue with a package installed from piwheels, please submit an issue on github.

Which Python versions does piwheels support?

Currently, piwheels supports Python 3 only. While most packages available will work in any version of Python 3, but some packages are only compatible with the minor version with which they were built.

Currently supported:

• Python 3.4

Planned support:

• Python 3.5
• Python 3.6
• Python 2.7

Where can I find the piwheels source code?

The source code for the piwheels project can be found on GitHub at github.com/bennuttall/piwheels.

What infrastructure does the piwheels project use?

The piwheels project is hosted on a cluster of Raspberry Pi 3s provided by Mythic Beasts.

The original build run was completed by a total of 20 Raspberry Pis, and now a smaller cluster (fewer than 5 Pis) remains active, continuing to build new package releases.

The package I installed took a long time to install, despite being available on piwheels.

First of all: check your pip configuration and confirm that piwheels is in fact being used. You should see downloading from https://www.piwheels.hostedpi.com.

If you are definitely using piwheels, it could be that while the package you installed was available on piwheels, perhaps one of its dependencies is not available, and requires downloading from PyPI and building from source.

Why are the wheel files are tagged with ARMv6 and ARMv7? What about ARMv8?

While the hardware in Raspberry Pi 1 and Zero is ARMv6, Pi 2 is ARMv7, Pi 3 is ARMv8, the Raspbian operating system images provided on raspberrypi.org are made to support all three architectures (and are compatible with all Raspberry Pi products to date) by operating in ARMv6 userland.

However, wheels built on a Raspberry Pi 2 or 3 are tagged armv7l. Since wheels built on a Raspberry Pi 3 will work on a Pi 2, 1 or Zero, we simply provide Pi 3-built wheels renamed armv6l.

Will wheels provided on piwheels work on other ARM platforms?

Some wheels may work on other ARM platforms, but we can’t guarantee this. Pure Python wheels will certainly work, but there’s a much smaller speed increase installing from wheels over source distribution. ARM platform wheels on piwheels are tagged armv6l and armv7l but actually both contain ARMv6-compatible code, which may not work on true ARMv7 platforms.

Does piwheels work for the Raspberry Pi Desktop operating system for x86?

See the following answer.

Can I install packages from piwheels on my PC/Mac to speed up installations?

Pure Python wheels will certainly work, but there’s a much smaller speed increase installing from wheels over source distribution. Packages requiring compilation are built for a specific architecture, so they will not be useful on non-ARM platforms and will be ignored. Assuming your pip configuration is set to fall back to PyPI, installation will continue from there, which may involve building from source.

Are you going to upload all the wheels to PyPI?

No - we can’t. Only package maintainers can upload files to PyPI.

I’m a package maintainer. Can I upload an ARM wheel to PyPI?

The current PyPI website does not allow ARM wheels to be uploaded. The next-generation PyPI project (pypi.org) does allow uploading ARMv6 and ARMv7 wheels (by setting repository=https://upload.pypi.io/legacy/ for PyPI in your ~/.pypirc file) but it should be noted that wheels built on a Raspberry Pi 3, although tagged armv7l are not truly ARMv7, and may be incompatible with other ARM platforms. If your package is made exclusively for the Raspberry Pi platform, there is no harm in uploading ARM wheels to PyPI, and this will speed up the installation for users who do not have pip configured to use piwheels.

If you intend for your package to be installed by users in multiple ARM platforms, you may be best not to upload ARM wheels to PyPI and instead leave Raspberry Pi users to get a wheel from piwheels, and leave other ARM platform users to build from source.

@waveform80 can you double-check over the answers, add any you think are missing, and create another html file (static is fine, we can push updates easy enough)?