Installing skill-date-time on Mark 1 fails because of numpy about skill-date-time HOT 8 CLOSED

If a dependency is removed I think depends on the method of installation. If it was part of the mycroft-core package I think it will be removed as part of the update (since the file belongs to the older deb package), if a skill installs a dependency I think will remain as it's outside of the control of the deb package.

happy to help :)

from skill-date-time.

Hmm, no that minor version difference doesn't seem to be the issue...possibly some other requirement that fails?

Also: Mark-1 is still at Jessie so that's correct. We added a custom built python3.7 so we could continue on a supported python version.

from skill-date-time.

Thanks @forslund. I checked again and I believe I found a solution.

The error I found seemed to be related (from the logs above) to the typing module, which is weird that it fails because it should be part of the python standard library. However, I saw in my log above references to /opt/venvs/mycroft-core/lib/python3.7/site-packages/typing.py as if the typing package was installed through pip (which should not happen on a python-3.7 venv).

And Bingo! Apparently there is a typing package installed, required by my msk:

(mycroft-core) pi@mycroft:~ $ pip show typing
Name: typing
Version: 3.7.4.3
Summary: Type Hints for Python
Home-page: https://docs.python.org/3/library/typing.html
Author: Guido van Rossum, Jukka Lehtosalo, Łukasz Langa, Ivan Levkivskyi
Author-email: [email protected]
License: PSF
Location: /opt/venvs/mycroft-core/lib/python3.7/site-packages
Requires: 
Required-by: msk

And msk (as you probably already know, I didn't 😃 ) is the mycroft-skills-kit:

(mycroft-core) pi@mycroft:~ $ pip show msk
Name: msk
Version: 0.3.15
Summary: Mycroft Skills Kit
Home-page: https://github.com/MycroftAI/mycroft-skills-kit
Author: Mycroft AI
Author-email: [email protected]
License: Apache-2.0
Location: /opt/venvs/mycroft-core/lib/python3.7/site-packages
Requires: GitPython, typing, msm, pygithub, requests, colorama
Required-by: mycroft-core

Fortunately, I've checked the msk repository and msk 0.3.16 is out there and fixes the issue (thanks to MycroftAI/mycroft-skills-kit@0c22d30 ). I could update it manually (pip install -U msk).

However, my mycroft-core has a strict dependency on msk==0.3.15. I got this warning:

mycroft-core 20.8.0 requires msk==0.3.15, but you'll have msk 0.3.16 which is incompatible.

I've checked the mycroft-core repository and on MycroftAI/mycroft-core@6cb219d the dependency on msk was updated, so on the next release the typing dependency will cease to exist. I could uninstall typing myself.

Related Question: When a package stops being required is it automatically removed from the venv?

I tried again building numpy, and I got another error, related to the uuid package.

(mycroft-core) root@mycroft:/home/pi# pip show uuid
Name: uuid
Version: 1.30
Summary: UUID object and generation functions
Home-page: http://zesty.ca/python/
Author: Ka-Ping Yee
Author-email: [email protected]
License: UNKNOWN
Location: /opt/venvs/mycroft-core/lib/python3.7/site-packages
Requires: 
Required-by: 

This seems another pip package stepping over the python standard library. I don't know how uuid got installed in the first place. I am removing it and trying again. The existence of this installed package makes me wonder if the answer to the question above is "No, packages that are not required anymore are not automatically removed".

I'm now building the numpy wheel. If it succeeds (I guess it will) I will try to install the skill again.

I guess doing a pip uninstall typing uuid in all mark 1 units after the next upgrade would make sense. I don't know if that would make sense in pycroft or other installations, nor whether to report mark1 issues like this one.

Having a mark1 pip repository with prebuilt wheels for numpy, instead of making all users build numpy would also be nice. numpy does not take as long as the mimic compilation, but it's not fast and it uses all cores, so my mycroft unit is not very responsive in the process 😆

I will close this issue, since this skill is not causing any trouble and I think the issue is solved. As far as I know you are not working for Mycroft anymore, so an extra reason for me to thank you for spending your time testing my issues.

from skill-date-time.

I know it took time to install the update when needing to install numpy but it should succeed...

If I check my Mark-1 I see this:

(mycroft-core) pi@mark_1:~ $ pip show numpy
Name: numpy
Version: 1.19.1
Summary: NumPy is the fundamental package for array computing with Python.
Home-page: https://www.numpy.org
Author: Travis E. Oliphant et al.
Author-email: None
License: BSD
Location: /opt/venvs/mycroft-core/lib/python3.7/site-packages
Requires: 
Required-by: timezonefinder

So it managed to install that version...will try to update to the exact version that fails for you

from skill-date-time.

My mark 1 ran out of memory while installing numpy and the building was killed :-(

Trying again (and leaving it for the night, using tmux) with:

pip install --install-option="--jobs=1" numpy

Oh, and based on the logs from my previous attempt I decided to apt-get install libatlas-base-dev, so the resulting numpy binaries use a good blas and lapack library and are faster.

from skill-date-time.

Is it worth us adding the libatlas-base-dev package to the dependencies anyway?

Thanks for all the digging into this too!

from skill-date-time.

@krisgesling with respect to the libatlas-base-dev package... The only usage of numpy in the Mycroft stack I am aware of is by the dependency here that takes care of "finding the timezone given these coordinates". Since this does not seem that expensive, I don't think it makes a big difference to have the atlas library or not. However, if in the future another skill uses numpy a bit more heavily, I guess having an optimized linear algebra library underneath such as libatlas will be beneficial.

I also think that installing libatlas probably reduces the compilation time of numpy, since all the linear algebra functions are already built and installed. I haven't benchmarked anything though.

Overall I think it is worth installing it.

It would be better to upgrade from Jessie to Stretch and to Buster, use the python version from Debian stable and add the https://piwheels.org pip repository, so it offers precompiled packages for all the pypi packages. This would speed up skill updates, especially when something big like numpy appears.

Piwheels precompiles packages for each Debian release, with their respective python version, so the wheels for python 3.7 do not work on Debian Jessie (since piwheels builds for Jessie-python 3.4 or Buster-3.7, but not Jessie-3.7).

Removing the typing and uuid pip packages in all mark 1 units on the next update (if they are installed) is something you should really take care of. The fact that pip packages that are not required anymore are kept installed may become a maintenance problem long term. The only easy solution I see is to re-create the venv, but that is very time consuming (and risky) on a raspberry pi, without precompiled wheels.

Technical debt on the Mark1 will eventually become a bill too large to pay. Being on oldoldstable is a concern to me, given that I don't see any migration plan from Mycroft being set in place. I understand there is a lot on your table, but in the end I will not want to have a functional but vulnerable device on my living room with a microphone attached to it.

from skill-date-time.

A clean could be performed at regular interval without too much issues. Could be made part of the major release updates.

We discussed the upgrade to buster last year but the work to make that a smooth and safe upgrade on the Mark-1 was deemed too big for the small team. Backporting python3.7 and required packages (openssl for example) was instead decided on.

The thought back then was also that the Mark-1 was closing in on End of Life...

The simplest possibility may be to create a new Mark-1 image using buster, and let it be a manual update switching the SD card...

Edit: but this is of course up to Mycroft to decide.

from skill-date-time.