ARM64 support? about thumbor HOT 15 CLOSED

This should now be available with multiarch builds. See #103

from thumbor.

I don't have any hardware by myself to test this yet, but it looks as if the first teammates will switch to M1 based Macs in the next weeks.

So I have a bit of a personal need, and I have helped another project switch to GHA and ARM64 builds before (although that was PHP-based and I am way more familiar with that).

In short, I'll try to make a first proposal. Seeing that you are soo responsive is a great first step!

from thumbor.

Seems some work has been put into Docker Desktop on OS X: It can cross-build amd64/arm64 platforms out of the box, and also run amd64 images on M1 hardware via some kind of emulation as you noticed.

My problem is that, for reasons, I need to run an arm64 Linux virtual machine (via Vagrant) and that all development happens inside that. It seems that when I run Docker inside that VM (and not on the OS X host), I don't get that cross-platform emulation support.

I've tried cross-building a direct arm64 image on OS X following the surprisingly simple instructions in https://www.docker.com/blog/multi-arch-images/.

The arm64 version build, however, fails with:

 > [linux/arm64  5/11] RUN pip install --trusted-host None --no-cache-dir    -r /app/requirements.txt:
#20 74.41 ERROR: Could not find a version that satisfies the requirement opencv-python==4.2.0.32 (from -r /app/requirements.txt (line 23)) (from versions: 3.4.0.14, 3.4.10.37, 4.3.0.38)
#20 74.80 ERROR: No matching distribution found for opencv-python==4.2.0.32 (from -r /app/requirements.txt (line 23))
#20 76.04 WARNING: You are using pip version 20.0.2; however, version 20.3.4 is available.
#20 76.04 You should consider upgrading via the '/usr/local/bin/python -m pip install --upgrade pip' command.

Now I really don't have any reasonable experience with Python, but to me it seems there is no arm64-based image for opencv-python?

Applicable issues:

• opencv/opencv-python#366
• opencv/opencv-python#429

If I get all that right, that support has been added rather recently with OpenVC 4.5.3. That version, however, requires Python 3 and thus cannot be used before we switch to Thumbor 7. Version 4.2.0.32 that is listed in requirements.txt seems to be the latest version supporting Python 2.7 (https://pypi.org/project/opencv-python/4.2.0.32/).

Does all that make sense? If so, I guess we're stuck until Thumbor 7 is a thing and can take over (when it comes to additional plugins etc.) from Thumbor 6.

from thumbor.

@guilhermef this may interest you...

from thumbor.

Hey @mpdude nice to hear from you again :)

What would be your requirements for accepting such a PR?

In general, PRs are always welcome :) especially if they improve things. I'm not sure what's involved however, but as long as it's simple and doesn't break existing functionality, we're happy :)

Have you thought about using GitHub actions to automatically build images?

Yes, I'm already using Semaphore, but perhaps it's better to switch to GH actions. I haven't had the time to dig into it.

Please keep in mind that there's some effort to upgrade to thumbor 7.0, so any other updates might have to take the back seat until this is done...

from thumbor.

I'm using an M1 Mac, but not sure I have that much time to test things locally ... I think typical thumbor deployment is on servers and they are typically linux (x64), so it doesn't feel like a high-priority use-case, but I guess there are some ARM64 servers out there as well? I'm not against adding this support, if it's simple :) but I won't be able to help much myself unfortunately.

from thumbor.

I'm using an M1 Mac, but not sure I have that much time to test things locally

Have you ever tried running this image as-is on it? Maybe it works right away?

All I need for the time being is a development-grade Thumbor running for local use only. Although I have read that there are arm64 based AWS instances (Gravitron?) that seem to be very cheap and might make an interesting use case for image conversion as well…

First check: There is a python:2.7.18-stretch Docker image with arm64v8 support.

Will have to figure out how I can work on this without having an M1 machine myself.

Update: The answer is here: https://www.docker.com/blog/multi-arch-images/

from thumbor.

@mpdude if I run docker desktop on my mac and then run docker run -p 80:80 minimalcompact/thumbor it seems to work. I'm not sure what compatibility layers docker desktop / MacOS have in place however, so can't really tell you how "native" this is running.

If AWS provides such instances, then this should hopefully provide a way for you to test things. Good luck!

from thumbor.

This is what docker desktop shows me btw when I run it, but I'm not sure what it means... probably that it is emulated rather than running natively?

from thumbor.

@gingerlime Your assessment is correct in that, since no arm64 version of minimalcompact/thumbor exists, Docker for Mac is running it in emulation mode.

I would also like to add a bit of verbal support to the idea of being able to run Thumbor on ARM when the time comes. We run a big chunk of our stack on AWS Elastic Container Service and have recently switched almost all of our production stack over to ARM due to the 20% reduction in costs offered by their Gravitron2 chips. Our Thumbor containers are the only things left on amd64.

Given @mpdude's assessment, this task looks like one for the post Thumbor 7 / Python 3 TODO list.

from thumbor.

Hope Thumbor 7 moves forward, but there are still some issues with some of the libraries, see #103

Otherwise, take a look at imagor. It's API-compatible with Thumbor, and whilst it doesn't have full feature-parity, it's getting closer. My benchmarks also show that it's much faster... like not 20% faster, but more like 900% (of course, it really depends on the source image, operations etc).

Besides that, you can save a lot of cost with caching. This repo includes an nginx-proxy container with caching for thumbor, plus of course you can use other caching layers like CDNs. So if it's all about cost-saving, there are other options :)

from thumbor.

Following the topic.

from thumbor.

As of this commit on the thumbor 7 branch the build seems to run on ARM64.

from thumbor.

Bump

from thumbor.

As the OP, I still have a general interest in getting this solved. However, I won’t be able to come up with a solution near-term.

I had to find an improvised solution last year, which turned out to be running Thumbor on Docker Desktop in OS X, outside of the usual development machine. That solved the immediate need for a solution.

Second, switching to arm64 means switching to Thumbor 7, and before I can do that, I need to get some other Thumbor extensions in other repos compatible for that (including Python 3). That’s a bit of a challenge for me, with practically zero Python experience. Before I’ve completed that, it makes no sense for me to continue here.

Maybe as a first step, see this comment #102 (comment), read the referenced blog article, build a Thumbor 7 image on arm64 and report back if it works?

from thumbor.