[FEAT] GPU support about docker-faster-whisper HOT 27 CLOSED

i came here to look into GPU support, and request if needed.
found I am not alone.

from docker-faster-whisper.

Thanks. Whisper-asr-webservice has added faster whisper as an alternate engine but his image doesn't have Wyoming protocol which is needed for Home Assistant's voice assistants. For someone looking for self hosted home automation the last thing one needs is to wait 3-4-5 seconds for whisper to process the voice on CPU so that the locally linked light switch turns on or off. Giving people an option to make their own decision if the compromise is worth it seems like the right way forward. As for a dedicated VM, it's not an option for me, and others probably as the thought of dedicating a full 24 GB tesla to whisper alone doesn't sit right with me. Seems like a big waste.

from docker-faster-whisper.

Cool, I'll get the PR reviewed and merged shortly. Thanks for the testing.

from docker-faster-whisper.

Try passing --gpus all as part of your run. Unfortunately my spare nvidia card is an ancient GTX 780 so it's not a good test case for trying to narrow down where the issue lies.

That did the trick. Thank you for all your hard work and time.

from docker-faster-whisper.

Thanks for opening your first issue here! Be sure to follow the relevant issue templates, or risk having this issue marked as invalid.

from docker-faster-whisper.

So I am not crazy :) I tried every possible combination of extra parameters on the docker container settings (--gpus all --runtime=nvidia, nvidia visible devices) but it still used the CPU and it's really slow. Got 2 Tesla P40 in my home server waiting to be used. So I second this request. Thanks for all your hard work.

from docker-faster-whisper.

So the short answer is that adding Nvidia runtime support will at least double the size of the image. I might look at adding a separate tag so the core image isn't burdened with it, but for example that image OP linked is an absolutely stupid 11.5Gb - at some point you might as well just run a dedicated VM for it.

from docker-faster-whisper.

Please take a look at #6 once it's finished building; the image tag will be posted to the PR to allow you to test.

from docker-faster-whisper.

Please take a look at #6 once it's finished building; the image tag will be posted to the PR to allow you to test.

At at first glance doesn't seem to work on GPUs. Nothing gets loaded into GPU ram. The new image works but still on CPU. Do I need to add any other parameters?

`docker run
-d
--name='faster-whisper'
--net='bridge'
-e TZ="Europe/Athens"
-e HOST_OS="Unraid"
-e HOST_HOSTNAME="ElderNet"
-e HOST_CONTAINERNAME="faster-whisper"
-e 'WHISPER_MODEL'='base'
-e 'WHISPER_BEAM'='1'
-e 'WHISPER_LANG'='en'
-e 'PUID'='99'
-e 'PGID'='100'
-e 'UMASK'='022'
-l net.unraid.docker.managed=dockerman
-l net.unraid.docker.icon='https://raw.githubusercontent.com/linuxserver/docker-templates/master/linuxserver.io/img/linuxserver-ls-logo.png'
-p '10300:10300/tcp'
-v '/mnt/user/appdata/faster-whisper':'/config':'rw'
--runtime=nvidia
--gpus=all 'lspipepr/faster-whisper:gpu-1.0.1-pkg-c8490d79-dev-f96940c1787b378f93093112f6bf0115e17f97fc-pr-6'
c273cb14c49a87acf11d46976b752bdf8c263387e863cb9a624c188ca18f155e

The command finished successfully!`

No errors in the logs.

from docker-faster-whisper.

Runtime needs to be set to nvidia and you need the Nvidia container toolkit installed on the host

from docker-faster-whisper.

It is:
--runtime=nvidia
I also have the Nvidia drivers plugin ( I am using Unraid - although it shouldn't matter)

I have other containers setup the same way (like Jellyfin) and they work on GPU without issues.

from docker-faster-whisper.

My mistake, I forgot to set the device in the service file, just fixing that now hopefully - the docs aren't great

from docker-faster-whisper.

New build is up, ignore the CI failure that's because the GitHub runners don't have a GPU

lspipepr/faster-whisper:gpu-1.0.1-pkg-c8490d79-dev-af5c5eb79d5e4a524997696b375db083119fcf65-pr-6

from docker-faster-whisper.

docker run
-d
--name='faster-whisper'
--net='bridge'
-e TZ="Europe/Athens"
-e HOST_OS="Unraid"
-e HOST_HOSTNAME="ElderNet"
-e HOST_CONTAINERNAME="faster-whisper"
-e 'WHISPER_MODEL'='base'
-e 'WHISPER_BEAM'='1'
-e 'WHISPER_LANG'='en'
-e 'runtime'='nvidia'
-e 'PUID'='99'
-e 'PGID'='100'
-e 'UMASK'='022'
-l net.unraid.docker.managed=dockerman
-l net.unraid.docker.icon='https://raw.githubusercontent.com/linuxserver/docker-templates/master/linuxserver.io/img/linuxserver-ls-logo.png'
-p '10300:10300/tcp'
-v '/mnt/user/appdata/faster-whisper':'/config':'rw'
--runtime=nvidia
--gpus=all 'lspipepr/faster-whisper:gpu-1.0.1-pkg-c8490d79-dev-af5c5eb79d5e4a524997696b375db083119fcf65-pr-6'

Unable to find image 'lspipepr/faster-whisper:gpu-1.0.1-pkg-c8490d79-dev-af5c5eb79d5e4a524997696b375db083119fcf65-pr-6' locally
docker: Error response from daemon: manifest for lspipepr/faster-whisper:gpu-1.0.1-pkg-c8490d79-dev-af5c5eb79d5e4a524997696b375db083119fcf65-pr-6 not found: manifest unknown: manifest unknown.
See 'docker run --help'.

The command failed.

from docker-faster-whisper.

Ugh, because the CI test failed it didn't push the image to the public registry.

I've kicked off a new build with the CI test disabled.

from docker-faster-whisper.

Image can be pulled now, however I am getting:

01:23:52 PM Traceback (most recent call last):
01:23:52 PM File "/usr/lib/python3.10/runpy.py", line 196, in _run_module_as_main
01:23:52 PM return _run_code(code, main_globals, None,
01:23:52 PM File "/usr/lib/python3.10/runpy.py", line 86, in _run_code
01:23:52 PM exec(code, run_globals)
01:23:52 PM File "/lsiopy/lib/python3.10/site-packages/wyoming_faster_whisper/main.py", line 136, in
01:23:52 PM asyncio.run(main())
01:23:52 PM File "/usr/lib/python3.10/asyncio/runners.py", line 44, in run
01:23:52 PM return loop.run_until_complete(main)
01:23:52 PM File "/usr/lib/python3.10/asyncio/base_events.py", line 649, in run_until_complete
01:23:52 PM return future.result()
01:23:52 PM File "/lsiopy/lib/python3.10/site-packages/wyoming_faster_whisper/main.py", line 112, in main
01:23:52 PM whisper_model = WhisperModel(
01:23:52 PM File "/lsiopy/lib/python3.10/site-packages/wyoming_faster_whisper/faster_whisper/transcribe.py", line 58, in init
01:23:52 PM self.model = ctranslate2.models.Whisper(
01:23:52 PM RuntimeError: CUDA failed with error CUDA driver version is insufficient for CUDA runtime version

What version of Cuda Toolkit is used? As far as I know: For 2.3 you need a 190.x driver, for 3.0 you need 195.x and for 3.1 you need 256.x (actually anything up to the next multiple of five is ok, e.g. 258.x for 3.1).
My drivers are reported as latest: latest: v535.54.03

from docker-faster-whisper.

The pip packages installed (as per the faster-whisper docs) are the latest versions of nvidia-cublas-cu11 ad nvidia-cudnn-cu11, so CUDA 11 in both cases,

I've got some hardware I can spin up later this week to be able to test locally because at the moment I'm just having to wing it based on the project docs.

from docker-faster-whisper.

I've run through a whole bunch of tests with both the CUDA 11 and 12 packages and multiple different drivers/runtimes and I can't get it to work. Everything results in the same error, even with the latest drivers and toolkit available.

from docker-faster-whisper.

i'm happy to test also if needed - got a GPU in my home server and hopefully will see some snappy responses (size of image likely wont be an issue for those with GPUs, unless its possible to use a Coral TPU?)

My Nvidia driver on Unraid is 535.104.05

from docker-faster-whisper.

I was able to run the a7597ab commit from the gpu-initial branch on my local machine using a Quadro P620 (Driver Version: 545.29.06, CUDA Version: 12.3). The trick was to install the NVIDIA Container Toolkit first and then exposing the GPU in Docker with

    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: 1
              capabilities: [gpu]

from docker-faster-whisper.

Was also able to get the gpu branch working, Gforce 1070TI and the performance is fantastic.

image: lspipepr/faster-whisper:gpu-version-1.0.1

from docker-faster-whisper.

It still doesn't work for me. Using this image: lspipepr/faster-whisper:gpu-version-1.0.1
I get:

Traceback (most recent call last):
File "/usr/lib/python3.10/runpy.py", line 196, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/usr/lib/python3.10/runpy.py", line 86, in _run_code
exec(code, run_globals)
File "/lsiopy/lib/python3.10/site-packages/wyoming_faster_whisper/main.py", line 136, in
asyncio.run(main())
File "/usr/lib/python3.10/asyncio/runners.py", line 44, in run
return loop.run_until_complete(main)
File "/usr/lib/python3.10/asyncio/base_events.py", line 649, in run_until_complete
return future.result()
File "/lsiopy/lib/python3.10/site-packages/wyoming_faster_whisper/main.py", line 112, in main
whisper_model = WhisperModel(
File "/lsiopy/lib/python3.10/site-packages/wyoming_faster_whisper/faster_whisper/transcribe.py", line 58, in init
self.model = ctranslate2.models.Whisper(
RuntimeError: CUDA failed with error CUDA driver version is insufficient for CUDA runtime version

from docker-faster-whisper.

Try passing --gpus all as part of your run. Unfortunately my spare nvidia card is an ancient GTX 780 so it's not a good test case for trying to narrow down where the issue lies.

from docker-faster-whisper.

I'm trying to get GPU version working with no luck. I'm using unRAID nvidia driver plugin with the following setup:

+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 550.40.07              Driver Version: 550.40.07      CUDA Version: 12.4     |
|-----------------------------------------+------------------------+----------------------+
|   0  NVIDIA GeForce RTX 2070        Off |   00000000:01:00.0 Off |                  N/A |
|  0%   58C    P2             65W /  185W |     665MiB /   8192MiB |      2%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+

unRAID creates the container using the following run command:

docker run
  -d
  --name='faster-whisper'
  --net='pipernet'
  --privileged=true
  -e TZ="America/New_York"
  -e HOST_OS="Unraid"
  -e HOST_CONTAINERNAME="faster-whisper"
  -e 'WHISPER_MODEL'='medium-int8'
  -e 'NVIDIA_DRIVER_CAPABILITIES'='all'
  -e 'PUID'='99'
  -e 'PGID'='100'
  -e 'UMASK'='022'
  -l net.unraid.docker.managed=dockerman
  -l net.unraid.docker.icon='https://raw.githubusercontent.com/linuxserver/docker-templates/master/linuxserver.io/img/linuxserver-ls-logo.png'
  -p '10300:10300/tcp'
  -v '/mnt/cache/appdata/faster-whisper':'/config':'rw' 'lscr.io/linuxserver/faster-whisper:gpu'
  --gpus all
  --runtime nvidia

However, the container logs just show the following:

RuntimeError: CUDA failed with error CUDA driver version is insufficient for CUDA runtime version

I've tried multiple different tags and also the lspipepr but can't seem to get it to work 🤷
Would a new image have to be created with FROM nvidia/cuda:12.4.0-base-ubuntu22.04 so the container version matches the runtime version?

from docker-faster-whisper.

You don't use the gpus flag nor have you set the Nvidia runtime correctly.

from docker-faster-whisper.

@j0nnymoe, I'm not sure what you mean, could you explain further?
I've also tried removing gpus flag and adding the =

  --runtime=nvidia

Do I need a different version of nvidia drivers? I have other containers that use my GPU (Emby, stable-diffusion, compreface, frigate) and they all seem to work as expected with those flags I typically provide.

EDIT: Ok, seems like I was adding the flag as an Post Argument not as an Extra Parameter. It works when I correct that mistake

from docker-faster-whisper.

You will need to add the runtime + the ENV's NVIDIA_VISIBLE_DEVICES & NVIDIA_DRIVER_CAPABILITIES.

If you continue to have issue, please either jump on our discord or forum as our GitHub issues isn't for general support. (ignoring the fact this is actually a closed issue anyway.)

from docker-faster-whisper.