Compiled Cublas version FileNotFound on windows about ctransformers HOT 6 CLOSED

@marella it appears that was the issue:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
NameError: name 'CUDA_PATH' is not defined

When adding the following code to my python file, it resolved the issue:

import os
cuda_path = "C:/Program Files/NVIDIA GPU Computing Toolkit/CUDA/v11.8" 
os.environ["CUDA_PATH"] = cuda_path

Interesting note -- using CPU only gives a short answer. Same question with the GPU involved is more verbose!

from ctransformers.

In the latest version 0.2.21, precompiled libraries are provided for CUDA 12, so no compilation is required and it no longer needs CUDA to be installed at system level:

pip install ctransformers ctransformers[cuda]

llm = AutoModelForCausalLM.from_pretrained("TheBloke/Llama-2-7B-GGML", gpu_layers=50)

See example in Colab.

Feel free to re-open if you are still facing an issue.

from ctransformers.

Hi, is the normal installation without CUBLAS (pip install ctransformers) working?

What is your Python version? Can you check if CUDA_PATH is set:

python3 -c 'import os; print(os.environ["CUDA_PATH"])'

from ctransformers.

Same issue here

from ctransformers.

Thanks! I think I checked path and env vars but I can't recall for sure! I'll try and provide more context

from ctransformers.

I get this error while running the colab code :
CUDA error 35 at D:\a\ctransformers\ctransformers\models\ggml\ggml-cuda.cu:5067: CUDA driver version is insufficient for CUDA runtime version
please help
Windows 11
CUDA 11.8
RTX 3090
nvidia-smi
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 522.25 Driver Version: 522.25 CUDA Version: 11.8 |
|-------------------------------+----------------------+----------------------+
| GPU Name TCC/WDDM | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|===============================+======================+======================|
| 0 NVIDIA GeForce ... WDDM | 00000000:41:00.0 On | N/A |
| 0% 41C P8 25W / 420W | 716MiB / 24576MiB | 3% Default |
| | | N/A |
+-------------------------------+----------------------+----------------------+

nvcc -V
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2021 NVIDIA Corporation
Built on Mon_Oct_11_22:11:21_Pacific_Daylight_Time_2021
Cuda compilation tools, release 11.4, V11.4.152
Build cuda_11.4.r11.4/compiler.30521435_0

from ctransformers.