GPU support may not be enabled about genai-stack HOT 5 CLOSED

If you are using the Linux profile, you should add this to the llm service:

deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities: [gpu]

And ensure that you have nvidia-container-toolkit installed

from genai-stack.

llama.go:333: skipping accelerated runner because num_gpu=0
genai-stack-llm-1 | 2023/10/21 03:52:30 llama.go:356: starting llama runner
genai-stack-llm-1 | 2023/10/21 03:52:30 llama.go:408: waiting for llama runner to start responding
genai-stack-llm-1 | [GIN] 2023/10/21 - 03:52:30 | 500 | 873.831145ms | 172.20.0.5 | POST "/api/generate"
genai-stack-llm-1 | 2023/10/21 03:52:30 llama.go:373: error starting llama runner: llama runner process has terminated
genai-stack-llm-1 | 2023/10/21 03:52:30 llama.go:438: llama runner stopped with error: signal: illegal instruction (core dumped)

from genai-stack.

Is that on linux? Can you share more of your OS and machine specs.
@mchiang0610

from genai-stack.

from genai-stack.

Added it to the readme.

from genai-stack.