no cuda implementation for rms-norm about candle HOT 2 CLOSED

Could you provide more details about what you ran? The following model works all well for me

cargo run --profile=release-with-debug --features cuda --example mistral -- --quantized --prompt "Hello "

The main thing that could trigger the error you're seing is the candle-nn create not having the cuda feature enabled but that should be the case if you're enabling the cuda feature flag for anything in candle-examples.

from candle.

I am not running the example, its a modified code using mistral based model for inference.

However I located the solution based on your feedback. In my Cargo.toml:

- candle-nn = { git = "https://github.com/huggingface/candle.git" }
+ candle-nn = { git = "https://github.com/huggingface/candle.git", features = ["cuda"] }

I didnt need to add any flag, and successfully loads and runs the quantized mistral based model using candle. Tnx,

Closing

from candle.