BitsNBytes 4 bit quantization error message typo and logical errors in error message handling about transformers HOT 3 OPEN

It seems like you are facing an issue with running the code snippet provided in a system where CUDA is not available, resulting in an import error related to torch.cuda. The issue stems from the logic within the Transformers library that checks for CUDA availability before enabling BitsAndBytes functionality.

Here are some steps and potential solutions to address this problem:

1. Identifying the Issue:

   • The code snippet includes logic that checks for the availability of CUDA before enabling BitsAndBytes functionality. This check is necessary to ensure that CUDA operations can be used efficiently. However, in systems without NVIDIA GPUs or proper CUDA configurations, this check can cause import errors.

2. Solutions:

   • Option 1: Error Handling:

     • Modify the code to handle the case where CUDA is not available gracefully. You can add conditional logic to bypass the CUDA check if the system does not support it.

   • Option 2: Environment Configuration:

     • Ensure that your Docker container is correctly configured to access the host system's CUDA driver. This configuration is crucial for running CUDA-dependent operations within the container.

   • Option 3: Alternative Device Configuration:

     • Consider specifying a different device for model computation, such as CPU, when CUDA is not available. This approach can prevent errors related to CUDA dependencies.

3. Further Investigation:

   • Review the logic within the Quantizer module of the Transformers library that handles CUDA availability checks. Understanding this logic may provide insights into why the conditional checks are not working as expected in your system configuration.

4. Community Support:

   • Reach out to the Transformers library maintainers, such as @SunMarc and @younesbelkada, for assistance with debugging the specific issue related to CUDA availability checks and BitsAndBytes functionality.

By exploring these solutions and seeking guidance from the Transformers community, you can work towards resolving the import error caused by the CUDA availability check when running the provided code snippet in systems without NVIDIA GPUs or CUDA support.

from transformers.

Hi @jkterry1, thanks for this detailed report ! For 3. and 4. , let me know if you want to submit a PR to fix the logger message and split into two checks ! Otherwise, I can do it ! For 1. and 5., this is indeed strange that the first cuda check passes but failed in the second check in bitsandbytes. We can pontentially remove the cuda import check in is_bitsandbytes_available but it would be better for the first check to work correctly.

from transformers.

Thank you so much!

If you'd be willing to do PRs yourself for 3 and 4, I'd be extraordinarily grateful (also note that it's likely that you'll need to make the fix in 3 to the 8 bit version of this file).

Regarding options 1 and 4, I personally believe that you should remove the cuda check from is_bitsandbytes_available so that when the function returns false it's only for the expected reason, and perform all cuda availability checks outside of this to prevent future unexpected errors.

Additionally, I think it would likely be prudent to verify that the code snippet to check for cuda described in 4 that threw a false positive for me and started me down this journey correctly throws a negative in test environments without GPUs, because it threw a false positive in the docker image I described as well as a random Arch linux laptop, suggesting something very unlikely happened to me or there's something wrong in the logic.

from transformers.