gguf dequantize failed about transformers HOT 11 CLOSED

The latest release of the gguf package is from Dec 13, 2023, but the gguf source still updates frequently. There are some incompatible settings between them. For anyone experiencing this issue, try the following commands:

pip install gguf==0.6.0 "numpy<2.0" --force-reinstall

from transformers.

Hi @SunMarc, just a reminder that gguf-py has been updated to 0.9.1 recently. There might be some issues with this version. If I find anything new, I will reopen this issue.

from transformers.

The correct workaround is to replace num_blocks in this code with -1, but I'm not sure if this is the correct behavior.

# transformers/integrations/ggml.py

def dequantize_q6_k(data):
    block_size = GGML_BLOCK_SIZES["Q6_K"]
    num_blocks = len(data) // block_size

    data_f16 = np.frombuffer(data, dtype=np.float16).reshape(-1, block_size // 2)
    data_u8 = np.frombuffer(data, dtype=np.uint8).reshape(-1, block_size)
    data_i8 = np.frombuffer(data, dtype=np.int8).reshape(-1, block_size)

    scales = data_f16[:, -1].reshape(-1, 1).astype(np.float32)

from transformers.

Hi @PenutChen, thanks for the warning ! It looks like we indeed have failing tests on side. We get the same error you experienced. I will reopen the issue =)

from transformers.

downgrading to gguf==0.6.0 solved it for me. Thanks for @PenutChen hint.

from transformers.

cc @SunMarc

from transformers.

Hey @PenutChen thanks for opening the issue ! I tried your snippet on the main branch of transformers and on v4.42.3, and everything looks fine ! I suggest you to clear your cache and try it again. Also, which version of numpy are you using ? Maybe this is an issue with the 2.0 version was released recently.

from transformers.

@SunMarc Thanks for the reply! I upgraded the numpy version to 1.26.4, but I still get the same error. After checking all my dependencies, I found that my gguf was installed from the source of the llama.cpp repo. I changed the version to the PyPI one, and it works!

from transformers.

Thanks for investigating ! Hopefully, for the next release of gguf, we won't have the issue you experienced.

from transformers.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.

from transformers.

solved by #32298

from transformers.