Comments (11)
The latest release of the gguf package is from Dec 13, 2023, but the gguf source still updates frequently. There are some incompatible settings between them. For anyone experiencing this issue, try the following commands:
pip install gguf==0.6.0 "numpy<2.0" --force-reinstall
from transformers.
Hi @SunMarc, just a reminder that gguf-py has been updated to 0.9.1 recently. There might be some issues with this version. If I find anything new, I will reopen this issue.
from transformers.
The correct workaround is to replace num_blocks
in this code with -1
, but I'm not sure if this is the correct behavior.
# transformers/integrations/ggml.py
def dequantize_q6_k(data):
block_size = GGML_BLOCK_SIZES["Q6_K"]
num_blocks = len(data) // block_size
data_f16 = np.frombuffer(data, dtype=np.float16).reshape(-1, block_size // 2)
data_u8 = np.frombuffer(data, dtype=np.uint8).reshape(-1, block_size)
data_i8 = np.frombuffer(data, dtype=np.int8).reshape(-1, block_size)
scales = data_f16[:, -1].reshape(-1, 1).astype(np.float32)
from transformers.
Hi @PenutChen, thanks for the warning ! It looks like we indeed have failing tests on side. We get the same error you experienced. I will reopen the issue =)
from transformers.
downgrading to gguf==0.6.0 solved it for me. Thanks for @PenutChen hint.
from transformers.
cc @SunMarc
from transformers.
Hey @PenutChen thanks for opening the issue ! I tried your snippet on the main branch of transformers and on v4.42.3, and everything looks fine ! I suggest you to clear your cache and try it again. Also, which version of numpy are you using ? Maybe this is an issue with the 2.0 version was released recently.
from transformers.
@SunMarc Thanks for the reply! I upgraded the numpy version to 1.26.4, but I still get the same error. After checking all my dependencies, I found that my gguf
was installed from the source of the llama.cpp repo. I changed the version to the PyPI one, and it works!
from transformers.
Thanks for investigating ! Hopefully, for the next release of gguf, we won't have the issue you experienced.
from transformers.
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.
Please note that issues that do not follow the contributing guidelines are likely to be ignored.
from transformers.
solved by #32298
from transformers.
Related Issues (20)
- Out of memory when using phi3 for token classification HOT 1
- Odd output from falcon-mamba-7b on mps device HOT 2
- Clarification on saving model checkpoints
- Can MPS use FP16 when training?Why I can't? HOT 1
- redundant checkpointing in example scripts
- offload to CPU will increase memory HOT 1
- Video-LLaVA-7B-hf doesn't work (returns nonsense) HOT 3
- audio pipeline utility ffmpeg_microphone_live only works with PulseAudio in Linux (KDE Manjaro) HOT 1
- RoBERTa config defaults are inconsistent with fairseq implementation HOT 4
- NLLB Code Examples out of date? HOT 3
- Inference API takes forever and output: "Model ... is currently loading" [couldn't reopen #23793] HOT 1
- `run_summarization` likely race condition creates flaky `test_tensorflow_examples`
- Constraint.advance() docstring is slighly incorrect HOT 2
- RuntimeError: Failed to import transformers.integrations.bitsandbytes because of the following error (look up to see its traceback): cannot import name 'get_keys_to_not_convert' from 'transformers.integrations' HOT 1
- fp16 support for grounding dino HOT 1
- class Cache must not be a subclass of `torch.nn.Module`
- Can't load Llama's tokenizer with add_prefix_space=True parameter. HOT 1
- tracker: move `prepare_inputs_for_generation` into the generation mixin ๐งน HOT 2
- `eval_on_start` triggers `AttributeError` in JupyterLab HOT 1
- Mamba slow implementation datatype mismatch HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. ๐๐๐
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google โค๏ธ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from transformers.