Dear team, Thanks a lot for reducing the barrier of entrance to work

Thanks for pointing this out and sharing <a class="user-mention notranslate" data-hove

gptq quantization fails ModuleNotFoundError about litgpt HOT 5 CLOSED

lightning-ai commented on August 23, 2024

gptq quantization fails ModuleNotFoundError

from litgpt.

Comments (5)

dwahdany commented on August 23, 2024 1

I think this should be re-opened and closed when added to requirements.txt

from litgpt.

escorciav commented on August 23, 2024

THe ModueleNotFoundError might dissapear after running, pip install datasets. Refer to this doc for more details.

from litgpt.

escorciav commented on August 23, 2024

In case, anyone is using RTX2080, you might face the following error 😓

Starting to quantize blocks
0 attn.attn collecting stats quantizing bin /env/lib/python3.10/site-packages/bitsandbytes/libbitsandbytes_cuda118.so
time 17s quantization error 29958.6
0 attn.proj collecting stats Traceback (most recent call last):
  File "/awesome-project/lit-parrot/quantize/gptq.py", line 376, in <module>
    CLI(main)
  File "/env/lib/python3.10/site-packages/jsonargparse/cli.py", line 85, in CLI
    return _run_component(component, cfg_init)
  File "/env/lib/python3.10/site-packages/jsonargparse/cli.py", line 147, in _run_component
    return component(**cfg)
  File "/awesome-project/lit-parrot/quantize/gptq.py", line 363, in main
    llama_blockwise_quantization(model, encoded_text, device, bits=4)
  File "/env/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "/awesome-project/lit-parrot/quantize/gptq.py", line 265, in llama_blockwise_quantization
    outs[j : j + 1], _ = block(
  File "/env/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1502, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/env/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1511, in _call_impl
    return forward_call(*args, **kwargs)
  File "/awesome-project/lit-parrot/lit_parrot/model.py", line 161, in forward
    h, new_kv_cache = self.attn(n_1, rope, mask, max_seq_length, input_pos, kv_cache)
  File "/env/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1502, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/env/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1511, in _call_impl
    return forward_call(*args, **kwargs)
  File "/awesome-project/lit-parrot/lit_parrot/model.py", line 198, in forward
    qkv = self.attn(x)
  File "/env/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1502, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/env/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1511, in _call_impl
    return forward_call(*args, **kwargs)
  File "/awesome-project/lit-parrot/quantize/bnb.py", line 332, in forward
    return qlinear_4bit_weight(inp, self.quant_weight, self.scales, self.zeros)
  File "/awesome-project/lit-parrot/quantize/bnb.py", line 253, in qlinear_4bit_weight
    linear_kernel_4bit_weight[grid](
  File "/env/lib/python3.10/site-packages/triton/runtime/autotuner.py", line 97, in run
    timings = {config: self._bench(*args, config=config, **kwargs)
  File "/env/lib/python3.10/site-packages/triton/runtime/autotuner.py", line 97, in <dictcomp>
    timings = {config: self._bench(*args, config=config, **kwargs)
  File "/env/lib/python3.10/site-packages/triton/runtime/autotuner.py", line 80, in _bench
    return do_bench(kernel_call, quantiles=(0.5, 0.2, 0.8))
  File "/env/lib/python3.10/site-packages/triton/testing.py", line 44, in do_bench
    fn()
  File "/env/lib/python3.10/site-packages/triton/runtime/autotuner.py", line 78, in kernel_call
    self.fn.run(*args, num_warps=config.num_warps, num_stages=config.num_stages, **current)
  File "<string>", line 42, in linear_kernel_4bit_weight
  File "/env/lib/python3.10/site-packages/triton/compiler/compiler.py", line 465, in compile
    next_module = compile_kernel(module)
  File "/env/lib/python3.10/site-packages/triton/compiler/compiler.py", line 361, in <lambda>
    lambda src: ptx_to_cubin(src, arch))
  File "/env/lib/python3.10/site-packages/triton/compiler/compiler.py", line 160, in ptx_to_cubin
    return _triton.compile_ptx_to_cubin(ptx, ptxas, arch)
RuntimeError: Internal Triton PTX codegen error: 
ptxas /tmp/compile-ptx-src-d48630, line 83; error   : Feature '.bf16' requires .target sm_80 or higher
ptxas /tmp/compile-ptx-src-d48630, line 83; error   : Feature 'cvt with .f32.bf16' requires .target sm_80 or higher

from litgpt.

escorciav commented on August 23, 2024

Sure, I have an environment setup. Just lemme know, and I'm happy to try again 😊

BTW, I'm not sure if the issue is due to an "old" GPU (RTX2080)
I had to drop off lit-parrot as the requirements of my project changed. Namely, export any (L)LM to onnx with opset 9.
I'm working directly with transformers and optimum. If you provide support for T5, I'm happy to keep playing with LIT-:parrot:

Motivation is mentioned here

cc @rasbt

from litgpt.

rasbt commented on August 23, 2024

Thanks for pointing this out and sharing @escorciav

I am pretty certain that the RTX2080 only supports float16 but not bfloat16 training. Bfloat16 is a relatively recent feature for NVIDIA cards. The RTX 3000 series (and A100 etc.) would support it.

(The alternative here would be to run it with float16 instead of bfloat16

from litgpt.

gptq quantization fails ModuleNotFoundError about litgpt HOT 5 CLOSED

Comments (5)

Related Issues (20)

Recommend Projects

React

Vue.js

Typescript

TensorFlow

Django

Laravel

D3

Recommend Topics

javascript

web

server

Machine learning

Visualization

Game

Recommend Org

Facebook

Microsoft

Google

Alibaba

D3

Tencent