Comments (5)
I think this should be re-opened and closed when added to requirements.txt
from litgpt.
THe ModueleNotFoundError might dissapear after running, pip install datasets
. Refer to this doc for more details.
from litgpt.
In case, anyone is using RTX2080, you might face the following error 😓
Starting to quantize blocks
0 attn.attn collecting stats quantizing bin /env/lib/python3.10/site-packages/bitsandbytes/libbitsandbytes_cuda118.so
time 17s quantization error 29958.6
0 attn.proj collecting stats Traceback (most recent call last):
File "/awesome-project/lit-parrot/quantize/gptq.py", line 376, in <module>
CLI(main)
File "/env/lib/python3.10/site-packages/jsonargparse/cli.py", line 85, in CLI
return _run_component(component, cfg_init)
File "/env/lib/python3.10/site-packages/jsonargparse/cli.py", line 147, in _run_component
return component(**cfg)
File "/awesome-project/lit-parrot/quantize/gptq.py", line 363, in main
llama_blockwise_quantization(model, encoded_text, device, bits=4)
File "/env/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
return func(*args, **kwargs)
File "/awesome-project/lit-parrot/quantize/gptq.py", line 265, in llama_blockwise_quantization
outs[j : j + 1], _ = block(
File "/env/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1502, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/env/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1511, in _call_impl
return forward_call(*args, **kwargs)
File "/awesome-project/lit-parrot/lit_parrot/model.py", line 161, in forward
h, new_kv_cache = self.attn(n_1, rope, mask, max_seq_length, input_pos, kv_cache)
File "/env/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1502, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/env/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1511, in _call_impl
return forward_call(*args, **kwargs)
File "/awesome-project/lit-parrot/lit_parrot/model.py", line 198, in forward
qkv = self.attn(x)
File "/env/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1502, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/env/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1511, in _call_impl
return forward_call(*args, **kwargs)
File "/awesome-project/lit-parrot/quantize/bnb.py", line 332, in forward
return qlinear_4bit_weight(inp, self.quant_weight, self.scales, self.zeros)
File "/awesome-project/lit-parrot/quantize/bnb.py", line 253, in qlinear_4bit_weight
linear_kernel_4bit_weight[grid](
File "/env/lib/python3.10/site-packages/triton/runtime/autotuner.py", line 97, in run
timings = {config: self._bench(*args, config=config, **kwargs)
File "/env/lib/python3.10/site-packages/triton/runtime/autotuner.py", line 97, in <dictcomp>
timings = {config: self._bench(*args, config=config, **kwargs)
File "/env/lib/python3.10/site-packages/triton/runtime/autotuner.py", line 80, in _bench
return do_bench(kernel_call, quantiles=(0.5, 0.2, 0.8))
File "/env/lib/python3.10/site-packages/triton/testing.py", line 44, in do_bench
fn()
File "/env/lib/python3.10/site-packages/triton/runtime/autotuner.py", line 78, in kernel_call
self.fn.run(*args, num_warps=config.num_warps, num_stages=config.num_stages, **current)
File "<string>", line 42, in linear_kernel_4bit_weight
File "/env/lib/python3.10/site-packages/triton/compiler/compiler.py", line 465, in compile
next_module = compile_kernel(module)
File "/env/lib/python3.10/site-packages/triton/compiler/compiler.py", line 361, in <lambda>
lambda src: ptx_to_cubin(src, arch))
File "/env/lib/python3.10/site-packages/triton/compiler/compiler.py", line 160, in ptx_to_cubin
return _triton.compile_ptx_to_cubin(ptx, ptxas, arch)
RuntimeError: Internal Triton PTX codegen error:
ptxas /tmp/compile-ptx-src-d48630, line 83; error : Feature '.bf16' requires .target sm_80 or higher
ptxas /tmp/compile-ptx-src-d48630, line 83; error : Feature 'cvt with .f32.bf16' requires .target sm_80 or higher
from litgpt.
Sure, I have an environment setup. Just lemme know, and I'm happy to try again 😊
- BTW, I'm not sure if the issue is due to an "old" GPU (RTX2080)
- I had to drop off lit-parrot as the requirements of my project changed. Namely, export any (L)LM to onnx with opset 9.
- I'm working directly with
transformers
andoptimum
. If you provide support for T5, I'm happy to keep playing with LIT-:parrot:
Motivation is mentioned here
cc @rasbt
from litgpt.
Thanks for pointing this out and sharing @escorciav
I am pretty certain that the RTX2080 only supports float16 but not bfloat16 training. Bfloat16 is a relatively recent feature for NVIDIA cards. The RTX 3000 series (and A100 etc.) would support it.
(The alternative here would be to run it with float16
instead of bfloat16
from litgpt.
Related Issues (20)
- Is there any support for visual generation? HOT 2
- Results of GPT2 (124M) reproduction with LitGPT HOT 2
- API has confusing/wrong description HOT 1
- Set kv cache size based on resource needs
- Pretrain then finetune on multiple GPUs error
- adding Aya model (checkpoints) to the supported checkpoints list that litgpt support
- did not convert the model to litgpt format, why? HOT 3
- Support downloading and using quantized weights (GGUF)
- Autodownload + conversion now fails on the first try
- Skip safetensors->bin file conversion
- Mistral Large 2 Checkpoints HOT 1
- Support sequential generation in litgpt serve HOT 1
- Provide minimal installation
- Implement prompt caching to speed up inference HOT 1
- access hidden layer(s) from a model
- TPU Pod Training
- MPS test gave out gibberish result HOT 1
- Use FlexAttention
- Tensor parallelism generates non-sensical outputs HOT 1
- Gemma 2B weights seem to have changed HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from litgpt.