🐛 Describe the bug i use pytorch==2.3.0 and peft to train llama3

this is my cuda cudnn infos: <div class="snippet-clipboard-content notranslate pos

some of my dataset like: <div class="highlight highlight-source-json notranslate p

RuntimeError: "_amp_foreach_non_finite_check_and_unscale_cuda" not implemented for 'BFloat16' about pytorch HOT 3 OPEN

ykallan commented on September 26, 2024

RuntimeError: "_amp_foreach_non_finite_check_and_unscale_cuda" not implemented for 'BFloat16'

from pytorch.

Comments (3)

crcrpar commented on September 26, 2024 1

For BFloat16 AMP, the grad scaler wouldn't be needed as its range is equivalent to fp32's.
Could you try the script without grad scaler?

from pytorch.

ykallan commented on September 26, 2024

this is my cuda cudnn infos:

(llama) D:\codes\llm_about\self-llm>nvcc -V
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2023 NVIDIA Corporation
Built on Mon_Apr__3_17:36:15_Pacific_Daylight_Time_2023
Cuda compilation tools, release 12.1, V12.1.105
Build cuda_12.1.r12.1/compiler.32688072_0


(llama) D:\codes\llm_about\self-llm>nvidia-smi
Sun May 26 00:17:50 2024       
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 555.85                 Driver Version: 555.85         CUDA Version: 12.5     |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                  Driver-Model | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA GeForce RTX 3090      WDDM  |   00000000:01:00.0  On |                  N/A |
| 50%   38C    P8             19W /  350W |    1076MiB /  24576MiB |      9%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+

+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI        PID   Type   Process name                              GPU Memory |
|        ID   ID                                                               Usage      |

from pytorch.

ykallan commented on September 26, 2024

some of my dataset like:

[{
	"instruction": "请针对以下商品名称生成商品简称，并直接给出答案",
	"input": "南孚 7号 电池 16粒/盒",
	"output": "电池"
}, {
	"instruction": "请针对以下商品名称生成商品简称，并直接给出答案",
	"input": "三星 SAMSUNG Galaxy Z Flip3 5G 8GB+256GB 折叠屏5G手机 立式交互体验 IPX8防水 梦境极光",
	"output": "电池"
}, {
	"instruction": "请针对以下商品名称生成商品简称，并直接给出答案",
	"input": "达伯埃   Double A 复印纸  80g-A4一箱5包-2500张",
	"output": "复印纸"
}, {
	"instruction": "请针对以下商品名称生成商品简称，并直接给出答案",
	"input": "苏宁超市自营",
	"output": "复印纸"
}, {
	"instruction": "请针对以下商品名称生成商品简称，并直接给出答案",
	"input": "戴尔DELL 成就3710 商用办公电脑整机 23.8英寸",
	"output": "商用办公电脑整机"
}, {
	"instruction": "请针对以下商品名称生成商品简称，并直接给出答案",
	"input": "华硕 灵耀14 2022 超轻薄商务办公笔记本电脑 14英寸",
	"output": "超轻薄商务办公笔记本电脑"
}, {
	"instruction": "请针对以下商品名称生成商品简称，并直接给出答案",
	"input": "国产 扎壶 1500ml",
	"output": "扎壶"
}, {
	"instruction": "请针对以下商品名称生成商品简称，并直接给出答案",
	"input": "白云 尘推布 60cm",
	"output": "尘推布"
}]