ROCm 6.x appears Cannot find CO in the bundle libhipblaslt.so for ISA about pytorch HOT 20 CLOSED

I was following the official site link https://rocm.docs.amd.com/projects/install-on-linux/en/latest/how-to/amdgpu-install.html to install ROCm6.1
"Installation via AMDGPU installer", Ubuntu22.04
sudo amdgpu-install --usecase=graphics,rocm

pytorch were using command pip3 install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/rocm6.0

from pytorch.

There is an example directory in SadTalker. Can you reproduce the problem with those files? Please list all the detailed steps for all the installations and the sequence of ROCm install, pytorch install, SadTalker installation after you can reproduce the issue with the examples. (this will help me to reproduce on my end, as I don't use SadTalker).

Here is the steps:

1. Install ROCm via AMDGPU installer

sudo apt update
wget https://repo.radeon.com/amdgpu-install/6.1.1/ubuntu/jammy/amdgpu-install_6.1.60101-1_all.deb
cp amdgpu-install_6.1.60101-1_all.deb /tmp/
sudo apt install /tmp/amdgpu-install_6.1.60101-1_all.deb
sudo amdgpu-install --usecase=graphics,rocm

2. For SadTalker, here is the following setup steps
   a. Install pip requirements

git clone https://github.com/OpenTalker/SadTalker.git
cd SadTalker
python3 -m venv venv
pip install --pre torch torchvision torchaudio --index-url https://download.pytorch.org/whl/nightly/rocm6.1
pip install -r req.txt
pip install tb-nightly
pip install coqui-tts
pip install gradio==3.50.0
pip install numpy==1.23.4

b. Start webui
./webui.sh

I've run examples using these files, and reproduced the same problem:
Image: examples/source_image/art_7.png
Audio: examples/driven_audio/chinese_news.wav

from pytorch.

@kevinyjn Can you please add steps to reproduce your problem? Thanks.
If there is a small reproducer script, that would be even better. Otherwise, please list steps.

from pytorch.

@kevinyjn Can you please add steps to reproduce your problem? Thanks. If there is a small reproducer script, that would be even better. Otherwise, please list steps.

I was using SadTalker, with ROCm6.1 (ROCm6.0 also tried, the same issue), using webui.sh to execute the program. Then in the WebUI, i dragged a picture with a single person into photo area, dragged a 19seconds audio wave file into audio area, press generation button, in the console it displays error info above. BTW, The error log were displayed when exported env AMD_LOG_LEVEL=2

from pytorch.

@kevinyjn Can you please copy/paste the links for SadWalker that you used. thanks.

from pytorch.

@kevinyjn Can you please copy/paste the links for SadWalker that you used. thanks.

the SadTalker link is: https://github.com/OpenTalker/SadTalker

from pytorch.

also, you mentioned audio, did you install torchaudio?

from pytorch.

please also share how you installed pytorch or ROCm 6.1.

from pytorch.

maybe the last thing is, i may need your image and audio file to reproduce the problem.

from pytorch.

The image were generated by stablediffusion, and the audio file were generated by tts from a simple text.
I've tried, I cannot upload the .wav file as the attach files, how can I give it to you?

from pytorch.

There is an example directory in SadTalker. Can you reproduce the problem with those files?
Please list all the detailed steps for all the installations and the sequence of ROCm install, pytorch install, SadTalker installation after you can reproduce the issue with the examples. (this will help me to reproduce on my end, as I don't use SadTalker).

from pytorch.

@kevinyjn Excellent. We will reproduce and let you know

from pytorch.

@kevinyjn I went through the process and fixed a bunch of compatibility issues on my end. I was able to "Generate" audio from text, and now is dealing with the problem of generating video from image/audio.
The issue is related to the model itself. Where did you download the model weights?

  raise Exception("Failed in loading audio2pose_checkpoint")
Exception: Failed in loading audio2pose_checkpoint

from pytorch.

@kevinyjn I downloaded some weights, and it seems it did generate, but now I am facing another issue (need to dig to see what caused it. If you know, please let me know):

using safetensor as default
{'checkpoint': 'checkpoints/SadTalker_V0.0.2_256.safetensors', 'dir_of_BFM_fitting': 'src/config', 'audio2pose_yaml_path': 'src/config/auido2pose.yaml', 'audio2exp_yaml_path': 'src/config/auido2exp.yaml', 'use_safetensor': True, 'mappingnet_checkpoint': 'checkpoints/mapping_00229-model.pth.tar', 'facerender_yaml': 'src/config/facerender.yaml'}
Downloading: "https://github.com/xinntao/facexlib/releases/download/v0.1.0/alignment_WFLW_4HG.pth" to /home/xxx/SadTalker/gfpgan/weights/alignment_WFLW_4HG.pth

100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 185M/185M [00:06<00:00, 31.6MB/s]
Downloading: "https://github.com/xinntao/facexlib/releases/download/v0.1.0/detection_Resnet50_Final.pth" to /home/xxx/SadTalker/gfpgan/weights/detection_Resnet50_Final.pth

100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 104M/104M [00:02<00:00, 43.6MB/s]
/tmp/gradio/a5afa407f9b4ed9122399ff20b7699cf0cf98aaf/image.png
landmark Det:: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00,  6.86it/s]
3DMM Extraction In Video:: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00, 14.25it/s]
mel:: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 200/200 [00:00<00:00, 129573.80it/s]
audio2exp:: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 20/20 [00:00<00:00, 63.44it/s]
Face Renderer:: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 100/100 [03:32<00:00,  2.12s/it]
IMAGEIO FFMPEG_WRITER WARNING: input image is not divisible by macro_block_size=16, resizing from (256, 226) to (256, 240) to ensure video compatibility with most codecs and players. To prevent resizing, make your input image divisible by the macro_block_size or set the macro_block_size to 1 (risking incompatibility).
sh: 1: ffmpeg: not found
Traceback (most recent call last):
  File "/usr/lib/python3.10/shutil.py", line 816, in move
    os.rename(src, real_dst)
FileNotFoundError: [Errno 2] No such file or directory: '37a395c3-8cf7-47a3-9366-40376d5fa7af.mp4' -> './results/cbf634b2-5c9f-4057-8f99-b1b6549fb4d0/image##chinese_news-0-100.mp4'

from pytorch.

I got it working now. So, it seems to be the installation problem. I will summarize what I have done next.

from pytorch.

If export AMD_LOG_LEVEL=2, the log will display "Cannot find CO in the bundle libhipblaslt.so" and "Cannot find the function: Cijk_Ailk_Bljk_SB_MT128x64x8_SN_1LDSB0_APM1_ABV0_ACED0_AF0EM1_AF1EM1_AMAS3_ASE_ASGT_ASLT_ASM_ASAE01_ASCE01_ASEM1_AAC0_BL1_BS1_CLR0_DTLA0_DTLB0_DTVA0_DTVB0_DVO0_ETSP_EPS1_ELFLR0_EMLL0_FSSC10_FL0_GLVWA4_GLVWB4_GRCGA1_GRCGB1_GRPM1_GRVW4_GSU4_GSUAMB_GLS0_ISA1100_IU1_K1_KLA_LBSPPA0_LBSPPB0_LPA0_LPB0_LDL1_LRVW4_LWPMn1_LDW0_FMA_MIAV0_MDA2_MO40_MMFGLC_MKFGSU256_NTA0_NTB0_NTC0_NTD0_NEPBS0_NLCA1_NLCB1_ONLL1_OPLV0_PK0_PAP0_PGR1_PLR1_SIA1_SS0_SU0_SUM0_SUS0_SCIUI1_SPO0_SRVW0_SSO0_SVW4_SNLL0_TSGRA0_TSGRB0_TT8_8_TLDS0_UMLDSA0_UMLDSB0_U64SL1_USFGROn1_VAW1_VSn1_VW4_VWB4_VFLRP0_WSGRA0_WSGRB0_WS32_WG16_8_1_WGM4"

Could you reproduce the issue if exported AMD_LOG_LEVEL env?
And, Face Renderer progress seems that not using tensor calculation because of the warning log, so that it generates very slow, as you tested, it's only 2.07s/it

from pytorch.

ok, will check.

from pytorch.

@kevinyjn I did see those warnings when turning on AMD_LOG_LEVEL. Those were just warnings, not errors.
For now, that is expected behavior.
The perf may be improved in the future for gfx1100 (no timeline is available).

from pytorch.

If export AMD_LOG_LEVEL=2, the log will display "Cannot find CO in the bundle libhipblaslt.so" and "Cannot find the function: Cijk_Ailk_Bljk_SB_MT128x64x8_SN_1LDSB0_APM1_ABV0_ACED0_AF0EM1_AF1EM1_AMAS3_ASE_ASGT_ASLT_ASM_ASAE01_ASCE01_ASEM1_AAC0_BL1_BS1_CLR0_DTLA0_DTLB0_DTVA0_DTVB0_DVO0_ETSP_EPS1_ELFLR0_EMLL0_FSSC10_FL0_GLVWA4_GLVWB4_GRCGA1_GRCGB1_GRPM1_GRVW4_GSU4_GSUAMB_GLS0_ISA1100_IU1_K1_KLA_LBSPPA0_LBSPPB0_LPA0_LPB0_LDL1_LRVW4_LWPMn1_LDW0_FMA_MIAV0_MDA2_MO40_MMFGLC_MKFGSU256_NTA0_NTB0_NTC0_NTD0_NEPBS0_NLCA1_NLCB1_ONLL1_OPLV0_PK0_PAP0_PGR1_PLR1_SIA1_SS0_SU0_SUM0_SUS0_SCIUI1_SPO0_SRVW0_SSO0_SVW4_SNLL0_TSGRA0_TSGRB0_TT8_8_TLDS0_UMLDSA0_UMLDSB0_U64SL1_USFGROn1_VAW1_VSn1_VW4_VWB4_VFLRP0_WSGRA0_WSGRB0_WS32_WG16_8_1_WGM4"

Could you reproduce the issue if exported AMD_LOG_LEVEL env? And, Face Renderer progress seems that not using tensor calculation because of the warning log, so that it generates very slow, as you tested, it's only 2.07s/it

@kevinyjn @hongxiayang ,
Checking these logs, hipblasLT is getting triggered which shouldnt happen since it supports only MI300 HW, on gfx1100 it should take the non-hipblasLT path.

from pytorch.

Close for now as this is not an error.

from pytorch.