Comments (20)
I also encountered this bug.
I have been using Docker images to install and use deepspeed, and the same code I have used worked before. However, when I created a new container through the Docker image and installed deepspeed, the same issue as described above occurs. Is this a problem with the 0.14.3 version of deepspeed? How can I resolve this?
Specific system information is as follows:
System info (please complete the following information):
- OS: Ubuntu 22.04
- GPU count and types A6000 x4
- Python version 3.10.6
- Cuda 12.0
- deepspeed 0.14.3
- torch 2.3.1
- accelerate 0.31.0
from deepspeed.
I get same error and noticed that cpu_adam.so didn't get properly built. In my case it seems a missing depedency. You can scroll back further to see what caused the module didn't load.
[2/3] c++ -MMD -MF cpu_adam_impl.o.d -DTORCH_EXTENSION_NAME=cpu_adam -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COM[138/58213]
\"_gcc\" -DPYBIND11_STDLIB=\"_libstdcpp\" -DPYBIND11_BUILD_ABI=\"_cxxabi1011\" -I/home/gma/DeepSpeed/csrc/includes -isystem /h
ome/gma/miniforge3/envs/ds/lib/python3.11/site-packages/torch/include -isystem /home/gma/miniforge3/envs/ds/lib/python3.11/sit
e-packages/torch/include/torch/csrc/api/include -isystem /home/gma/miniforge3/envs/ds/lib/python3.11/site-packages/torch/inclu
de/TH -isystem /home/gma/miniforge3/envs/ds/lib/python3.11/site-packages/torch/include/THC -isystem /home/gma/miniforge3/envs/
ds/include/python3.11 -D_GLIBCXX_USE_CXX11_ABI=0 -fPIC -std=c++17 -O3 -std=c++17 -g -Wno-reorder -L/usr/local/cuda/lib64 -lcud
art -lcublas -g -march=native -fopenmp -D__AVX256__ -D__ENABLE_CUDA__ -DBF16_AVAILABLE -c /home/gma/DeepSpeed/csrc/adam/cpu_ad
am_impl.cpp -o cpu_adam_impl.o
[3/3] c++ cpu_adam.o cpu_adam_impl.o -shared -lcurand -L/home/gma/miniforge3/envs/ds/lib/python3.11/site-packages/torch/lib -l
c10 -ltorch_cpu -ltorch -ltorch_python -o cpu_adam.so
FAILED: cpu_adam.so
c++ cpu_adam.o cpu_adam_impl.o -shared -lcurand -L/home/gma/miniforge3/envs/ds/lib/python3.11/site-packages/torch/lib -lc10 -l
torch_cpu -ltorch -ltorch_python -o cpu_adam.so
/usr/bin/ld: cannot find -lcurand: No such file or directory
from deepspeed.
@delock
that's not the case with me.
I tried
DS_BUILD_CPU_ADAM=1 pip install deepspeed
and I got
gcc -pthread -B /opt/shared/anaconda/2024.02/compiler_compat -DNDEBUG -fwrapv -O2 -Wall -fPIC -O2 -isystem /opt/shared/anaconda/2024.02/include -fPIC -O2 -isystem /opt/shared/anaconda/2024.02/include -fPIC -I/tmp/pip-install-gibr8vp7/deepspeed_b3851d81a4bb41fd902ed8835bbe2ecd/csrc/includes -I/home/3458/pytorch/venv/lib/python3.11/site-packages/torch/include -I/home/3458/pytorch/venv/lib/python3.11/site-packages/torch/include/torch/csrc/api/include -I/home/3458/pytorch/venv/lib/python3.11/site-packages/torch/include/TH -I/home/3458/pytorch/venv/lib/python3.11/site-packages/torch/include/THC -I/opt/shared/cuda/12.4.1-550.54.15/include -I/home/3458/pytorch/venv/include -I/opt/shared/anaconda/2024.02/include/python3.11 -c csrc/adam/cpu_adam.cpp -o build/temp.linux-x86_64-cpython-311/csrc/adam/cpu_adam.o -O3 -std=c++17 -g -Wno-reorder -L/opt/shared/cuda/12.4.1-550.54.15/lib64 -lcudart -lcublas -g -march=native -fopenmp -D__AVX256__ -D__ENABLE_CUDA__ -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE=\"_gcc\" -DPYBIND11_STDLIB=\"_libstdcpp\" -DPYBIND11_BUILD_ABI=\"_cxxabi1011\" -DTORCH_EXTENSION_NAME=cpu_adam_op -D_GLIBCXX_USE_CXX11_ABI=0
In file included from /tmp/pip-install-gibr8vp7/deepspeed_b3851d81a4bb41fd902ed8835bbe2ecd/csrc/includes/cpu_adam.h:14,
from csrc/adam/cpu_adam.cpp:6:
...
...
gcc: fatal error: Killed signal terminated program cc1plus
compilation terminated.
error: command '/opt/shared/gcc/11.2.0/bin/gcc' failed with exit code 1
[end of output]
note: This error originates from a subprocess, and is likely not a problem with pip.
ERROR: Failed building wheel for deepspeed
Running setup.py clean for deepspeed
Failed to build deepspeed
ERROR: Could not build wheels for deepspeed, which is required to install pyproject.toml-based projects
from deepspeed.
@delock i got same error,infact i have the libcurand.so
from deepspeed.
@delock that's not the case with me. I tried
DS_BUILD_CPU_ADAM=1 pip install deepspeed
and I gotgcc -pthread -B /opt/shared/anaconda/2024.02/compiler_compat -DNDEBUG -fwrapv -O2 -Wall -fPIC -O2 -isystem /opt/shared/anaconda/2024.02/include -fPIC -O2 -isystem /opt/shared/anaconda/2024.02/include -fPIC -I/tmp/pip-install-gibr8vp7/deepspeed_b3851d81a4bb41fd902ed8835bbe2ecd/csrc/includes -I/home/3458/pytorch/venv/lib/python3.11/site-packages/torch/include -I/home/3458/pytorch/venv/lib/python3.11/site-packages/torch/include/torch/csrc/api/include -I/home/3458/pytorch/venv/lib/python3.11/site-packages/torch/include/TH -I/home/3458/pytorch/venv/lib/python3.11/site-packages/torch/include/THC -I/opt/shared/cuda/12.4.1-550.54.15/include -I/home/3458/pytorch/venv/include -I/opt/shared/anaconda/2024.02/include/python3.11 -c csrc/adam/cpu_adam.cpp -o build/temp.linux-x86_64-cpython-311/csrc/adam/cpu_adam.o -O3 -std=c++17 -g -Wno-reorder -L/opt/shared/cuda/12.4.1-550.54.15/lib64 -lcudart -lcublas -g -march=native -fopenmp -D__AVX256__ -D__ENABLE_CUDA__ -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE=\"_gcc\" -DPYBIND11_STDLIB=\"_libstdcpp\" -DPYBIND11_BUILD_ABI=\"_cxxabi1011\" -DTORCH_EXTENSION_NAME=cpu_adam_op -D_GLIBCXX_USE_CXX11_ABI=0 In file included from /tmp/pip-install-gibr8vp7/deepspeed_b3851d81a4bb41fd902ed8835bbe2ecd/csrc/includes/cpu_adam.h:14, from csrc/adam/cpu_adam.cpp:6: ... ... gcc: fatal error: Killed signal terminated program cc1plus compilation terminated. error: command '/opt/shared/gcc/11.2.0/bin/gcc' failed with exit code 1 [end of output] note: This error originates from a subprocess, and is likely not a problem with pip. ERROR: Failed building wheel for deepspeed Running setup.py clean for deepspeed Failed to build deepspeed ERROR: Could not build wheels for deepspeed, which is required to install pyproject.toml-based projects
The full error message from gcc might give an indication what might have gone wrong. The real reason for kernel build failure might be different in your case. One thing I usually try is execute the following command printed out by DeepSpeed manually so this specific error can be reproduced and triaged.
gcc -pthread -B /opt/shared/anaconda/2024.02/compiler_compat -DNDEBUG -fwrapv -O2 -Wall -fPIC -O2 -isystem /opt/shared/anaconda/2024.02/include -fPIC -O2 -isystem /opt/shared/anaconda/2024.02/include -fPIC -I/tmp/pip-install-gibr8vp7/deepspeed_b3851d81a4bb41fd902ed8835bbe2ecd/csrc/includes -I/home/3458/pytorch/venv/lib/python3.11/site-packages/torch/include -I/home/3458/pytorch/venv/lib/python3.11/site-packages/torch/include/torch/csrc/api/include -I/home/3458/pytorch/venv/lib/python3.11/site-packages/torch/include/TH -I/home/3458/pytorch/venv/lib/python3.11/site-packages/torch/include/THC -I/opt/shared/cuda/12.4.1-550.54.15/include -I/home/3458/pytorch/venv/include -I/opt/shared/anaconda/2024.02/include/python3.11 -c csrc/adam/cpu_adam.cpp -o build/temp.linux-x86_64-cpython-311/csrc/adam/cpu_adam.o -O3 -std=c++17 -g -Wno-reorder -L/opt/shared/cuda/12.4.1-550.54.15/lib64 -lcudart -lcublas -g -march=native -fopenmp -D__AVX256__ -D__ENABLE_CUDA__ -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE=\"_gcc\" -DPYBIND11_STDLIB=\"_libstdcpp\" -DPYBIND11_BUILD_ABI=\"_cxxabi1011\" -DTORCH_EXTENSION_NAME=cpu_adam_op -D_GLIBCXX_USE_CXX11_ABI=0
from deepspeed.
Hi @delock
I followed your advice.
The compilation is stuck at some point when building from source or when using pip.
This is when using
DS_BUILD_CPU_ADAM=1 pip install deepspeed
or
DS_BUILD_CPU_ADAM=1 pip install . # inside DeepSpeed directory
Here is the command it is stuck at
gcc -pthread -B /opt/shared/anaconda/2024.02/compiler_compat -DNDEBUG -fwrapv -O2 -Wall -fPIC -O2 -isystem /opt/shared/anaconda/2024.02/include -fPIC -O2 -isystem /opt/shared/anaconda/2024.02/include -fPIC -I/home/3458/pytorch/DeepSpeed/csrc/includes -I/home/3458/pytorch/venv/lib/python3.11/site-packages/torch/include -I/home/3458/pytorch/venv/lib/python3.11/site-packages/torch/include/torch/csrc/api/include -I/home/3458/pytorch/venv/lib/python3.11/site-packages/torch/include/TH -I/home/3458/pytorch/venv/lib/python3.11/site-packages/torch/include/THC -I/opt/shared/cuda/12.4.1-550.54.15/include -I/home/3458/pytorch/venv/include -I/opt/shared/anaconda/2024.02/include/python3.11 -c csrc/adam/cpu_adam.cpp -o build/temp.linux-x86_64-cpython-311/csrc/adam/cpu_adam.o -O3 -std=c++17 -g -Wno-reorder -L/opt/shared/cuda/12.4.1-550.54.15/lib64 -lcudart -lcublas -g -march=native -fopenmp -D__AVX256__ -D__ENABLE_CUDA__ -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE=\"_gcc\" -DPYBIND11_STDLIB=\"_libstdcpp\"
And here is where it is stuck
building 'deepspeed.ops.adam.cpu_adam_op' extension
gcc -pthread -B /opt/shared/anaconda/2024.02/compiler_compat -DNDEBUG -fwrapv -O2 -Wall -fPIC -O2 -isystem /opt/shared/anaconda/2024.02/include -fPIC -O2 -isystem /opt/shared/anaconda/2024.02/include -fPIC -I/home/3458/pytorch/DeepSpeed/csrc/includes -I/home/3458/pytorch/venv/lib/python3.11/site-packages/torch/include -I/home/3458/pytorch/venv/lib/python3.11/site-packages/torch/include/torch/csrc/api/include -I/home/3458/pytorch/venv/lib/python3.11/site-packages/torch/include/TH -I/home/3458/pytorch/venv/lib/python3.11/site-packages/torch/include/THC -I/opt/shared/cuda/12.4.1-550.54.15/include -I/home/3458/pytorch/venv/include -I/opt/shared/anaconda/2024.02/include/python3.11 -c csrc/adam/cpu_adam.cpp -o build/temp.linux-x86_64-cpython-311/csrc/adam/cpu_adam.o -O3 -std=c++17 -g -Wno-reorder -L/opt/shared/cuda/12.4.1-550.54.15/lib64 -lcudart -lcublas -g -march=native -fopenmp -D__AVX256__ -D__ENABLE_CUDA__ -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE=\"_gcc\" -DPYBIND11_STDLIB=\"_libstdcpp\" -DPYBIND11_BUILD_ABI=\"_cxxabi1011\" -DTORCH_EXTENSION_NAME=cpu_adam_op -D_GLIBCXX_USE_CXX11_ABI=0
In file included from /home/3458/pytorch/DeepSpeed/csrc/includes/cpu_adam.h:14,
from csrc/adam/cpu_adam.cpp:6:
/home/3458/pytorch/DeepSpeed/csrc/includes/simd.h:133: warning: ignoring '#pragma unroll ' [-Wunknown-pragmas]
133 | #pragma unroll
|
/home/3458/pytorch/DeepSpeed/csrc/includes/simd.h:154: warning: ignoring '#pragma unroll ' [-Wunknown-pragmas]
154 | #pragma unroll
|
/home/3458/pytorch/DeepSpeed/csrc/includes/simd.h:163: warning: ignoring '#pragma unroll ' [-Wunknown-pragmas]
163 | #pragma unroll
|
/home/3458/pytorch/DeepSpeed/csrc/includes/simd.h:184: warning: ignoring '#pragma unroll ' [-Wunknown-pragmas]
184 | #pragma unroll
|
/home/3458/pytorch/DeepSpeed/csrc/includes/simd.h:191: warning: ignoring '#pragma unroll ' [-Wunknown-pragmas]
191 | #pragma unroll
|
/home/3458/pytorch/DeepSpeed/csrc/includes/simd.h:199: warning: ignoring '#pragma unroll ' [-Wunknown-pragmas]
199 | #pragma unroll
|
/home/3458/pytorch/DeepSpeed/csrc/includes/simd.h:207: warning: ignoring '#pragma unroll ' [-Wunknown-pragmas]
207 | #pragma unroll
|
/home/3458/pytorch/DeepSpeed/csrc/includes/simd.h:215: warning: ignoring '#pragma unroll ' [-Wunknown-pragmas]
215 | #pragma unroll
|
/home/3458/pytorch/DeepSpeed/csrc/includes/simd.h:221: warning: ignoring '#pragma unroll ' [-Wunknown-pragmas]
221 | #pragma unroll
|
/home/3458/pytorch/DeepSpeed/csrc/includes/simd.h:227: warning: ignoring '#pragma unroll ' [-Wunknown-pragmas]
227 | #pragma unroll
|
/home/3458/pytorch/DeepSpeed/csrc/includes/simd.h:233: warning: ignoring '#pragma unroll ' [-Wunknown-pragmas]
233 | #pragma unroll
|
/home/3458/pytorch/DeepSpeed/csrc/includes/simd.h:239: warning: ignoring '#pragma unroll ' [-Wunknown-pragmas]
239 | #pragma unroll
|
/home/3458/pytorch/DeepSpeed/csrc/includes/simd.h:245: warning: ignoring '#pragma unroll ' [-Wunknown-pragmas]
245 | #pragma unroll
|
/home/3458/pytorch/DeepSpeed/csrc/includes/simd.h:251: warning: ignoring '#pragma unroll ' [-Wunknown-pragmas]
251 | #pragma unroll
|
/home/3458/pytorch/DeepSpeed/csrc/includes/simd.h:257: warning: ignoring '#pragma unroll ' [-Wunknown-pragmas]
257 | #pragma unroll
|
/home/3458/pytorch/DeepSpeed/csrc/includes/simd.h:263: warning: ignoring '#pragma unroll ' [-Wunknown-pragmas]
263 | #pragma unroll
|
/home/3458/pytorch/DeepSpeed/csrc/includes/simd.h:269: warning: ignoring '#pragma unroll ' [-Wunknown-pragmas]
269 | #pragma unroll
|
/home/3458/pytorch/DeepSpeed/csrc/includes/simd.h:277: warning: ignoring '#pragma unroll ' [-Wunknown-pragmas]
277 | #pragma unroll
|
/home/3458/pytorch/DeepSpeed/csrc/includes/simd.h:283: warning: ignoring '#pragma unroll ' [-Wunknown-pragmas]
283 | #pragma unroll
|
/home/3458/pytorch/DeepSpeed/csrc/includes/simd.h:289: warning: ignoring '#pragma unroll ' [-Wunknown-pragmas]
289 | #pragma unroll
|
/home/3458/pytorch/DeepSpeed/csrc/includes/simd.h:295: warning: ignoring '#pragma unroll ' [-Wunknown-pragmas]
295 | #pragma unroll
|
from deepspeed.
Now when using
DS_BUILD_CPU_ADAM=1 ./install.sh
It gives the same above error but telling which line
gcc: fatal error: Killed signal terminated program cc1plus
compilation terminated.
error: command '/opt/shared/gcc/11.2.0/bin/gcc' failed with exit code 1
Error on line 155
Fail to install deepspeed
from deepspeed.
Related Issues (20)
- [BUG] oneapi/ccl.hpp: No such file or directory. HOT 1
- [BUG]模型卡在trainer.train()一直不训练
- [BUG] Running llama2-7b step3 with tensor parallel and HE fails due to incompatible shapes
- RuntimeError: Error building extension 'cpu_adam', because /usr/bin/ld: can not find -lcurand,help! HOT 1
- Fail to use zero_init to construct llama2 with deepspeed zero3 and bnb!
- does DeepSpeed support AMSP (a new DP shard strategy)
- [BUG] 'Invalidate trace cache' with Seq2SeqTrainer+predict_with_generate+Zero3
- AssertionError: Unable to pre-compile ops without torch installed. Please install torch before attempting to pre-compile ops. HOT 3
- How to set different learning rates for different parameters of LLMs
- Getting parameters of embeddings (safe_get_local_fp32_param)and setting the weight of embeddings (safe_set_local_fp32_param) does not work (bug?).
- [BUG] DeepSpeed on pypi not compatible with latest `numpy` HOT 5
- [BUG] GPU memory leaking after deleting deepspeed engine HOT 2
- Bug Report: Issues Building DeepSpeed on Windows HOT 4
- [BUG] Logs full of FutureWarning when training with nightly PyTorch HOT 1
- [BUG] inference ValueError
- Running out of CPU memory. Dataset is loaded for each created process
- [ERROR] [launch.py:321:sigkill_handler] exits with return code = -11
- [BUG] Regression: 0.14.3 causes grad_norm to be zero HOT 1
- [BUG] Deepspeed does not seem to work using GPUDirect, should it? HOT 3
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from deepspeed.