openppl-public / ppl.llm.serving Goto Github PK
View Code? Open in Web Editor NEWLicense: Apache License 2.0
License: Apache License 2.0
想问下有性能分析的工具嘛?profiler相关,还是只能用nsight profile这种自己去看一些算子性能
GPU:A100-80G-SXM4
Ubuntu 20.04.4
cuda:12.3
cudnn:8904
trt:9.2.0
gcc 11.4
cmake version 3.27.9
Cuda compilation tools, release 12.3, V12.3.107
commit id:51c3b3d5c5eba25c276a84388f04a2c9e198699f
Cloning into '/workspace/dev/inference/ppl.llm.serving/deps/grpc/third_party/xds'...
Cloning into '/workspace/dev/inference/ppl.llm.serving/deps/grpc/third_party/zlib'...
Submodule path 'third_party/abseil-cpp': checked out 'c2435f8342c2d0ed8101cb43adfd605fdc52dca2'
Submodule path 'third_party/benchmark': checked out '015d1a091af6937488242b70121858bce8fd40e9'
error: Server does not allow request for unadvertised object 60209eb1ccc34d5deefb002d1b7f37545204f7f2
fatal: Fetched in submodule path 'third_party/bloaty', but it did not contain 60209eb1ccc34d5deefb002d1b7f37545204f7f2. Direct fetching of that commit failed.
fatal:
CMake Error at grpc-subbuild/grpc-populate-prefix/tmp/grpc-populate-gitclone.cmake:62 (message):
Failed to update submodules in:
'/workspace/dev/inference/ppl.llm.serving/deps/grpc'
gmake[2]: *** [CMakeFiles/grpc-populate.dir/build.make:102: grpc-populate-prefix/src/grpc-populate-stamp/grpc-populate-download] Error 1
gmake[1]: *** [CMakeFiles/Makefile2:83: CMakeFiles/grpc-populate.dir/all] Error 2
gmake: *** [Makefile:91: all] Error 2
I have installed libnccl, but still get the error.
/usr/bin/ld: CMakeFiles/ppl_llama_server.dir/src/models/llama/llama_server.cc.o: in function ResourceManager::~ResourceManager()': llama_server.cc:(.text._ZN15ResourceManagerD2Ev[_ZN15ResourceManagerD5Ev]+0xa8): undefined reference to
ncclCommDestroy'
/usr/bin/ld: CMakeFiles/ppl_llama_server.dir/src/models/llama/llama_server.cc.o: in function ResourceManager::Init(unsigned int, unsigned long, unsigned long, float, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)': llama_server.cc:(.text._ZN15ResourceManager4InitEjmmfRKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE[_ZN15ResourceManager4InitEjmmfRKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE]+0x13e): undefined reference to
ncclCommInitAll'
collect2: error: ld returned 1 exit status
make[2]: *** [CMakeFiles/ppl_llama_server.dir/build.make:195: ppl_llama_server] Error 1
make[1]: *** [CMakeFiles/Makefile2:1738: CMakeFiles/ppl_llama_server.dir/all] Error 2
make: *** [Makefile:156: all] Error 2
GPU
Ubuntu20.04
gcc/g++ 9.4
master
./build.sh -DPPLNN_USE_LLM_CUDA=ON -DPPLNN_CUDA_ENABLE_NCCL=OFF -DPPLNN_ENABLE_CUDA_JIT=OFF -DPPLNN_CUDA_ARCHITECTURES="'80;'" -DPPLCOMMON_CUDA_ARCHITECTURES="'80'"
An error was encountered while executing client_qps_measure.
Platform: llama-13B on 2 V100 GPUS
[INFO][2023-09-13 03:35:21.764][llama_server.cc:539] max_tokens: 75630
[INFO][2023-09-13 03:35:21.827][llama_server.cc:484] VOCAB_SIZE: 32000; BOS ID: 1; EOS ID: 2; PAD ID: -1
[INFO][2023-09-13 03:35:21.827][llama_server.cc:606] End init nccl, cuda engine, kv cache, kv scale manager
[INFO][2023-09-13 03:35:21.827][llama_server.cc:626] Init llama worker successed
[INFO][2023-09-13 03:35:21.827][llama_worker.cc:1043] waiting for request ...
[INFO][2023-09-13 03:35:21.829][llama_server.cc:639] listening on [0.0.0.0:23333]
[ERROR][2023-09-13 03:35:34.009][gemm.cu:113] cublasLt failed: the requested functionality is not supported
[ERROR][2023-09-13 03:35:34.009][kernel.cc:176] DoExecute kernel [/layers.0/wqkv/ColumnParallelLinear] failed: device runtime error
[ERROR][2023-09-13 03:35:34.009][gemm.cu:113] cublasLt failed: the requested functionality is not supported
[ERROR][2023-09-13 03:35:34.009][sequential_scheduler.cc:130] exec kernel[/layers.0/wqkv/ColumnParallelLinear] of type[pmx:ColumnParallelLinear:1] failed: device runtime error
[ERROR][2023-09-13 03:35:34.009][kernel.cc:176] DoExecute kernel [/layers.0/wqkv/ColumnParallelLinear] failed: device runtime error
[ERROR][2023-09-13 03:35:34.009][runtime_impl.cc:333] Run() failed: device runtime error
[ERROR][2023-09-13 03:35:34.009][sequential_scheduler.cc:130] exec kernel[/layers.0/wqkv/ColumnParallelLinear] of type[pmx:ColumnParallelLinear:1] failed: device runtime error
[ERROR][2023-09-13 03:35:34.009][llm_cuda_device.cc:276] [ERROR][2023-09-13 03:35:34.009][runtime_impl.cc:333] Run() failed: device runtime error
cudaStreamSynchronize failed: 700, an illegal memory access was encountered
[ERROR][2023-09-13 03:35:34.009][llm_cuda_device.cc:276] cudaStreamSynchronize failed: 700, an illegal memory access was encountered
[ERROR][2023-09-13 03:35:34.009][runtime_impl.cc:316] sync device[llm_cuda] failed: internal error
[ERROR][2023-09-13 03:35:34.009][runtime_impl.cc:316] sync device[llm_cuda] failed: internal error
[ERROR][2023-09-13 03:35:34.009][llama_worker.cc:922] ParallelExecute(RunModelTask) failed.
[INFO][2023-09-13 03:35:34.009][llama_worker.cc:1043] waiting for request ...
[ERROR][2023-09-13 03:35:34.010][llm_cuda_device.cc:112] cudaMemcpyAsync failed: 700, an illegal memory access was encountered
[ERROR][2023-09-13 03:35:34.010][llm_cuda_device.cc:112] [ERROR][2023-09-13 03:35:34.010][llama_worker.cc:724] cudaMemcpyAsync failed: 700, an illegal memory access was encountered
set token_ids [token_ids] failed: other error
[ERROR][2023-09-13 03:35:34.010][llama_worker.cc:724] set token_ids [token_ids] failed: other error
[ERROR][2023-09-13 03:35:34.010][llama_worker.cc:910] ParallelExecute(SetInputTask) failed.
[INFO][2023-09-13 03:35:34.010][llama_worker.cc:1043] waiting for request ...
[INFO][2023-09-13 03:35:34.010][llama_worker.cc:1043] waiting for request ...
[INFO][2023-09-13 03:35:34.011][llama_worker.cc:1043] waiting for request ...
[INFO][2023-09-13 03:35:34.011][llama_worker.cc:1043] waiting for request ...
[INFO][2023-09-13 03:35:34.011][llama_worker.cc:1043] waiting for request ...
On A30, It works fine.
On V100, run llama-13B using two GPUs.
NVIDIA-SMI 510.85.02 Driver Version: 510.85.02 CUDA Version: 11.7
[ERROR][2023-09-12 05:21:16.518][nccl_utils.h:110] NCCL error(code:1) on ncclGroupEnd
[ERROR][2023-09-12 05:21:16.518][nccl_utils.h:110] NCCL error(code:1) on ncclGroupEnd
[ERROR][2023-09-12 05:21:16.518][kernel.cc:176] [ERROR][2023-09-12 05:21:16.518][kernel.cc:176] DoExecute kernel [/tok_embeddings/ParallelEmbedding] failed: other error
DoExecute kernel [/tok_embeddings/ParallelEmbedding] failed: other error
[ERROR][2023-09-12 05:21:16.518][sequential_scheduler.cc:130] [ERROR][2023-09-12 05:21:16.518][sequential_scheduler.cc:130] exec kernel[/tok_embeddings/ParallelEmbedding] of type[pmx:ParallelEmbedding:1] failed:
other error
exec kernel[/tok_embeddings/ParallelEmbedding] of type[pmx:ParallelEmbedding:1] failed: other error
[ERROR][2023-09-12 05:21:16.518][runtime_impl.cc:333] Run() failed: other error
[ERROR][2023-09-12 05:21:16.518][runtime_impl.cc:333] Run() failed: other error
[ERROR][2023-09-12 05:21:16.519][llama_worker.cc:922] ParallelExecute(RunModelTask) failed.
[INFO][2023-09-12 05:21:16.519][llama_worker.cc:1043] waiting for request ...
[ERROR][2023-09-12 05:21:16.520][nccl_utils.h:110] NCCL error(code:1) on ncclGroupEnd
[ERROR][2023-09-12 05:21:16.520][kernel.cc:176] DoExecute kernel [/tok_embeddings/ParallelEmbedding] failed: other error
[ERROR][2023-09-12 05:21:16.520][sequential_scheduler.cc:130] exec kernel[/tok_embeddings/ParallelEmbedding] of type[pmx:ParallelEmbedding:1] failed: other error
[ERROR][2023-09-12 05:21:16.520][runtime_impl.cc:333] Run() failed: other error
[ERROR][2023-09-12 05:21:16.520][nccl_utils.h:110] NCCL error(code:1) on ncclGroupEnd
Here is my config.json:
{
"model_dir": "/data/codes/ppl/llama-13b",
"model_param_path": "/data/codes/ppl/llama-13b/params.json",
"tokenizer_path": "/data/LLaMA-7B/tokenizer.model",
"tensor_parallel_size": 2,
"top_p": 0.0,
"top_k": 1,
"max_tokens_scale": 0.94,
"max_tokens_per_request": 4096,
"max_running_batch": 1024,
"host": "0.0.0.0",
"port": 10086
}
Support ONNX format.
Intel / Nvidia
Linux / Debian
gcc 9.0
latest
GPU:A100-80G-SXM4
Ubuntu 20.04.4
cuda:12.3
cudnn:8904
trt:9.2.0
gcc 11.4
cmake version 3.27.9
Cuda compilation tools, release 12.3, V12.3.107
commit id:c2bf8614ea7bce0cb9838255fb3cd6ab9d75039b
./build.sh -DPPLNN_USE_LLM_CUDA=ON -DPPLNN_CUDA_ENABLE_NCCL=ON -DPPLNN_ENABLE_CUDA_JIT=OFF -DPPLNN_CUDA_ARCHITECTURES="'80;86;87'" -DPPLCOMMON_CUDA_ARCHITECTURES="'80;86;87'"
None
None
None
When I run the provided example by:
[server]
./ppl-build/ppl_llm_server llama_7b_config_example.json
[client]
./ppl-build/client_sample 127.0.0.1:23333
The program got stuck and not proceed.
After a 2-hour debugging, I found that this maybe caused by miss-use of StaticThreadPool
. The worker thread is implemented as:
void* StaticThreadPool::ThreadWorker(void* arg) {
auto info = (ThreadInfo*)arg;
auto pool = info->pool;
while (true) {
pool->barrier_.Wait();
if (!pool->func_) {
break;
}
pool->func_(pool->threads_.size(), info->thread_idx);
pool->barrier_.Wait();
}
return nullptr;
}
There are two barriers: one before func_
run and the other is after. However, for asynchronism, the asynchronous method such as StaticThreadPool::RunAsync
only call barrier(Wait()
) once. So, it leaves to caller to take another barrier(Wait()
) before the next run. And this is also noted by author in comment.
It seems strange to find that there may be still missing the SECOND barrier. When I add a pool->Wait()
at two places, the program starts to proceed. It seems that client and server interact with each other normally.
The two modifications are:
[utils.h]
template <typename TaskType, typename... TaskArgType>
ppl::common::RetCode ParallelExecute(ppl::common::StaticThreadPool* pool, TaskArgType&&... rest_args) {
auto n = pool->GetNumThreads();
ppl::common::Barrier finish_barrier;
ppl::common::RetCode thr_rc[n];
finish_barrier.Reset(n + 1);
pool->RunAsync([&](uint32_t nthr, uint32_t ithr) {
auto task = TaskType(ithr, std::forward<TaskArgType>(rest_args)...);
thr_rc[ithr] = task.Process();
finish_barrier.Wait();
});
finish_barrier.Wait();
ppl::common::RetCode retcode = ppl::common::RC_SUCCESS;
for (uint32_t i = 0; i < n; ++i) {
if (thr_rc[i] != ppl::common::RC_SUCCESS) {
LOG(ERROR) << "ParallelExecute task[" << i << "] failed";
retcode = thr_rc[i];
break;
}
}
pool->Wait();
return retcode;
}
and
[llama_worker.cc]
if (is_first_run_) {
is_first_run_ = false;
} else {
decoder_barrier_.Wait();
decoder_thread_pool_.Wait();
}
So I'd like to know if this modification is correct and is reasonable according to the design?
/data/openppl/ppl.llm.serving$ ./ppl-build/ppl_llama_server src/models/llama/conf/llama_13b_config_example.json
[INFO][2023-09-19 16:51:43.346][llama_server.cc:149] server_config.host: 0.0.0.0
[INFO][2023-09-19 16:51:43.346][llama_server.cc:150] server_config.port: 23333
[INFO][2023-09-19 16:51:43.346][llama_server.cc:152] server_config.model_dir: /data/openppl/ppl.pmx/model_zoo/llama/huggingface/llama_chinese_13b_ppl
[INFO][2023-09-19 16:51:43.346][llama_server.cc:153] server_config.model_param_path: /data/openppl/ppl.pmx/model_zoo/llama/huggingface/llama_chinese_13b_ppl/pmx_params.json
[INFO][2023-09-19 16:51:43.346][llama_server.cc:154] server_config.tokenizer_path: /data/wenda_llama/wenda-main/model/Chinese-LlaMA2-chat-7B-sft-v0.3
[INFO][2023-09-19 16:51:43.346][llama_server.cc:156] server_config.top_k: 1
[INFO][2023-09-19 16:51:43.346][llama_server.cc:157] server_config.top_p: 0
[INFO][2023-09-19 16:51:43.346][llama_server.cc:159] server_config.tensor_parallel_size: 2
[INFO][2023-09-19 16:51:43.347][llama_server.cc:160] server_config.max_tokens_scale: 0.93
[INFO][2023-09-19 16:51:43.347][llama_server.cc:161] server_config.max_tokens_per_request: 4096
[INFO][2023-09-19 16:51:43.347][llama_server.cc:162] server_config.max_running_batch: 1024
[ERROR][2023-09-19 16:51:43.347][llama_server.cc:221] find key [cache_quant_bit] failed
[ERROR][2023-09-19 16:51:43.347][llama_server.cc:561] PaseModelConfig failed, model_param_path: /data/openppl/ppl.pmx/model_zoo/llama/huggingface/llama_chinese_13b_ppl/pmx_params.json
模型我用huggingface的,利用pmx成功进行转换,并使用demo.py测试是成功的,但在serving这里就出错了
能提个建议吗,你们的说明文档能写得详细点吗,谢谢就好像pmx里面的转换和export两者是有什么区别吧,都要运行吗
We use vicuna in huggingface format. I notice only meta format is supported ? Is that true?
In file included from /home/liuxiandong/workspace/ppl/ppl.llm.serving/src/models/llama/llama_worker.h:25:0,
from /home/liuxiandong/workspace/ppl/ppl.llm.serving/src/models/llama/llama_worker.cc:18:
/home/liuxiandong/workspace/ppl/ppl.llm.serving/src/models/llama/../../utils/mpsc_request_scheduler.h:21:10: fatal error: ppl/common/event_count.h: No such file or directory
#include "ppl/common/event_count.h"
^~~~~~~~~~~~~~~~~~~~~~~~~~
In file included from /home/liuxiandong/workspace/ppl/ppl.llm.serving/src/models/llama/llama_worker.h:25:0,
from /home/liuxiandong/workspace/ppl/ppl.llm.serving/src/models/factory.cc:19:
/home/liuxiandong/workspace/ppl/ppl.llm.serving/src/models/llama/../../utils/mpsc_request_scheduler.h:21:10: fatal error: ppl/common/event_count.h: No such file or directory
#include "ppl/common/event_count.h"
^~~~~~~~~~~~~~~~~~~~~~~~~~
compilation terminated.
compilation terminated.
CMakeFiles/ppl_llm_static.dir/build.make:75: recipe for target 'CMakeFiles/ppl_llm_static.dir/src/models/factory.cc.o' failed
GPU
when I run this command:
./build.sh -DPPLNN_USE_LLM_CUDA=ON -DPPLNN_CUDA_ENABLE_NCCL=ON -DPPLNN_ENABLE_CUDA_JIT=OFF -DPPLNN_CUDA_ARCHITECTURES="'80;86;87'" -DPPLCOMMON_CUDA_ARCHITECTURES="'80;86;87'"
then:
mkdir: cannot create directory '/home/limenxin/openPPL/ppl.llm.serving/ppl-build': File exists
cmd -> cmake -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX=install -DPPLNN_USE_LLM_CUDA=ON -DPPLNN_CUDA_ENABLE_NCCL=ON -DPPLNN_ENABLE_CUDA_JIT=OFF -DPPLNN_CUDA_ARCHITECTURES='80;86;87' -DPPLCOMMON_CUDA_ARCHITECTURES='80;86;87' .. && cmake --build . -j 128 --config Release && cmake --build . --target install -j 128 --config Release
-- Populating hpcc
-- Configuring done
-- Generating done
-- Build files have been written to: /home/limenxin/openPPL/ppl.llm.serving/deps/hpcc-subbuild
[ 11%] Creating directories for 'hpcc-populate'
[ 22%] Performing download step (git clone) for 'hpcc-populate'
Cloning into 'hpcc'...
Already on 'master'
Your branch is up to date with 'origin/master'.
[ 33%] Performing update step for 'hpcc-populate'
HEAD is now at e674b69 use c++11 by default
[ 44%] No patch step for 'hpcc-populate'
[ 55%] No configure step for 'hpcc-populate'
[ 66%] No build step for 'hpcc-populate'
[ 77%] No install step for 'hpcc-populate'
[ 88%] No test step for 'hpcc-populate'
[100%] Completed 'hpcc-populate'
[100%] Built target hpcc-populate
CMake Error at /usr/local/share/cmake-3.23/Modules/FetchContent.cmake:754 (message):
No content details recorded for grpc
Call Stack (most recent call first):
/usr/local/share/cmake-3.23/Modules/FetchContent.cmake:1140 (__FetchContent_getSavedDetails)
/usr/local/share/cmake-3.23/Modules/FetchContent.cmake:1260 (FetchContent_Populate)
deps/hpcc/cmake/hpcc-common.cmake:92 (FetchContent_MakeAvailable)
CMakeLists.txt:24 (hpcc_populate_dep)
linux A800
Thank you for great job!
I just use your serving framework, but with some questions below:
how to control the request rate? I noticed that you just push all the batchrequest into the vector and send it to the server continuously, but I just want to fix the numbers of the request per second.
example like the vllm: https://github.com/vllm-project/vllm/blob/90eb3f43ca5228647c834243d98881d53c7745d0/benchmarks/benchmark_serving.py#L224
I try to disable this feature by setting --quantized_cache 0
when exporting model, but when loading model, it coredumps,
I see this:
if (model_config_.cache_quant_bit != 8 && model_config_.cache_quant_group != 8) {
LOG(ERROR) << "only support cache_quant_bit == 8 and cache_quant_group == 8";
return RC_INVALID_VALUE;
}
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.