shouxieai / infer Goto Github PK

I get error message :"[yolo.cu:339]: CUDA Runtime error💥 cudaPeekAtLastError() # too many resources requested for launch, code = cudaErrorLaunchOutOfResources [ 701 ]"
It points to the Kernal function: checkKernel(warp_affine_bilinear_and_normalize_plane_kernel << <grid, block, 0, stream >> > (src, src_line_size, src_width, src_height, dst, dst_width, dst_height, const_value, matrix_2_3, norm));
There is no problem with the kernal function at simple_yolo.cu (The previous version).

想在推理完之后加一个专门的显存释放函数？该怎么写？（我是C++菜鸟。。

怎么写一个 CMakeLists.txt 而不是用 kiwi？假设我已经装好 所需要的库。

Hello, I reported this error when using the yolov5 model generated by myself for reasoning

Dear shouxieai,

I encountered an issue while using your document and I'm seeking your assistance to resolve it. The details are as follows:

CMakeLists.txt

cmake_minimum_required(VERSION 2.6)
project(pro)
add_definitions(-std=c++11)

option(CUDA_USE_STATIC_CUDA_RUNTIME OFF)
set(CMAKE_CXX_STANDARD 11)
set(CMAKE_BUILD_TYPE Debug)
set(EXECUTABLE_OUTPUT_PATH ${PROJECT_SOURCE_DIR}/workspace)

set(CUDA_GEN_CODE "-gencode=arch=compute_72,code=sm_72")

set(OpenCV_DIR "/usr/local/include/opencv4/")
set(CUDA_DIR "/usr/local/cuda-10.2")
set(TENSORRT_INCLUDE_DIR "/usr/include/aarch64-linux-gnu")
set(TENSORRT_LIB_DIR "/usr/lib/aarch64-linux-gnu")

find_package(CUDA REQUIRED)
find_package(OpenCV)

include_directories(
${PROJECT_SOURCE_DIR}/src
${OpenCV_INCLUDE_DIRS}
${CUDA_DIR}
${TENSORRT_INCLUDE_DIR}
)

link_directories(
${TENSORRT_INCLUDE_DIR}/lib
${CUDA_DIR}/lib64
)

set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -std=c++11 -Wall -O0 -Wfatal-errors -pthread -w -g")
set(CUDA_NVCC_FLAGS "${CUDA_NVCC_FLAGS} -std=c++11 -O0 -Xcompiler -fPIC -g -w ${CUDA_GEN_CODE}")
file(GLOB_RECURSE cpp_srcs ${PROJECT_SOURCE_DIR}/src/.cpp)
file(GLOB_RECURSE cuda_srcs ${PROJECT_SOURCE_DIR}/src/.cu)
cuda_add_library(cucodes SHARED ${cuda_srcs})

add_executable(pro ${cpp_srcs})

target_link_libraries(cucodes nvinfer nvonnxparser)
target_link_libraries(cucodes cuda cublas cudart cudnn)
target_link_libraries(pro ${OpenCV_LIBS})
target_link_libraries(pro cucodes)

add_custom_target(
run
DEPENDS pro
WORKING_DIRECTORY ${PROJECT_SOURCE_DIR}/workspace
COMMAND ./pro
)

error:

CMakeFiles/pro.dir/src/main.cpp.o: In function single_inference()': /infer/src/main.cpp:107: undefined reference to yolo::Infer::forward(yolo::Image const&, void*)'
collect2: error: ld returned 1 exit status
CMakeFiles/pro.dir/build.make:165: recipe for target '../workspace/pro' failed

I appreciate your guidance and valuable resources.

Sincerely,
yidasahe92905

TensorRT版本：8.2.3.0
转换命令：trtexec --onnx=weights/best.onnx --minShapes=images:1x3x640x640 --maxShapes=images:16x3x640x640 --optShapes=images:1x3x640x640 --saveEngine=./yolov5s.engine
报错：
[infer.cu:393]: Infer 0x5598119488f0 [DynamicShape]
[infer.cu:405]: Inputs: 1
[infer.cu:409]: 0.images : shape {-1x3x640x640}
[infer.cu:412]: Outputs: 4
[infer.cu:416]: 0.520 : shape {-1x256x80x80}
[infer.cu:416]: 1.552 : shape {-1x512x40x40}
[infer.cu:416]: 2.584 : shape {-1x1024x20x20}
[infer.cu:416]: 3.output : shape {-1x13x8400}
[2023-06-14 16:44:42][warn][trt_infer.cpp:26]:NVInfer: TensorRT was linked against cuBLAS/cuBLAS LT 11.6.5 but loaded cuBLAS/cuBLAS LT 11.5.1
[2023-06-14 16:44:42][warn][trt_infer.cpp:26]:NVInfer: TensorRT was linked against cuBLAS/cuBLAS LT 11.6.5 but loaded cuBLAS/cuBLAS LT 11.5.1
[infer.cu:209]: NVInfer: 1: [reformat.cpp::executeCutensor::385] Error Code 1: CuTensor (Internal cuTensor permutate execute failed)
[yolo.cu:597]: Failed to tensorRT forward.
[infer.cu:209]: NVInfer: 1: [reformat.cpp::executeCutensor::385] Error Code 1: CuTensor (Internal cuTensor permutate execute failed)
[yolo.cu:597]: Failed to tensorRT forward.
[infer.cu:209]: NVInfer: 3: [executionContext.cpp::enqueueInternal::330] Error Code 3: API Usage Error (Parameter check failed at: runtime/api/executionContext.cpp::enqueueInternal::330, condition: bindings[x] != nullptr
)
[yolo.cu:597]: Failed to tensorRT forward.
[infer.cu:209]: NVInfer: 3: [executionContext.cpp::enqueueInternal::330] Error Code 3: API Usage Error (Parameter check failed at: runtime/api/executionContext.cpp::enqueueInternal::330, condition: bindings[x] != nullptr
)
[yolo.cu:597]: Failed to tensorRT forward.
[infer.cu:209]: NVInfer: 3: [executionContext.cpp::enqueueInternal::330] Error Code 3: API Usage Error (Parameter check failed at: runtime/api/executionContext.cpp::enqueueInternal::330, condition: bindings[x] != nullptr
)
[yolo.cu:597]: Failed to tensorRT forward.
[infer.cu:209]: NVInfer: 1: [reformat.cpp::executeCutensor::385] Error Code 1: CuTensor (Internal cuTensor permutate execute failed)
[yolo.cu:597]: Failed to tensorRT forward.
[infer.cu:209]: NVInfer: 1: [pointWiseV2Helpers.h::launchPwgenKernel::546] Error Code 1: Cuda Driver (an illegal memory access was encountered)
[yolo.cu:597]: Failed to tensorRT forward.
[yolo.cu:503]: CUDA Runtime error💥 cudaMemcpyAsync(image_device, image_host, size_image, cudaMemcpyHostToDevice, stream_) # an illegal memory access was encountered, code = cudaErrorIllegalAddress [ 700 ]
已放弃 (核心已转储)

https://github.com/ultralytics/yolov5/releases/download/v7.0/yolov5s.onnx

trtexec --onnx=workspace/yolov5s.onnx
--minShapes=images:1x3x640x640
--maxShapes=images:16x3x640x640
--optShapes=images:1x3x640x640
--saveEngine=workspace/yolov5s.engine

[03/05/2023-13:17:08] [W] [TRT] onnx2trt_utils.cpp:377: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32.
[03/05/2023-13:17:08] [I] Finish parsing network model
[03/05/2023-13:17:08] [E] Static model does not take explicit shapes since the shape of inference tensors will be determined by the model itself
[03/05/2023-13:17:08] [E] Network And Config setup failed
[03/05/2023-13:17:08] [E] Building engine failed
[03/05/2023-13:17:08] [E] Failed to create engine from model or file.
[03/05/2023-13:17:08] [E] Engine set up failed

你好，我在用上述代码进行YOLOV8模型的engine模型进行推理已经成功了，我想在此基础上查看推理的时间也就是FPS值，请问如何再main.cpp里实现该功能，谢谢

使用yolov5m和yolov5s生成engine文件，然后make run，图像中的目标没有被检测到

输出caption 和 rectangle第一个point，结果是
person nan [0,0]
Save result to Result.jpg，1024 objects

请问如何将yolo::load()模型加载函数和yolo->forward预测函数导出动态链接库，方便其他程序调用？

cls怎么基于这套代码编写

我在代码里面，没看到transpose操作，是需要提前自己手动转好吗？

yolov8的实例分割推理结果的mask，怎么还原回原图？怎么把mask的结果渲染到原图上？谢谢！

我有多张显卡，我想在其中一张上做推理，如何指定显卡呢？

需要更改这个仓库什么文件，才能让我部署到我的jetson上

包括但不限于enqueueV2()更改成enqueueV3

(base) cvai@ubuntu:~/yolov8/infer-main$ make run
Compile depends CXX src/main.cpp
Compile CXX src/main.cpp
Link workspace/pro
[infer.cu:393]: Infer 0xffff5c000db0 [StaticShape]
[infer.cu:405]: Inputs: 1
[infer.cu:409]: 	0.images : shape {1x3x640x640}
[infer.cu:412]: Outputs: 4
[infer.cu:416]: 	0.onnx::Reshape_577 : shape {1x66x80x80}
[infer.cu:416]: 	1.onnx::Reshape_588 : shape {1x66x40x40}
[infer.cu:416]: 	2.onnx::Reshape_599 : shape {1x66x20x20}
[infer.cu:416]: 	3.output0 : shape {1x8400x6}
[infer.cu:209]: NVInfer: 1: [genericReformat.cu::executeMemcpy::1455] Error Code 1: Cuda Runtime (invalid argument)
[yolo.cu:597]: Failed to tensorRT forward.
[infer.cu:209]: NVInfer: 3: [executionContext.cpp::enqueueInternal::622] Error Code 3: API Usage Error (Parameter check failed at: runtime/api/executionContext.cpp::enqueueInternal::622, condition: bindings[x] || nullBindingOK
)
[yolo.cu:597]: Failed to tensorRT forward.
[infer.cu:209]: NVInfer: 3: [executionContext.cpp::enqueueInternal::622] Error Code 3: API Usage Error (Parameter check failed at: runtime/api/executionContext.cpp::enqueueInternal::622, condition: bindings[x] || nullBindingOK
)
[yolo.cu:597]: Failed to tensorRT forward.
[infer.cu:209]: NVInfer: 3: [executionContext.cpp::enqueueInternal::622] Error Code 3: API Usage Error (Parameter check failed at: runtime/api/executionContext.cpp::enqueueInternal::622, condition: bindings[x] || nullBindingOK

Who knows what I'm supposed to do with this? I have four outputs, and I think this is the one that's broken, so how do I change that to use the last output?

你好，我想请教下关于Image数据传递的问题。
Image(const uint8_t *bgrptr, int width, int height) : bgrptr(bgrptr), width(width), height(height) {}
Image(const uint8_t *data_device, int width, int height, int device_id, cudaStream_t stream, ImageType type)
: device_data(data_device), width(width), height(height), device_id(device_id), stream(stream), type(type) {}
好像只做了bgrptr地址的赋值，并没有进行memcpy操作。
下面的代码，如果必须在循环内定义Mat变量，每次循环commit过后，img指向的内存有可能在推理之前释放吗
int main()
{
cv::VideoCapture cap("1.mp4");
while(1)
{
cv::Mat img;
cap.read(img);
cpm.commit(img);
}
}

environment：cuda10.2 opencv4.2 tensorrt7.x
use CmakeLists.txt compile,cmakelists contents :
cmake_minimum_required(VERSION 3.16)
PROJECT(pro VERSION 1.0.0 LANGUAGES C CXX CUDA)
option(CUDA_USE_STATIC_CUDA_RUNTIME OFF)
set(CMAKE_CXX_STANDARD 11)
set(CMAKE_BUILD_TYPE Debug)
set(EXECUTABLE_OUTPUT_PATH ${PROJECT_SOURCE_DIR}/workspace)
set(CUDA_GEN_CODE "-gencode=arch=compute_75,code=sm_75")
if(NOT DEFINED CMAKE_CUDA_ARCHITECTURES)
set(CMAKE_CUDA_ARCHITECTURES 70 75 80)
endif(NOT DEFINED CMAKE_CUDA_ARCHITECTURES)
set(OpenCV_DIR "/usr/include/opencv4/")
set(CUDA_TOOLKIT_ROOT_DIR "/usr/local/cuda-10.2")
set(CUDNN_DIR "/usr/include/aarch64-linux-gnu")
set(TENSORRT_DIR "/usr/include/aarch64-linux-gnu")

find_package(CUDA REQUIRED)
find_package(OpenCV)

include_directories(
${PROJECT_SOURCE_DIR}/src
${OpenCV_DIR}/include
${OpenCV_INCLUDE_DIRS}
${CUDA_TOOLKIT_ROOT_DIR}/include
${TENSORRT_DIR}/include
${TENSORRT_DIR}
${CUDNN_DIR}/include
${CUDNN_DIR}
)
link_directories(
${TENSORRT_DIR}/lib
${CUDA_TOOLKIT_ROOT_DIR}/lib
${CUDNN_DIR}
${OpenCV_DIR}/x64/vc15/lib
)
set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -std=c++11 -Wall -O0 -Wfatal-errors -pthread -w -g")
set(CUDA_NVCC_FLAGS "${CUDA_NVCC_FLAGS} -std=c++11 -O0 -Xcompiler -fPIC -g -w ${CUDA_GEN_CODE}")
file(GLOB_RECURSE cpp_srcs ${PROJECT_SOURCE_DIR}/src/.cpp)
file(GLOB_RECURSE cuda_srcs ${PROJECT_SOURCE_DIR}/src/.cu)
cuda_add_library(plugin_list SHARED ${cuda_srcs})
target_link_libraries(plugin_list nvinfer nvinfer_plugin nvonnxparser)
target_link_libraries(plugin_list cuda cublas cudart cudnn)
target_link_libraries(plugin_list stdc++ dl)
target_link_libraries(plugin_list ${OpenCV_LIBS})
add_executable(pro ${cpp_srcs})
target_link_libraries(pro nvinfer nvinfer_plugin nvonnxparser)
target_link_libraries(pro cuda cublas cudart cudnn)
target_link_libraries(pro stdc++ dl)
target_link_libraries(pro plugin_list)
target_link_libraries(pro ${OpenCV_LIBS})

but errors occur when make ：

error: argument of type "void *const *" is incompatible with parameter of type "void **"

1 error detected in the compilation of "/tmp/tmpxft_00000d3a_00000000-6_infer.cpp1.ii".
CMake Error at plugin_list_generated_infer.cu.o.Debug.cmake:280 (message):
Error generating file
/media/tye/7b97c143-3564-430b-838e-6051e9005553/infer-main/build/CMakeFiles/plugin_list.dir/src/./plugin_list_generated_infer.cu.o

CMakeFiles/plugin_list.dir/build.make:75: recipe for target 'CMakeFiles/plugin_list.dir/src/plugin_list_generated_infer.cu.o' failed

what's the reason how to solve it?thanks!

Hello,
In perf() function, it looks like we have to do future get() after each commit.
It means the thread running the “cpmi.commit(cvimg(image))“ command is blocked by get().
For example, the following codes works:
std::vector<std::shared_futureyolo::BoxArray> outputs;
for (int idx = 0; idx < files.size(); ++idx) {
auto image = cv::imread(files[idx]);
auto fu = cpmi.commit(cvimg(image));
outputs.emplace_back(fu);
auto objs = outputs[idx].get(); //we can get the objs result successfully
};
The following codes do not work:
std::vector<std::shared_futureyolo::BoxArray> outputs;
for (int idx = 0; idx < files.size(); ++idx) {
auto image = cv::imread(files[idx]);
auto fu = cpmi.commit(cvimg(image));
outputs.emplace_back(fu);
};
auto objs = outputs[0].get(); //we cannot get correct objs result here !!! , it will have 0 or a random size.
Any suggestion if I want use another thread to manipulate the post-processed inference results? Thanks.

I have two GPUs. It would be great if I can point to which GPU to use by "auto yolo = yolo::load(modelpath, modelType, gpuID); " I am trying to modify codes by add "SetDevice(const int gpuID)" to the function construct() at infer.cn. But without sucess. Any sugestion? Thanks.

使用yolov8推理单通道的灰度图该怎么修改infer代码？我已经从训练就把模型改成了单通道，但是推理时也想改成单通道灰度图推理。训练灰度图参考了https://github.com/ultralytics/ultralytics/issues/948

环境：
显卡3080ti
系统：ubuntu20
cuda11.3.1
cudnn8.2.1
tensorrt8.2.0.6
模型：yolov5-6.1 yolov5s.onnx

[infer.cu:396]: Infer 000002327B533F10 [StaticShape]
[infer.cu:408]: Inputs: 1
[infer.cu:412]: 0.images : shape {1x3x800x800}
[infer.cu:415]: Outputs: 4
[infer.cu:419]: 0.onnx::Sigmoid_339 : shape {1x3x100x100x42}
[infer.cu:419]: 1.onnx::Sigmoid_391 : shape {1x3x50x50x42}
[infer.cu:419]: 2.onnx::Sigmoid_443 : shape {1x3x25x25x42}
[infer.cu:419]: 3.output : shape {1x39375x42}

image name: ./test_imgs/2023-5-5 15-13-27.bmp

[infer.cu:209]: NVInfer: 3: [executionContext.cpp::nvinfer1::rt::ExecutionContext::enqueueInternal::328] Error Code 3: Internal Error (Parameter check failed at: executionContext.cpp::nvinfer1::rt::ExecutionContext::enqueueInternal::328, condition: bindings[x] != nullptr
)

有看到别人提过这个问题，您回答的是The infer requires only one output binding. And you provide 4, which is not adapted.
请问如何将output修改成1呢？