RuntimeError: src/spconv/indice.cu 120 cuda execution failed with error 98 about clocs HOT 9 CLOSED

I got it working for Maxwell architecture using juimoisnono's method.
For compiling CUDA files for a specific architecture using nvcc, gencode flag need to be specified appropriately. For example, for compute capability 5.0, the nvcc flag is -gencode arch=compute_50,code=sm_50.
For spconv (commit 8da6f96), this flag can be specified using -DCMAKE_CUDA_FLAGS in spconv's setup.py.

diff --git a/setup.py b/setup.py
index 1e68a29..71b2d95 100644
--- a/setup.py
+++ b/setup.py
@@ -45,7 +45,7 @@ class CMakeBuild(build_ext):
                       '-DCMAKE_PREFIX_PATH=' + LIBTORCH_ROOT,
                       '-DPYBIND11_PYTHON_VERSION={}'.format(PYTHON_VERSION),
                       '-DSPCONV_BuildTests=OFF',
-                      '-DCMAKE_CUDA_FLAGS="--expt-relaxed-constexpr"']
+                      '-DCMAKE_CUDA_FLAGS=--expt-relaxed-constexpr -gencode arch=compute_50,code=sm_50']
 
         cfg = 'Debug' if self.debug else 'Release'
         # cfg = 'Debug'

from clocs.

I have not met or seen this type of error before, maybe this will help: open-mmlab/OpenPCDet#442

from clocs.

@pangsu0613 . Thanks for this. I ran it on ubuntu 20 with GPU of pascal architecture. Everything is running fine.

from clocs.

Thank you very much for sharing your solution @jayadeepk !

from clocs.

Iam using the ubuntu 20. cuda is failing with error 98 . Is there any solution for this . By the way iam using spcov1.0 version. pytorch version 1.8.0.
cuda:11.1.1-cudnn8

from clocs.

Hello @vignesh628 , most of the content you showed here are numba warnings, you dont need to worry about them, I found the mistake is "FileNotFoundError: [Errno 2] No such file or directory: '/home/developer/deep_learning/Projects/KITTI_DATASET_ROOT/KITTI_DATASET_ROOT/training/velodyne_reduced/000001.bin'", so it means that your KITTI dataset directory is not configured correctly, please check that. BTW, you could add the following code at the beginning of train.py to ignore these numba warnings: import warnings
warnings.simplefilter('ignore', category=NumbaDeprecationWarning)
warnings.simplefilter('ignore', category=NumbaPendingDeprecationWarning)
warnings.simplefilter('ignore', category=NumbaPerformanceWarning)
warnings.simplefilter('ignore', category=NumbaWarning)
warnings.simplefilter('ignore')
warnings.filterwarnings('ignore')

from clocs.

For ignoring the warnings, I missed one line: from numba.core.errors import NumbaDeprecationWarning, NumbaPendingDeprecationWarning,NumbaPerformanceWarning,NumbaWarning

from clocs.

Hello @pangsu0613 , Thanks for your reply. I have supressed the numba warnings and provided the correct path. Still facing the cuda execution failed with error 98 issue.

python /home/developer/deep_learning/deepti_ubuntu20/CLOCs/second/pytorch/train.py evaluate --config_path=/home/developer/deep_learning/deepti_ubuntu20/CLOCs/second/configs/car.fhd.config --model_dir=/home/developer/deep_learning/deepti_ubuntu20/CLOCs/CLOCs_SecCas_pretrained --measure_time=True --batch_size=1
Predict_test: False
sparse_shape: [ 41 1600 1408]
num_class is : 1
load existing model
load existing model for fusion layer
Restoring parameters from /home/developer/deep_learning/deepti_ubuntu20/CLOCs/CLOCs_SecCas_pretrained/fusion_layer-11136.tckpt
remain number of infos: 3769
Generate output labels...
Traceback (most recent call last):
File "/home/developer/deep_learning/deepti_ubuntu20/CLOCs/second/pytorch/train.py", line 928, in
fire.Fire()
File "/opt/conda/lib/python3.6/site-packages/fire/core.py", line 141, in Fire
component_trace = _Fire(component, args, parsed_flag_args, context, name)
File "/opt/conda/lib/python3.6/site-packages/fire/core.py", line 471, in _Fire
target=component.name)
File "/opt/conda/lib/python3.6/site-packages/fire/core.py", line 681, in _CallAndUpdateTrace
component = fn(*varargs, **kwargs)
File "/home/developer/deep_learning/deepti_ubuntu20/CLOCs/second/pytorch/train.py", line 679, in evaluate
model_cfg.lidar_input,global_set)
File "/home/developer/deep_learning/deepti_ubuntu20/CLOCs/second/pytorch/train.py", line 470, in predict_kitti_to_anno
all_3d_output_camera_dict, all_3d_output, top_predictions, fusion_input,torch_index = net(example,detection_2d_path)
File "/opt/conda/lib/python3.6/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
result = self.forward(*input, **kwargs)
File "/home/developer/deep_learning/deepti_ubuntu20/CLOCs/second/pytorch/models/voxelnet.py", line 304, in forward
voxel_features, coors, batch_size_dev)
File "/opt/conda/lib/python3.6/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
result = self.forward(*input, **kwargs)
File "/home/developer/deep_learning/deepti_ubuntu20/CLOCs/second/pytorch/models/middle.py", line 545, in forward
ret = self.middle_conv(ret)
File "/opt/conda/lib/python3.6/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
result = self.forward(*input, **kwargs)
File "/opt/conda/lib/python3.6/site-packages/spconv/modules.py", line 123, in forward
input = module(input)
File "/opt/conda/lib/python3.6/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
result = self.forward(*input, **kwargs)
File "/opt/conda/lib/python3.6/site-packages/spconv/conv.py", line 151, in forward
self.stride, self.padding, self.dilation, self.output_padding, self.subm, self.transposed, grid=input.grid)
File "/opt/conda/lib/python3.6/site-packages/spconv/ops.py", line 89, in get_indice_pairs
stride, padding, dilation, out_padding, int(subm), int(transpose))
RuntimeError: /home/developer/deep_learning/deepti_ubuntu20/spconv-8da6f967fb9a054d8870c3515b1b44eca2103634/src/spconv/indice.cu 120
cuda execution failed with error 98

from clocs.

That's awesome !

from clocs.