bigwode / rebar-detection-competition Goto Github PK

I can train it normally,but when i use python test.py ,I got the following error.
i use:
cuda8.0
cupy-cuda8.0
pytorch=0.4.1
make export:
export CUDA_HOME=/usr/local/cuda-8.0
export PATH=${CUDA_HOME}/bin:${PATH}
export LD_LIBRARY_PATH=${CUDA_HOME}/lib64:$LD_LIBRARY_PATH

WARNING:root:Setting up a new session...
WARNING:visdom:Without the incoming socket you cannot receive events from the server or register event handlers to your Visdom client.
519C12BD
Traceback (most recent call last):
File "/home/yuqing/anaconda3/lib/python3.6/site-packages/cupy/cuda/compiler.py", line 241, in compile
nvrtc.compileProgram(self.ptr, options)
File "cupy/cuda/nvrtc.pyx", line 98, in cupy.cuda.nvrtc.compileProgram
File "cupy/cuda/nvrtc.pyx", line 108, in cupy.cuda.nvrtc.compileProgram
File "cupy/cuda/nvrtc.pyx", line 53, in cupy.cuda.nvrtc.check_status
cupy.cuda.nvrtc.NVRTCError: NVRTC_ERROR_COMPILATION (6)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "test.py", line 26, in
_bboxes, _labels, _scores = trainer.faster_rcnn.predict(img, visualize=True)
File "/home/yuqing/tsf/rebar-detection-competition/model/faster_rcnn.py", line 19, in new_f
return f(*args,**kwargs)
File "/home/yuqing/tsf/rebar-detection-competition/model/faster_rcnn.py", line 227, in predict
roi_cls_loc, roi_scores, rois, _ = self(img, scale=scale)
File "/home/yuqing/anaconda3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 477, in call
result = self.forward(*input, **kwargs)
File "/home/yuqing/tsf/rebar-detection-competition/model/faster_rcnn.py", line 121, in forward
self.rpn(h, img_size, scale)
File "/home/yuqing/anaconda3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 477, in call
result = self.forward(*input, **kwargs)
File "/home/yuqing/tsf/rebar-detection-competition/model/region_proposal_network.py", line 119, in forward
scale=scale)
File "/home/yuqing/tsf/rebar-detection-competition/model/utils/creator_tool.py", line 411, in call
thresh=self.nms_thresh)
File "/home/yuqing/tsf/rebar-detection-competition/model/utils/nms/non_maximum_suppression.py", line 26, in non_maximum_suppression
return _non_maximum_suppression_gpu(bbox, thresh, score, limit)
File "/home/yuqing/tsf/rebar-detection-competition/model/utils/nms/non_maximum_suppression.py", line 38, in _non_maximum_suppression_gpu
order = cp.arange(n_bbox, dtype=np.int32)
File "/home/yuqing/anaconda3/lib/python3.6/site-packages/cupy/creation/ranges.py", line 57, in arange
_arange_ufunc(typ(start), typ(step), ret, dtype=dtype)
File "cupy/core/_kernel.pyx", line 821, in cupy.core._kernel.ufunc.call
File "cupy/util.pyx", line 48, in cupy.util.memoize.decorator.ret
File "cupy/core/_kernel.pyx", line 612, in cupy.core._kernel._get_ufunc_kernel
File "cupy/core/_kernel.pyx", line 46, in cupy.core._kernel._get_simple_elementwise_kernel
File "cupy/core/carray.pxi", line 148, in cupy.core.core.compile_with_cache
File "/home/yuqing/anaconda3/lib/python3.6/site-packages/cupy/cuda/compiler.py", line 164, in compile_with_cache
ptx = compile_using_nvrtc(source, options, arch)
File "/home/yuqing/anaconda3/lib/python3.6/site-packages/cupy/cuda/compiler.py", line 82, in compile_using_nvrtc
ptx = prog.compile(options)
File "/home/yuqing/anaconda3/lib/python3.6/site-packages/cupy/cuda/compiler.py", line 245, in compile
raise CompileException(log, self.src, self.name, options)
cupy.cuda.compiler.CompileException: /home/yuqing/anaconda3/lib/python3.6/site-packages/cupy/core/include/cupy/carray.cuh(55): error: identifier "__half_raw" is undefined

/home/yuqing/anaconda3/lib/python3.6/site-packages/cupy/core/include/cupy/carray.cuh(45): error: no suitable constructor exists to convert from "float" to "__half"

/home/yuqing/anaconda3/lib/python3.6/site-packages/cupy/core/include/cupy/carray.cuh(47): error: no suitable constructor exists to convert from "float" to "__half"

/home/yuqing/anaconda3/lib/python3.6/site-packages/cupy/core/include/cupy/carray.cuh(48): error: no suitable constructor exists to convert from "double" to "__half"

/home/yuqing/anaconda3/lib/python3.6/site-packages/cupy/core/include/cupy/carray.cuh(49): error: no suitable constructor exists to convert from "int" to "__half"

/home/yuqing/anaconda3/lib/python3.6/site-packages/cupy/core/include/cupy/carray.cuh(50): error: no suitable constructor exists to convert from "unsigned int" to "__half"

/home/yuqing/anaconda3/lib/python3.6/site-packages/cupy/core/include/cupy/carray.cuh(51): error: no suitable constructor exists to convert from "long long" to "__half"

/home/yuqing/anaconda3/lib/python3.6/site-packages/cupy/core/include/cupy/carray.cuh(52): error: no suitable constructor exists to convert from "unsigned long long" to "__half"

/home/yuqing/anaconda3/lib/python3.6/site-packages/cupy/core/include/cupy/carray.cuh(57): error: no suitable conversion function from "const half" to "float" exists

/home/yuqing/anaconda3/lib/python3.6/site-packages/cupy/core/include/cupy/carray.cuh(62): error: identifier "__half_raw" is undefined

/home/yuqing/anaconda3/lib/python3.6/site-packages/cupy/core/include/cupy/carray.cuh(66): error: identifier "__half_raw" is undefined

/home/yuqing/anaconda3/lib/python3.6/site-packages/cupy/core/include/cupy/carray.cuh(66): error: identifier "__half_raw" is undefined

/home/yuqing/anaconda3/lib/python3.6/site-packages/cupy/core/include/cupy/carray.cuh(71): error: identifier "__half_raw" is undefined

/home/yuqing/anaconda3/lib/python3.6/site-packages/cupy/core/include/cupy/carray.cuh(75): error: identifier "__half_raw" is undefined

/home/yuqing/anaconda3/lib/python3.6/site-packages/cupy/core/include/cupy/carray.cuh(79): error: identifier "__half_raw" is undefined

/home/yuqing/anaconda3/lib/python3.6/site-packages/cupy/core/include/cupy/carray.cuh(107): error: identifier "__half_raw" is undefined

/home/yuqing/anaconda3/lib/python3.6/site-packages/cupy/core/include/cupy/carray.cuh(107): error: identifier "__half_raw" is undefined

/home/yuqing/anaconda3/lib/python3.6/site-packages/cupy/core/include/cupy/carray.cuh(108): error: expected a ";"

/home/yuqing/anaconda3/lib/python3.6/site-packages/cupy/core/include/cupy/carray.cuh(109): error: expected a ";"

/home/yuqing/anaconda3/lib/python3.6/site-packages/cupy/core/include/cupy/carray.cuh(110): error: identifier "ret_raw_" is undefined

/home/yuqing/anaconda3/lib/python3.6/site-packages/cupy/core/include/cupy/carray.cuh(110): error: identifier "y_raw_" is undefined

/home/yuqing/anaconda3/lib/python3.6/site-packages/cupy/core/include/cupy/carray.cuh(115): error: identifier "__half_raw" is undefined

/home/yuqing/anaconda3/lib/python3.6/site-packages/cupy/core/include/cupy/carray.cuh(115): error: identifier "__half_raw" is undefined

/home/yuqing/anaconda3/lib/python3.6/site-packages/cupy/core/include/cupy/carray.cuh(116): error: expected a ";"

/home/yuqing/anaconda3/lib/python3.6/site-packages/cupy/core/include/cupy/carray.cuh(117): error: expected a ";"

/home/yuqing/anaconda3/lib/python3.6/site-packages/cupy/core/include/cupy/carray.cuh(119): error: identifier "ret_raw_" is undefined

/home/yuqing/anaconda3/lib/python3.6/site-packages/cupy/core/include/cupy/carray.cuh(121): error: identifier "ret_raw_" is undefined

/home/yuqing/anaconda3/lib/python3.6/site-packages/cupy/core/include/cupy/carray.cuh(123): error: identifier "ret_raw_" is undefined

/home/yuqing/anaconda3/lib/python3.6/site-packages/cupy/core/include/cupy/carray.cuh(123): error: identifier "y_raw_" is undefined

/home/yuqing/anaconda3/lib/python3.6/site-packages/cupy/core/include/cupy/carray.cuh(125): error: identifier "y_raw_" is undefined

/home/yuqing/anaconda3/lib/python3.6/site-packages/cupy/core/include/cupy/carray.cuh(126): error: identifier "ret_raw_" is undefined

/home/yuqing/anaconda3/lib/python3.6/site-packages/cupy/core/include/cupy/carray.cuh(128): error: identifier "ret_raw_" is undefined

/home/yuqing/anaconda3/lib/python3.6/site-packages/cupy/core/include/cupy/carray.cuh(130): error: identifier "y_raw_" is undefined

/home/yuqing/anaconda3/lib/python3.6/site-packages/cupy/core/include/cupy/carray.cuh(131): error: identifier "ret_raw_" is undefined

/home/yuqing/anaconda3/lib/python3.6/site-packages/cupy/core/include/cupy/carray.cuh(133): error: identifier "ret_raw_" is undefined

/home/yuqing/anaconda3/lib/python3.6/site-packages/cupy/core/include/cupy/carray.cuh(135): error: identifier "ret_raw_" is undefined

/home/yuqing/anaconda3/lib/python3.6/site-packages/cupy/core/include/cupy/carray.cuh(140): error: identifier "__half_raw" is undefined

/home/yuqing/anaconda3/lib/python3.6/site-packages/cupy/core/include/cupy/carray.cuh(140): error: identifier "__half_raw" is undefined

/home/yuqing/anaconda3/lib/python3.6/site-packages/cupy/core/include/cupy/carray.cuh(141): error: expected a ";"

/home/yuqing/anaconda3/lib/python3.6/site-packages/cupy/core/include/cupy/carray.cuh(142): error: identifier "y_raw_" is undefined

/home/yuqing/anaconda3/lib/python3.6/site-packages/cupy/core/include/cupy/atomics.cuh(29): error: no suitable constructor exists to convert from "unsigned short" to "__half"

/home/yuqing/anaconda3/lib/python3.6/site-packages/cupy/core/include/cupy/atomics.cuh(38): error: identifier "__half_raw" is undefined

/home/yuqing/anaconda3/lib/python3.6/site-packages/cupy/core/include/cupy/carray.cuh(281): warning: statement is unreachable
detected during instantiation of "void CIndexer<_ndim>::set(ptrdiff_t) [with _ndim=1]"
/tmp/tmp5dr6mwg9/kern.cu(12): here

42 errors detected in the compilation of "/tmp/tmp5dr6mwg9/kern.cu".

$ python train.py train --env='fasterrcnn-caffe' --plot-every=100 --caffe-pretrain
======user config========
{'caffe_pretrain': True,
'caffe_pretrain_path': '/usr/local/workspace/python/rebar-detection-competition/checkpoints/vgg16_caffe.pth',
'data': 'voc',
'debug_file': './tmp/debugf',
'env': 'fasterrcnn-caffe',
'epoch': 14,
'load_path': None,
'lr': 0.001,
'lr_decay': 0.1,
'max_size': 2666,
'min_size': 2000,
'num_workers': 8,
'plot_every': 100,
'port': 8097,
'pretrained_model': 'vgg16',
'roi_sigma': 1.0,
'rpn_sigma': 3.0,
'test_num': 1000,
'test_num_workers': 8,
'use_adam': False,
'use_chainer': False,
'use_drop': False,
'voc_data_dir': '/usr/local/workspace/python/VOC_GJ/',
'weight_decay': 0.0005}
==========end============
load data
model construct completed
WARNING:root:Setting up a new session...
WARNING:visdom:Without the incoming socket you cannot receive events from the server or register event handlers to your Visdom client.
0it [00:00, ?it/s]
Traceback (most recent call last):
File "train.py", line 115, in
fire.Fire()
File "/home/ajax/anaconda3/lib/python3.7/site-packages/fire/core.py", line 127, in Fire
component_trace = _Fire(component, args, context, name)
File "/home/ajax/anaconda3/lib/python3.7/site-packages/fire/core.py", line 366, in _Fire
component, remaining_args)
File "/home/ajax/anaconda3/lib/python3.7/site-packages/fire/core.py", line 542, in _CallCallable
result = fn(*varargs, **kwargs)
File "train.py", line 71, in train
trainer.train_step(img, bbox, label, scale)
File "/usr/local/workspace/python/rebar-detection-competition/trainer.py", line 169, in train_step
losses = self.forward(imgs, bboxes, labels, scale)
File "/usr/local/workspace/python/rebar-detection-competition/trainer.py", line 100, in forward
self.faster_rcnn.rpn(features, img_size, scale) # roi_indices:[2000,0],anchor:[16650,4]
File "/home/ajax/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py", line 489, in call
result = self.forward(*input, **kwargs)
File "/usr/local/workspace/python/rebar-detection-competition/model/region_proposal_network.py", line 119, in forward
scale=scale)
File "/usr/local/workspace/python/rebar-detection-competition/model/utils/creator_tool.py", line 411, in call
thresh=self.nms_thresh)
File "/usr/local/workspace/python/rebar-detection-competition/model/utils/nms/non_maximum_suppression.py", line 26, in non_maximum_suppression
return _non_maximum_suppression_gpu(bbox, thresh, score, limit)
File "/usr/local/workspace/python/rebar-detection-competition/model/utils/nms/non_maximum_suppression.py", line 42, in _non_maximum_suppression_gpu
sorted_bbox, thresh)
File "/usr/local/workspace/python/rebar-detection-competition/model/utils/nms/non_maximum_suppression.py", line 125, in _call_nms_kernel
mask_dev = cp.zeros((n_bbox * col_blocks,), dtype=np.uint64)
File "/home/ajax/anaconda3/lib/python3.7/site-packages/cupy/creation/basic.py", line 147, in zeros
a = cupy.ndarray(shape, dtype, order=order)
File "cupy/core/core.pyx", line 151, in cupy.core.core.ndarray.init
File "cupy/cuda/memory.pyx", line 517, in cupy.cuda.memory.alloc
File "cupy/cuda/memory.pyx", line 1065, in cupy.cuda.memory.MemoryPool.malloc
File "cupy/cuda/memory.pyx", line 1086, in cupy.cuda.memory.MemoryPool.malloc
File "cupy/cuda/memory.pyx", line 900, in cupy.cuda.memory.SingleDeviceMemoryPool.malloc
File "cupy/cuda/memory.pyx", line 921, in cupy.cuda.memory.SingleDeviceMemoryPool._malloc
File "cupy/cuda/memory.pyx", line 695, in cupy.cuda.memory._try_malloc
cupy.cuda.memory.OutOfMemoryError: out of memory to allocate 18048000 bytes (total 18480128 bytes)

If you suspect this is an IPython bug, please report it at:
https://github.com/ipython/ipython/issues
or send an email to the mailing list at [email protected]

You can print a more detailed traceback right now with "%tb", or use "%debug"
to interactively debug it.

Extra-detailed tracebacks for bug-reporting purposes can be enabled via:
%config Application.verbose_crash=True

你好，我是刚接触目标检测，我运行了你的模型，训练没有报错，checkpoints里面也有训练模型，但是测试时，就得不出结果，submit_csv()文件为空。感谢指点!

    raise Exception('Your CUDA environment is invalid. '
Exception: Your CUDA environment is invalid. Please check above error log.

----------------------------------------

Command "python setup.py egg_info" failed with error code 1 in C:\Users\AppData\Local\Temp\pip-install-9chkl6om\cupy

''python train.py train --env='fasterrcnn-caffe' --plot-every=100 --caffe-pretrain'', when I ran this command,the following error occurred.While I have installed cupy via ''pip install cupy-cuda100'', corresponding to cuda10,cudnn7.3,rtx2080ti.

load data
Traceback (most recent call last):
File "/home/baiyang/anaconda3/lib/python3.5/site-packages/cupy/cuda/compiler.py", line 241, in compile
nvrtc.compileProgram(self.ptr, options)
File "cupy/cuda/nvrtc.pyx", line 98, in cupy.cuda.nvrtc.compileProgram
File "cupy/cuda/nvrtc.pyx", line 108, in cupy.cuda.nvrtc.compileProgram
File "cupy/cuda/nvrtc.pyx", line 53, in cupy.cuda.nvrtc.check_status
cupy.cuda.nvrtc.NVRTCError: NVRTC_ERROR unknown (7)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "train.py", line 118, in
fire.Fire()
File "/home/baiyang/anaconda3/lib/python3.5/site-packages/fire/core.py", line 127, in Fire
component_trace = _Fire(component, args, context, name)
File "/home/baiyang/anaconda3/lib/python3.5/site-packages/fire/core.py", line 366, in _Fire
component, remaining_args)
File "/home/baiyang/anaconda3/lib/python3.5/site-packages/fire/core.py", line 542, in _CallCallable
result = fn(*varargs, **kwargs)
File "train.py", line 60, in train
faster_rcnn = FasterRCNNVGG16()
File "/home/baiyang/Project/gangjin/rebar-detection-competition-master/model/faster_rcnn_vgg16.py", line 76, in init
classifier=classifier
File "/home/baiyang/Project/gangjin/rebar-detection-competition-master/model/faster_rcnn_vgg16.py", line 115, in init
self.roi = RoIPooling2D(self.roi_size, self.roi_size, self.spatial_scale)
File "/home/baiyang/Project/gangjin/rebar-detection-competition-master/model/roi_module.py", line 82, in init
self.RoI = RoI(outh, outw, spatial_scale)
File "/home/baiyang/Project/gangjin/rebar-detection-competition-master/model/roi_module.py", line 31, in init
self.forward_fn = load_kernel('roi_forward', kernel_forward)
File "cupy/util.pyx", line 48, in cupy.util.memoize.decorator.ret
File "/home/baiyang/Project/gangjin/rebar-detection-competition-master/model/roi_module.py", line 18, in load_kernel
kernel_code = cupy.cuda.compile_with_cache(code)
File "/home/baiyang/anaconda3/lib/python3.5/site-packages/cupy/cuda/compiler.py", line 135, in compile_with_cache
base = _preprocess('', options, arch)
File "/home/baiyang/anaconda3/lib/python3.5/site-packages/cupy/cuda/compiler.py", line 98, in _preprocess
result = prog.compile(options)
File "/home/baiyang/anaconda3/lib/python3.5/site-packages/cupy/cuda/compiler.py", line 245, in compile
raise CompileException(log, self.src, self.name, options)
cupy.cuda.compiler.CompileException: nvrtc: error: failed to load builtins for compute_70.
Any help would be very much appreciated.

RuntimeError: Error(s) in loading state_dict for FasterRCNNVGG16: size mismatch for head.cls_loc.bias: copying a param of torch.Size([8]) from checkpoint, where the shape is torch.Size([84]) in current model. size mismatch for head.cls_loc.weight: copying a param of torch.Size([8, 4096]) from checkpoint, where the shape is torch.Size([84, 4096]) in current model. size mismatch for head.score.bias: copying a param of torch.Size([2]) from checkpoint, where the shape is torch.Size([21]) in current model. size mismatch for head.score.weight: copying a param of torch.Size([2, 4096]) from checkpoint, where the shape is torch.Size([21, 4096]) in current model.
Any help would be very much appreciated.

I have a question. Will your code would resume training checkpoint? Thanks for your help.

can you share the dataset , [email protected] .thanks

运行的时候出现：
load data
model construct completed
Without the incoming socket you cannot receive events from the server or register event handlers to your Visdom client.

nohub.out出现
INFO:root:Application Started
INFO:tornado.access:200 POST /events (127.0.0.1) 178.00ms
INFO:tornado.access:200 POST /events (127.0.0.1) 0.99ms
INFO:tornado.access:200 POST /events (127.0.0.1) 0.84ms
INFO:tornado.access:200 POST /events (127.0.0.1) 1.21ms
INFO:tornado.access:200 POST /events (127.0.0.1) 0.54ms
INFO:tornado.access:200 POST /events (127.0.0.1) 0.52ms
INFO:tornado.access:200 POST /events (127.0.0.1) 1.02ms
INFO:tornado.access:200 POST /events (127.0.0.1) 1.22ms
INFO:tornado.access:200 POST /events (127.0.0.1) 0.99ms
INFO:tornado.access:200 POST /events (127.0.0.1) 0.93ms
INFO:tornado.access:200 POST /events (127.0.0.1) 0.98ms
INFO:tornado.access:200 POST /events (127.0.0.1) 0.94ms
INFO:tornado.access:200 POST /events (127.0.0.1) 3.25ms

这个是那里出错了呢 Visdom没有开启成功吗

"install other dependencies: pip install -r requirements.txt"
However, I can't find the requirements.txt"

我能问下你是如何做到单模型0.99的分数吗，我目前试了retinanet,faster r-cnn单模型，但是最高分数只有0.973,你在baseline的基础上还做了哪些工作呢？