Loading pretrained model weights from data/pretrain_model/Resnet50.npy，I met this problem about fpn HOT 5 CLOSED

@chl916185
Are you training FPN end-to-end?
I haven't encountered problems like this under end-to-end training.
While I have encountered problems like this under alt-opt training when I set gpu=1.
And if I set gpu=0, this problem disappeared.
#8517 is also discussing this problem.

from fpn.

I solved the problem, when I test, appear this problem：
python ./faster_rcnn/test_net.py --gpu 3 --weights
output/FPN_end2end/voc_2007_trainval/FPN_iter_200.ckpt --imdb voc_2007_test --cfg ./experiments/cfgs/FPN_end2end.yml --network FPN_test
invalid device ordinal
cudaCheckError() failed in ROIPoolForward: invalid device ordinal
cudaCheckError() failed in ROIPoolForward: driver shutting down
Segmentation fault (core dumped)
failed in ROIPoolForward: invalid device ordinal
2017-08-21 22:13:15.234379: E tensorflow/stream_executor/cuda/cuda_driver.cc:1087] could not wait stream on event: CUDA_ERROR_DEINITIALIZED
2017-08-21 22:13:15.234443: I tensorflow/stream_executor/stream.cc:4125] stream 0x784b380 did not memcpy device-to-host; source: 0x23050e5100
2017-08-21 22:13:15.234462: E tensorflow/stream_executor/stream.cc:289] Error recording event in stream: error recording CUDA event on stream 0x7849980: CUDA_ERROR_DEINITIALIZED; not marking stream as bad, as the Event object may be at fault. Monitor for further errors.
2017-08-21 22:13:15.234482: E tensorflow/stream_executor/cuda/cuda_event.cc:49] Error polling for event status: failed to query event: CUDA_ERROR_DEINITIALIZED
2017-08-21 22:13:15.234494: F tensorflow/core/common_runtime/gpu/gpu_event_mgr.cc:203] Unexpected Event status: 1
Aborted (core dumped)

@xmyqsh

from fpn.

Which version is your tensorflow？
@xmyqsh

from fpn.

@chl916185
1.1.0-rc0
For these problems, set gpu=0 will be OK.

from fpn.

@chl916185 can you share your solution for this problem?
cudaCheckError() failed in ROIPoolForward: invalid device function

from fpn.