Comments (9)
@mahyarnajibi
HI:
I also met the same problem.
Recently, I compiled the new master. Then I run the demo successfully. Thank you very much.
But, When I compile the code,I met some problems. Such as /home/jerry/SNIPER/SNIPER-mxnet/3rdparty/mshadow/mshadow/./base.h:155:23: fatal error: cblas.h: No such file or directory
. I link several file to /usr/include folder and solve it. And When I run the demo.py, some error raised. I solved it by the last two link.
libopenblas.so -> /home/jerry/softs/OpenBLAS/libopenblas.so* libopenblas.so.0 -> libopenblas.so*
I think my method is stupid. I'm poor at C and linux. But I hope it will be help.
from sniper.
after installation, I encounter a problem when running demo.py
:
mxnet.base.MXNetError: [07:35:37] /liang_volume/SNIPER/SNIPER-mxnet/src/operator/nn/batch_norm.cu:527: Check failed: err == cudaSuccess (7 vs. 0) Name: BatchNormalizationUpdateOutput ErrStr:too many resources requested for launch
Stack trace returned 10 entries:
[bt] (0) /liang_volume/SNIPER/SNIPER-mxnet/python/mxnet/../../build/libmxnet.so(dmlc::StackTraceabi:cxx11+0x1bc) [0x7f70067b0f7c]
[bt] (1) /liang_volume/SNIPER/SNIPER-mxnet/python/mxnet/../../build/libmxnet.so(dmlc::LogMessageFatal::~LogMessageFatal()+0x28) [0x7f70067b2328]
[bt] (2) /liang_volume/SNIPER/SNIPER-mxnet/python/mxnet/../../build/libmxnet.so(void mxnet::op::BatchNormForwardImpl<mshadow::gpu, mshadow::half::half_t, float>(mshadow::Streammshadow::gpu, mxnet::OpContext const&, mxnet::op::BatchNormParam const&, std::vector<mxnet::TBlob, std::allocatormxnet::TBlob > const&, std::vector<mxnet::OpReqType, std::allocatormxnet::OpReqType > const&, std::vector<mxnet::TBlob, std::allocatormxnet::TBlob > const&, std::vector<mxnet::TBlob, std::allocatormxnet::TBlob > const&)+0x31e) [0x7f70096675fe]
[bt] (3) /liang_volume/SNIPER/SNIPER-mxnet/python/mxnet/../../build/libmxnet.so(void mxnet::op::BatchNormForward<mshadow::gpu, mshadow::half::half_t, float>(mxnet::OpContext const&, mxnet::op::BatchNormParam const&, std::vector<mxnet::TBlob, std::allocatormxnet::TBlob > const&, std::vector<mxnet::OpReqType, std::allocatormxnet::OpReqType > const&, std::vector<mxnet::TBlob, std::allocatormxnet::TBlob > const&, std::vector<mxnet::TBlob, std::allocatormxnet::TBlob > const&)+0x301) [0x7f7009667de1]
[bt] (4) /liang_volume/SNIPER/SNIPER-mxnet/python/mxnet/../../build/libmxnet.so(void mxnet::op::BatchNormComputemshadow::gpu(nnvm::NodeAttrs const&, mxnet::OpContext const&, std::vector<mxnet::TBlob, std::allocatormxnet::TBlob > const&, std::vector<mxnet::OpReqType, std::allocatormxnet::OpReqType > const&, std::vector<mxnet::TBlob, std::allocatormxnet::TBlob > const&)+0x852) [0x7f700965f492]
[bt] (5) /liang_volume/SNIPER/SNIPER-mxnet/python/mxnet/../../build/libmxnet.so(mxnet::exec::FComputeExecutor::Run(mxnet::RunContext, bool)+0x50) [0x7f700823a050]
[bt] (6) /liang_volume/SNIPER/SNIPER-mxnet/python/mxnet/../../build/libmxnet.so(+0x2559258) [0x7f700820f258]
[bt] (7) /liang_volume/SNIPER/SNIPER-mxnet/python/mxnet/../../build/libmxnet.so(mxnet::engine::ThreadedEngine::ExecuteOprBlock(mxnet::RunContext, mxnet::engine::OprBlock)+0x429) [0x7f70067ba489]
[bt] (8) /liang_volume/SNIPER/SNIPER-mxnet/python/mxnet/../../build/libmxnet.so(void mxnet::engine::ThreadedEnginePerDevice::GPUWorker<(dmlc::ConcurrentQueueType)0>(mxnet::Context, bool, mxnet::engine::ThreadedEnginePerDevice::ThreadWorkerBlock<(dmlc::ConcurrentQueueType)0>, std::shared_ptrdmlc::ManualEvent const&)+0xeb) [0x7f70067be99b]
[bt] (9) /liang_volume/SNIPER/SNIPER-mxnet/python/mxnet/../../build/libmxnet.so(std::_Function_handler<void (std::shared_ptrdmlc::ManualEvent), mxnet::engine::ThreadedEnginePerDevice::PushToExecute(mxnet::engine::OprBlock, bool)::{lambda()#3}::operator()() const::{lambda(std::shared_ptrdmlc::ManualEvent)#1}>::_M_invoke(std::_Any_data const&, std::shared_ptrdmlc::ManualEvent&&)+0x4e) [0x7f70067bec1e]
Can you give me some suggestions ?
CUDA VERSION : 9.0
CUDNN VERSION : 7.0
from sniper.
BTW, I use 4 TITAN Xp GPU with 12GB memory.
from sniper.
This seems to be a GPU memory issue. Have you checked the memory usage before it crashed?
from sniper.
If it’s memory you can reduce the concurrent jobs
from sniper.
Hey Ang,
It might be the case that your admin set the NVIDIA EXCLUSIVE_PROCESS flag that allow only one process to be ran on each gpu at a time. The demo would be simplified in the next commit to avoid such problems.
from sniper.
@mahyarnajibi Thanks, I am looking forward to your next commit.
from sniper.
Hey @Angzz ,
Can you pull the master
branch of both SNIPER and SNIPER-mxnet and recompile the MXNet using make
instead of cmake
(the README is also updated) and see if the problem continues to exist when you run the demo?
from sniper.
Feel free to re-open the issue if the update did not solve the problem.
from sniper.
Related Issues (20)
- The process of main_train.py never ends
- PyYAML yaml.load(input) Deprecation
- KeyError: 'conv_new_2_weight' HOT 1
- I have got lost in INSTALLATION HOT 1
- detection score is low HOT 3
- errors in INSTALLATION
- Changes required to run demo.py to detect all classes in openimages
- ImportError: lib/nms/gpu_nms.so: undefined symbol: __cudaPopCallConfiguration HOT 1
- Sniper HOT 1
- I got lost in the Installation. What is your configuration environment? such as the version of gcc, python, cuda, etc. HOT 2
- AssertionError: number of boxes matrix must match number of images. HOT 1
- COCO_train2017_rpn.pkl
- broken link to imagenet pretrained checkpoint HOT 1
- How to understand the resolution of pretrained dataset refered in snip paper?
- "soft" is missing in cvpr3k branch
- cannot download the pretrained model nor sniper model HOT 1
- error: ‘PyThreadState {aka struct _ts}’ has no member named ‘exc_type’; did you mean ‘curexc_type’? *type = tstate->exc_type;
- would the focus mask branch downgrade performance of original detector in single-scale training and inference
- inferences: value error HOT 1
- the download links were invalid HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from sniper.