[Mobile] Segmentation fault after repeated inference about onnxruntime HOT 5 OPEN

Hard to say without a stack trace. with symbol names

ORT will do most allocations during model initialization and the first inference. After that it's using a cache for memory so segfaults would typically be an out-of-memory scenario or bad input (e.g. input tensor is freed while ORT is using it).

If you're building from source can you build a debug version? May need to ensure the Android build doesn't strip the binary of symbols though as typically it.

Does the issue happen if you run on the Android emulator? Would be easier to debug if it did.

Another option would be to copy onnxruntime_perf_test using adb to the phone (use /data/local/tmp), along with the model, and run. you can specify the number of iterations or amount of time to run for, and it can generate dummy input data.

from onnxruntime.

Hi @skottmckay thanks for your response.

I have created an MRE in the form of a demo app that has the bug. Please check out this repo. The bug is reproducible on Android emulator, it will crash anywhere in the range of 100-1000 inference runs, which should only take a few minutes to reach. Does this help in debugging?

I would like to provide a stack trace of the crash also, but I don't know how to get that on the native layer. Any pointers you can give me for that? In any case, I appreciate the help :)

from onnxruntime.

this issue same with :
#21097

which I solved by including generated header files.
In my case, it's caused by function mapping.
maybe you can try. Hope it helps. 0x0

from onnxruntime.

@laurenspriem is it reproducible by running onnxruntime_perf_test in a shell on the emulator? If so that would rule out the issue being in the flutter plugin you're using (which we don't own).

Use adb push <file> /data/local/tmp to copy onnxruntime_perf_test and your model to /data/local/tmp. Using adb shell, chmod +x /data/local/tmp/onnxruntime_perf_test to make it executable. cd /data/local/tmp. ./onnxruntime_perf_test -I -r 2000 <model.onnx> will run the model 1000 times, generating random input that matches the model inputs. If that does not crash, most likely the issue is with the flutter plugin.

May be possible to get symbols using ndk-stack: https://developer.android.com/ndk/guides/ndk-stack.html

from onnxruntime.

I am trying to run onnxruntime_perf_test in the emulator as you suggested. However, it stops and gives me the following text back:

/onnxruntime/onnxruntime/test/onnx/TestCase.cc:705 OnnxTestCase::OnnxTestCase(const std::string &, std::unique_ptr<TestModelInfo>, double, double) test case dir doesn't exist

Any clue what is going wrong?

from onnxruntime.