Segmentation fault when -O2 used to compile application. about dyninst HOT 4 OPEN

Debugging an instrumented binary can be incredibly difficult. At this time, there is no way to rewrite DWARF information, so the stack traces tend to be unhelpful (as you can see here).

Also, have you ever seen instrumentation fail with certain optimization flags?

I can't think of a specific instance, but it could be that Dyninst is making an assumption about code layout that is incorrect for a particular optimization pass.

A few things to help us direct you:

1. Are you doing static or dynamic instrumentation?
2. What kind of instrumentation are you inserting (e.g., changing function arguments, tracing memory access, etc.)?
3. Which hardware architecture and OS?

from dyninst.

Since dyninst doesn't rewrite the DWARF information, if a function is shifted slightly or instrumented code overwrites existing instruction bytes, the debug information maps to wrong the function/source/line. If code is moved or added at a location that did not previously have code (or the PC get corrupted) then gdb fails to find any mapping and you get ???. Then since the data to unwind the stack doesn't know how to unwind the stack because the data is missing for PC location, the stack unwinding can fail and you get nonsensical stack frames.

One thing to try or confirm is the use of -fno-omit-frame-pointer when compiling which will enable the fallback stack unwinding mechanism. Often gcc is configured to include -fomit-frame-pointer at -O1 or -O2. This may allow the discovery functions for more the stack frames.

To get a sense of what shared object the crash is happening in (assuming the PC is valid at the time of the crash), you can look for the address in the list of mapped addresses from the gdb command maint info sections -all-objects.

from dyninst.

@hainest
Hi Tim,

We are using TAU and inserting TAU instrumentation with dyninst, similar to the codeCoverage example, but we are inserting our own instrumentation at the start and end of functions and an initialization at the start of the application. However, the application fails after some iterations, where it has already called the instrumented functions for some time, not at the start. The tested case is static instrumentation.
The architecture is 2 x Xeon 4410T with Ubuntu 22.

from dyninst.

@kupsch

With the -fno-omit-frame-pointer flag, the bug I was seeing is not triggered anymore (at least the times I tried), but another appeared and I can see some execution information.

Thank for the help, I will try to debug as I can see some symbols now.

from dyninst.