Git Product home page Git Product logo

gotcha's Introduction

GOTCHA v1.0.6

GOTCHA Build and Test Coverage Status Documentation Status

Gotcha is a library that wraps functions. Tools can use gotcha to install hooks into other libraries, for example putting a wrapper function around libc's malloc.
It is similar to LD_PRELOAD, but operates via a programmable API. This enables easy methods of accomplishing tasks like code instrumentation or wholesale replacement of mechanisms in programs without disrupting their source code.

Quick Start

Building Gotcha is trivial. In the root directory of the repo

mkdir build
cd build
cmake -DCMAKE_INSTALL_PREFIX=<where you want to install GOTCHA> ../
make install

Usage is fairly simple. For us to wrap a function, we need to know its name, what you want it wrapped with (the wrapper), and we need to give you some ability to call the function you wrapped (wrappee). Gotcha works on triplets containing this information. We have small sample uses, but the standard workflow looks like

  #include <gotcha/gotcha.h>
  static gotcha_wrappee_handle_t wrappee_puts_handle;
  static int puts_wrapper(const char* str); //this is the declaration of your wrapper
  static gotcha_wrappee_handle_t wrappee_fputs_handle;
  static int fputs_wrapper(const char* str, FILE* f);
  struct gotcha_binding_t wrap_actions [] = {
    { "puts", puts_wrapper, &wrappee_puts_handle },
    { "fputs", fputs_wrapper, &wrappee_fputs_handle },
  };
  int init_mytool(){
    gotcha_wrap(wrap_actions, sizeof(wrap_actions)/sizeof(struct gotcha_binding_t), "my_tool_name");
  }
  static int fputs_wrapper(const char* str, FILE* f){
    // insert clever tool logic here
    typeof(&fputs_wrapper) wrappee_fputs = gotcha_get_wrappee(wrappee_fputs_handle); // get my wrappee from Gotcha
    return wrappee_fputs(str, f); //wrappee_fputs was directed to the original fputs by gotcha_wrap
  }

Building your tool changes little, you just need to add the prefix you installed Gotcha to your include directories, the location the library was installed (default is <that_prefix>/lib) to your library search directories (-L...), and link libgotcha.so (-lgotcha) with your tool. Very often this becomes "add -lgotcha to your link line," and nicer CMake integration is coming down the pipe.

A more advanced example can be seen in the GOTCHA-tracer project. This example shows how to implement a simple tracer with GOTCHA, including linking to GOTCHA through CMake and testing GOTCHA through direct invocation and using LD_PRELOAD.

That should represent all the work your application needs to do to use Gotcha.

Contact/Legal

The license is LGPL.

Primary contact/Lead developer

Hariharan Devarajan ([email protected])

gotcha's People

Contributors

alehaa avatar bertwesarg avatar bkmgit avatar bwelton avatar daboehme avatar davidpoliakoff avatar davidpoliakoff-backup avatar gonsie avatar hariharan-devarajan avatar jrmadsen avatar mplegendre avatar mxz297 avatar tonyhutter avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

gotcha's Issues

Wraps different symbol version

GOTCHA currently does not honor symbol versions used in glibc.

Case at hands: I wanted to wrap the pthread_cond_* family of functions. Though in glibc there are two symbol versions:

  • GLIBC_2.2.5
  • GLIBC_2.3.2

Unfortunately, for glibc 2.31 (Ubuntu 20.04), GOTCHA wraps pthread_cond_init to __pthread_cond_init_2_0 but pthread_cond_broadcast to __pthread_cond_broadcast. I.e., one to the old ABI and the other to the new ABI which breaks at runtime.

The reason seems to be that the order of the symbols in libpthread-2.31.so is:

   267: 0000000000010610    31 FUNC    GLOBAL DEFAULT   16 pthread_cond_init@GLIBC_2.2.5
   268: 000000000000ed90    58 FUNC    GLOBAL DEFAULT   16 pthread_cond_init@@GLIBC_2.3.2
   205: 0000000000010290   894 FUNC    GLOBAL DEFAULT   16 pthread_cond_broadcast@@GLIBC_2.3.2
   206: 00000000000107c0   101 FUNC    GLOBAL DEFAULT   16 pthread_cond_broadcast@GLIBC_2.2.5

Caliper Segmentation fault on Xeon Phi Knight landing

I annotate LULESH with caliper macros. LULESH experienced segmentation fault with the following call stack information (More details, please refer to this caliper issue) :

#0 0x00002b06b0cc005f in ?? ()
#1 0x00002b06b11c8041 in (anonymous namespace)::cali_pthread_create_wrapper (thread=0x7ffff8a1d0d8, attr=0x7ffff8a1d338,
fn=0x2b06b0d74eb0 <_INTERNAL_26_______src_z_Linux_util_cpp_313effc4::__kmp_launch_worker(void*)>, arg=0x2b06b490e400)
at /work/03915/taoncsu/stampede2/Caliper/src/services/pthread/PthreadService.cpp:83
#2 0x00002b06b0d759c9 in __kmp_create_worker (gtid=-123612968, th=0x7ffff8a1d338, stack_size=47307741232816) at ../../src/z_Linux_util.cpp:878
#3 0x00002b06b0d42579 in __kmp_allocate_thread (root=0x7ffff8a1d0d8, team=0x7ffff8a1d338, new_tid=-1323532624)
at ../../src/kmp_runtime.cpp:4521
#4 0x00002b06b0d472a7 in __kmp_allocate_team (root=0x7ffff8a1d0d8, new_nproc=-123612360, max_nproc=-1323532624, new_proc_bind=30692880,
new_icvs=0x0, argc=16, master=0x2b06b490f200) at ../../src/kmp_runtime.cpp:5138
#5 0x00002b06b0d45a18 in __kmp_fork_call (loc=0x7ffff8a1d0d8, gtid=-123612360, call_context=(unknown: 2971434672), argc=30692880,
microtask=0x0, invoker=0x10, ap=0x7ffff8a1d7b0) at ../../src/kmp_runtime.cpp:2150
#6 0x00002b06b0d1be2a in __kmpc_fork_call (loc=0x7ffff8a1d0d8, argc=-123612360,
microtask=0x2b06b11c7eb0 <(anonymous namespace)::thread_wrapper(void*)>) at ../../src/kmp_csupport.cpp:328
#7 0x000000000040434d in main ()

Add Indirection Interface

Work has already started on this in #42, but in order to work with libdl, we need to add indirection, so that we can hand different "underlying functions" to different libraries.

Cannot wrap IFUNC functions

Gotcha is not correctly handling wrapping of functions with IFUNC relocations. We need a different procedure for calcluating the wrappee function of these.

Argument-free ABI-agnostic wrappers

Many tools want to do simple wrappers which don't care about the arguments of the functions being wrapped. Caring about arguments is a source of ABI problems. Gotcha should support as a configuration option the creation of such wrappers rather than the current argument-requiring ones. A common use case would be MPI wrappers.

Issue originally reported by @jmellorcrummey , who can feel free to tell me if we got anything wrong from those requirements, or if there's anything else we'd need to support

Add User Manual

Right now we have minimal API documentation, and some examples. A more in-depth guide to the functionality of Gotcha would be beneficial, especially as things like the configuration interface come online

symver test fails to build

Consider updating the test in https://github.com/LLNL/GOTCHA/tree/develop/test/symver as it fails to build with newer versions of GCC

cd /builddir/build/BUILD/GOTCHA-1.0.5/redhat-linux-build/test/symver && /usr/bin/cmake -E cmake_link_script CMakeFiles/retX.dir/link.txt --verbose=1
/usr/bin/gcc -fPIC -O2 -flto=auto -ffat-lto-objects -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Werror=implicit-function-declaration -Werror=implicit-int -Wp,-U_FORTIFY_SOURCE,-D_FORTIFY_SOURCE=3 -Wp,-D_GLIBCXX_ASSERTIONS -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fstack-protector-strong -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1  -mbranch-protection=standard -fasynchronous-unwind-tables -fstack-clash-protection -fno-omit-frame-pointer -mno-omit-leaf-frame-pointer -Wl,--version-script=/builddir/build/BUILD/GOTCHA-1.0.5/test/symver/symver.map -shared -Wl,-soname,libretX.so -o libretX.so CMakeFiles/retX.dir/retX_old.c.o CMakeFiles/retX.dir/retX_new.c.o 
/builddir/build/BUILD/GOTCHA-1.0.5/src/gotcha_auxv.c: In function ‘get_vdso_from_auxv’:
/builddir/build/BUILD/GOTCHA-1.0.5/src/gotcha_auxv.c:108:21: warning: ‘vdso_dynamic’ may be used uninitialized [-Wmaybe-uninitialized]
  108 |       if (m->l_addr + vdso_dynamic == (ElfW(Addr))m->l_ld) {
      |           ~~~~~~~~~~^~~~~~~~~~~~~~
/builddir/build/BUILD/GOTCHA-1.0.5/src/gotcha_auxv.c:93:14: note: ‘vdso_dynamic’ was declared here
   93 |   ElfW(Addr) vdso_dynamic;
      |              ^~~~~~~~~~~~
gmake[2]: Leaving directory '/builddir/build/BUILD/GOTCHA-1.0.5/redhat-linux-build'
[ 22%] Built target num3
{standard input}: Assembler messages:
{standard input}: Error: invalid attempt to declare external version name as default in symbol `retX@@GOTCHA_2'

Full log https://kojipkgs.fedoraproject.org//work/tasks/3462/109123462/build.log

Error Handling interface

@mplegendre : usual "this can be reassigned to me" rules apply, I'll stop prefixing issues with this.

Right now if things go wrong, users will be aware in only a limited sense. We need to think about what users might want to know about their calls into GOTCHA, how they might want to ask for it, and provide them with useful info. I won't prejudice you with designs in advance.

End-to-end Testing

Right now we have the following tests

  1. Minimal (wrap a symbol, call it)
  2. dlopen (fails)
  3. A unit test suite

Obvious todo's

  1. Stacking

What else do we need?

Don't rely on the bindings list in gotcha_wrap() being static!

Turns out Gotcha internally keeps pointers to the user-provided gotcha_binding_t structs in gotcha_wrap() and happily accesses them later on, e.g. when someone calls dl_open(). That was new to me! Not good when the original buffer is long since gone, as that leads to issues like LLNL/Caliper#288. Would be great if Gotcha would copy the user-provided list instead.

Transparently handle main wrapping

As David discovered, you can't wrap "main" in LD_PRELOADED binary as a user might expect. We should watch for attempts to wrap main, and translate them into the correct wrapping of libc_start_main.

C++ Transient interface

#50 and #51 discuss transient interfaces and C ones. Obviously it would be nice for the C++ interface to make the transient interface nicer, much as the simple C++ interface makes the rest of the other system nicer.

On closing those two bugs, check whether we need to update the C++ interface to make the transient features nicer

Exposing localized/faster wrapping

@mplegendre

Non-urgent design question: #21 adds a function

void setLibraryFilterFunc(int(*new_func)(struct link_map*));

Which sets a function which GOTCHA uses to decided whether to do wrapping in a given library (there are also conveniences like re-setting that to "always," and a way to make it work on libraries with a certain name).

Right now this is used in GOTCHA's internal handling of libdl (which may be faulty itself). Do we want to expose this ability to users in some form? I could imagine users wanting to say "hey, the thing I care about is an MPI library, don't bother with things that aren't MPI libraries."

I'm probably going to start adding these discussion issues on Github as I think of them, I just want us to think about these things and not forget them, but they should be treated as the lowest priority of issues in the project.

Build fails on Cray: attempted static link of dynamic object

Build fails on platforms that prefer static linking, here Cray:

Linking C executable autotee_test
cd /global/homes/b/boehme3/src/GOTCHA/build-cori/src/example/autotee && /usr/bin/cmake -E cmake_link_script CMakeFiles/autotee_test.dir/link.txt --verbose=1
/opt/cray/pe/craype/2.5.7/bin/cc      CMakeFiles/autotee_test.dir/test_autotee.c.o  -o autotee_test -rdynamic libautotee.so ../../libgotcha.so.0.0.1 -Wl,-rpath,/global/homes/b/boehme3/src/GOTCHA/build-cori/src/example/autotee:/global/homes/b/boehme3/src/GOTCHA/build-cori/src 
/usr/bin/ld: attempted static link of dynamic object 'libautotee.so'
src/example/autotee/CMakeFiles/autotee_test.dir/build.make:90: recipe for target 'src/example/autotee/autotee_test' failed
make[2]: *** [src/example/autotee/autotee_test] Error 1

Add configuration interface

Right now Gotcha simply takes in a list of "rewrites" to do. However, people might want to tell us more about how things can be done. In terms of how things are configured we need

  • Replacement of "tool names" as naive things we mostly ignore with "tools" as possibly nested namespaces on which rewrites can be configured
  • Semantics for the nesting of those configurations (if "outer/inner" has the same setting as "outer" does it overwrite the behavior in "outer"? Does it do so silently?)

We also need to have these configurations be useful, some things to configure could include

  • Specification of libraries in which to find underlying functions. Our issues with Intel OpenMP (#40) (seem to) stem from us redirecting their pthread_create to a wrapped system one.
  • Statements of intent. Some tools might use us to wrap functions which the tool then calls, others may want to replace calls wholesale (not calling the underlying function). Letting us know that could let us better debug tool stacking problems
  • Statements of priority: if multiple tools wrap malloc, what order should their wrappers be called in?

Especially the second list is likely to be incomplete, and will require discussion

Figure out how to handle dlsym with RTLD_NEXT

The RTLD_NEXT parameter to dlsym has at least two issues with gotcha:

  • When the symbol being passed to dlsym isn't wrapped: the behavior of dlsym with RTLD_NEXT changes based on the caller library. Wrapping dlsym will break this. We need to simulate the bottom level call as if coming from the top level caller.

  • Calling dlsym with RTLD_NEXT can return a different "bottom-level symbol" than without. If the symbol is wrapped, should we return the same wrapper function with and without RTLD_NEXT? Should we support different stacks of wrappers?

Make types match gotcha documentation

I'm working on the gotcha documentation, and writing the types as they will be. We'll need to modify the gotcha source to match.

Specifically, gotcha_binding_t should be:

typedef struct {
const char *name;
void *wrapper_pointer;
gotcha_wrappee_handle_t function_handle;
} gotcha_binding_t;

and gotcha_error_t:

typedef enum {
...
} gotcha_error_t;

unable to wrap functions in an init constructor

Is it possible to wrap a function in an init constructor? I've tried
a couple things and I can wrap a function, but I can't get it to apply
in another library's init ctor.

(1) The "pure gotcha" attempt.

I tried to wrap 'sigprocmask' (it's part of a cycle of deadlock
involving tcmalloc, libunwind and libmonitor, but that's another
story).

I created a shared library libmygotcha.so containing:

typedef int (* sigprocmask_fptr) (int, void *, void *);

static gotcha_wrappee_handle_t sigprocmask_handle;

static sigprocmask_fptr real_sigprocmask;

static int
sigprocmask_wrap(int how, void * set, void * old_set)
{
    ...
}

static gotcha_binding_t bindings [] = {
    { "sigprocmask", sigprocmask_wrap, &sigprocmask_handle },
};

void __attribute__ ((constructor))
monitor_init_ctor(void)
{
    gotcha_wrap(bindings, 1, "mymonitor");
    real_sigprocmask = (sigprocmask_fptr) gotcha_get_wrappee(sigprocmask_handle);
}

Put the above into a shared library and LD_PRELOAD it. Run an app
that uses its own shared library and calls sigprocmask from both the
main program and the init ctor for a shared library.

I can tell that the above code gets run, and indeed it wraps
sigprocmask after main, but not in the library's init ctor.

Ok, so maybe it's a matter of timing. There are two init ctors in
play, if the other one gets run first, then maybe the wrapping happens
too late.


(2) The "hybrid" attempt.

Override __libc_start_main by the traditional LD_PRELOAD and
dlsym(RTLD_NEXT) method. (This is what libmonitor does for the
dynamic case.)

Run gotcha_wrap from inside the __libc_start_main override. Same
result. The wrapping happens and applies after main, but not in the
init ctor.

The hope was that by wrapping __libc_start_main maybe the gotcha
wrapping happens before the init ctor. But no.


So, is there a way to call gotcha_wrap where it will happen before
init constructors?

The traditional LD_PRELOAD and dlsym(RTLD_NEXT) will do this.

This is important for hpctoolkit. There are libraries that start
pthreads (and other functions) inside init constructors and we need to
catch them.

ping @mplegendre and @DavidPoliakoff

Build fails starting with glibc v2.34

As of version 2.34 of glibc (which is the latest as of Oct 2021), GOTCHA fails to build. The problem is that GOTCHA uses _dl_sym and in v2.34, this is no longer exported.

We don't wrap the main function unless it's called "main"

In our translations.c, we have the ability to wrap main, in spite of the weirdness of libc_start_main. It would be nice if we had some way to instead of wrapping main, wrap the first argument to libc_start_main, something I think we can do in Gotcha.

gotcha_atoi too good

@mplegendre

In the testing work, I've gotten builds with libc and gotcha-libc. The libc one fails when I try --52 (0), but gotchas does not. We're the ones breaking the standard, they're the ones who are bad at math.

Do we want to fix this?

Unwrapping gotcha

Can y'all just add a gotcha_unwrap function? And then add a test to ensure multiple paired gotcha_wrap and gotcha_unwrap work?

I have a C++ struct that handles the generation of a gotcha wrapper and it uses some scoping to wrap and unwrap the function(s). I have been able to do a single scope w/o issue. The problem is re-wrapping the function. Right now I am segfaulting on the second iteration. At some point, I was able to get through two iterations before segfaulting.

I am fairly convinced by now that I should not be the one that has to figure out how to do this reliably. malloc has a free, a constructor has a destructor, so gotcha_wrap should have a gotcha_unwrap.

Here are the details that lead to this issue:

Single Scope Code

double before = exp(2.0);
printf("\n[%i]> BEFORE: exp(2.0) = %f\n\n", rank, before);
printf("[%i]> BEGIN SCOPED GOTCHA....\n", rank);
{
     // generates the wrapper until out of scope
     auto wrapper = generate_gotcha(&exp); 
     do_exp_work();
}
printf("[%i]> END SCOPED GOTCHA....\n", rank);
double after = exp(2.0);
printf("\n[%i]> AFTER: exp(2.0) = %f\n\n", rank, after);

Single Scope Output

[0]> BEFORE: exp(2.0) = 7.389056

[0]> BEGIN SCOPED GOTCHA....

> [do_exp_work@'../examples/ex-gotcha/ex_gotcha_lib.cpp':61] ...

	[itr:  0]> float  expf(  2.000) =                7.389
	[itr:  1]> float  expf(  2.250) =               16.877
	[itr:  2]> float  expf(  2.750) =               32.519

	executing modified exp function :                2.000...	[itr:  0]> double exp (  2.000) =                7.389
	executing modified exp function :                2.250...	[itr:  1]> double exp (  2.250) =               16.877
	executing modified exp function :                2.750...	[itr:  2]> double exp (  2.750) =               32.519

[0]> END SCOPED GOTCHA....

[0]> AFTER: exp(2.0) = 7.389056

As you can see, exp(2.0) = 7.389056 both before and after the gotcha wrapper, which seems to indicate that I am doing the unwrapping successfully.

Multiple Scope Code

If I tweak this to:

for(int i = 0; i < 2; ++i)
{
    double before = exp(2.0);
    printf("\n[%i]> BEFORE: exp(2.0) = %f\n\n", rank, before);
    printf("[%i]> BEGIN SCOPED GOTCHA....\n", rank);
    {
         // generates the wrapper until out of scope
         auto wrapper = generate_gotcha(&exp); 
         do_exp_work();
    }
    printf("[%i]> END SCOPED GOTCHA....\n", rank);
    double after = exp(2.0);
    printf("\n[%i]> AFTER: exp(2.0) = %f\n\n", rank, after);
}

Multiple Scope Output

[0]> BEFORE: exp(2.0) = 7.389056

[0]> BEGIN SCOPED GOTCHA....

> [do_exp_work@'../examples/ex-gotcha/ex_gotcha_lib.cpp':61] ...

	[itr:  0]> float  expf(  2.000) =                7.389
	[itr:  1]> float  expf(  2.250) =               16.877
	[itr:  2]> float  expf(  2.750) =               32.519

	executing modified exp function :                2.000...	[itr:  0]> double exp (  2.000) =                7.389
	executing modified exp function :                2.250...	[itr:  1]> double exp (  2.250) =               16.877
	executing modified exp function :                2.750...	[itr:  2]> double exp (  2.750) =               32.519

[0]> END SCOPED GOTCHA....

[0]> AFTER: exp(2.0) = 7.389056


[0]> BEFORE: exp(2.0) = 7.389056

[0]> BEGIN SCOPED GOTCHA....

> [do_exp_work@'../examples/ex-gotcha/ex_gotcha_lib.cpp':61] ...

	[itr:  0]> float  expf(  2.000) =                7.389
	[itr:  1]> float  expf(  2.250) =               16.877
	[itr:  2]> float  expf(  2.750) =               32.519

Segmentation fault

Multiple Scope Debug Output

Here is the output debugger output with GOTCHA_DEBUG enabled:

[0]> BEFORE: exp(2.0) = 7.389056

[0]> BEGIN SCOPED GOTCHA....
[10727/10727][gotcha_utils.c:45] - Gotcha debug initialized at level 4
[10727/10727][gotcha.c:276] - User called gotcha_wrap for tool gotcha with 2 bindings
	0: dlopen will map to 0x7ff2e9008773
	1: dlsym will map to 0x7ff2e90088e8
[10727/10727][gotcha.c:284] - Initializing 2 user binding entries to NULL
[10727/10727][gotcha.c:36] - Updating binding address pointer at 0x7ff2e920b2d0 to (nil)
[10727/10727][gotcha.c:36] - Updating binding address pointer at 0x7ff2e920b2d8 to (nil)
[10727/10727][tool.c:66] - Found no existing tool with name gotcha
[10727/10727][tool.c:82] - Created new tool gotcha
[10727/10727][gotcha.c:300] - Moved current_generation to 1 in gotcha_wrap
[10727/10727][gotcha.c:302] - Creating internal binding data structures and adding binding to tool
[10727/10727][tool.c:134] - Created new binding table of size 2 for tool gotcha
[10727/10727][gotcha.c:309] - Processing 2 bindings
[10727/10727][gotcha.c:143] - gotcha_rewrite_wrapper_orders for binding dlopen in tool gotcha of priority -1
[10727/10727][gotcha.c:150] - Adding new entry for dlopen to hash table
[10727/10727][gotcha.c:315] - Symbol dlopen needs lookup operation
[10727/10727][gotcha.c:56] - Looking up exported symbols for dlopen
[10727/10727][gotcha.c:60] - Creating new library object for [EMPTY]
[10727/10727][gotcha.c:69] - Searching for exported symbols in [EMPTY]
[10727/10727][gotcha.c:78] - Checking GNU hash for dlopen in [EMPTY]
[10727/10727][gotcha.c:90] - dlopen not found in [EMPTY]
[10727/10727][gotcha.c:60] - Creating new library object for linux-vdso.so.1
[10727/10727][gotcha.c:65] - Skipping VDSO library at 0x7fffe2d5d000 with name linux-vdso.so.1
[10727/10727][gotcha.c:60] - Creating new library object for /home/timemory/build-docker-debug/libex_gotcha_lib.so
[10727/10727][gotcha.c:69] - Searching for exported symbols in /home/timemory/build-docker-debug/libex_gotcha_lib.so
[10727/10727][gotcha.c:78] - Checking GNU hash for dlopen in /home/timemory/build-docker-debug/libex_gotcha_lib.so
[10727/10727][gotcha.c:90] - dlopen not found in /home/timemory/build-docker-debug/libex_gotcha_lib.so
[10727/10727][gotcha.c:60] - Creating new library object for /home/timemory/build-docker-debug/libgotcha.so.1
[10727/10727][gotcha.c:69] - Searching for exported symbols in /home/timemory/build-docker-debug/libgotcha.so.1
[10727/10727][gotcha.c:78] - Checking GNU hash for dlopen in /home/timemory/build-docker-debug/libgotcha.so.1
[10727/10727][gotcha.c:90] - dlopen not found in /home/timemory/build-docker-debug/libgotcha.so.1
[10727/10727][gotcha.c:60] - Creating new library object for /lib/x86_64-linux-gnu/libpthread.so.0
[10727/10727][gotcha.c:69] - Searching for exported symbols in /lib/x86_64-linux-gnu/libpthread.so.0
[10727/10727][gotcha.c:78] - Checking GNU hash for dlopen in /lib/x86_64-linux-gnu/libpthread.so.0
[10727/10727][gotcha.c:84] - Checking ELF hash for dlopen in /lib/x86_64-linux-gnu/libpthread.so.0
[10727/10727][gotcha.c:90] - dlopen not found in /lib/x86_64-linux-gnu/libpthread.so.0
[10727/10727][gotcha.c:60] - Creating new library object for /usr/lib/x86_64-linux-gnu/libstdc++.so.6
[10727/10727][gotcha.c:69] - Searching for exported symbols in /usr/lib/x86_64-linux-gnu/libstdc++.so.6
[10727/10727][gotcha.c:78] - Checking GNU hash for dlopen in /usr/lib/x86_64-linux-gnu/libstdc++.so.6
[10727/10727][gotcha.c:90] - dlopen not found in /usr/lib/x86_64-linux-gnu/libstdc++.so.6
[10727/10727][gotcha.c:60] - Creating new library object for /lib/x86_64-linux-gnu/libm.so.6
[10727/10727][gotcha.c:69] - Searching for exported symbols in /lib/x86_64-linux-gnu/libm.so.6
[10727/10727][gotcha.c:78] - Checking GNU hash for dlopen in /lib/x86_64-linux-gnu/libm.so.6
[10727/10727][gotcha.c:84] - Checking ELF hash for dlopen in /lib/x86_64-linux-gnu/libm.so.6
[10727/10727][gotcha.c:90] - dlopen not found in /lib/x86_64-linux-gnu/libm.so.6
[10727/10727][gotcha.c:60] - Creating new library object for /lib/x86_64-linux-gnu/libgcc_s.so.1
[10727/10727][gotcha.c:69] - Searching for exported symbols in /lib/x86_64-linux-gnu/libgcc_s.so.1
[10727/10727][gotcha.c:78] - Checking GNU hash for dlopen in /lib/x86_64-linux-gnu/libgcc_s.so.1
[10727/10727][gotcha.c:90] - dlopen not found in /lib/x86_64-linux-gnu/libgcc_s.so.1
[10727/10727][gotcha.c:60] - Creating new library object for /lib/x86_64-linux-gnu/libc.so.6
[10727/10727][gotcha.c:69] - Searching for exported symbols in /lib/x86_64-linux-gnu/libc.so.6
[10727/10727][gotcha.c:78] - Checking GNU hash for dlopen in /lib/x86_64-linux-gnu/libc.so.6
[10727/10727][gotcha.c:84] - Checking ELF hash for dlopen in /lib/x86_64-linux-gnu/libc.so.6
[10727/10727][gotcha.c:90] - dlopen not found in /lib/x86_64-linux-gnu/libc.so.6
[10727/10727][gotcha.c:60] - Creating new library object for /lib64/ld-linux-x86-64.so.2
[10727/10727][gotcha.c:69] - Searching for exported symbols in /lib64/ld-linux-x86-64.so.2
[10727/10727][gotcha.c:78] - Checking GNU hash for dlopen in /lib64/ld-linux-x86-64.so.2
[10727/10727][gotcha.c:84] - Checking ELF hash for dlopen in /lib64/ld-linux-x86-64.so.2
[10727/10727][gotcha.c:90] - dlopen not found in /lib64/ld-linux-x86-64.so.2
[10727/10727][gotcha.c:106] - Symbol dlopen was found in program
[10727/10727][gotcha.c:318] - Stashing dlopen in notfound_binding table to re-lookup on dlopens
[10727/10727][gotcha.c:325] - Symbol dlopen needs binding from application
[10727/10727][gotcha.c:143] - gotcha_rewrite_wrapper_orders for binding dlsym in tool gotcha of priority -1
[10727/10727][gotcha.c:150] - Adding new entry for dlsym to hash table
[10727/10727][gotcha.c:315] - Symbol dlsym needs lookup operation
[10727/10727][gotcha.c:56] - Looking up exported symbols for dlsym
[10727/10727][gotcha.c:69] - Searching for exported symbols in [EMPTY]
[10727/10727][gotcha.c:78] - Checking GNU hash for dlsym in [EMPTY]
[10727/10727][gotcha.c:90] - dlsym not found in [EMPTY]
[10727/10727][gotcha.c:65] - Skipping VDSO library at 0x7fffe2d5d000 with name linux-vdso.so.1
[10727/10727][gotcha.c:69] - Searching for exported symbols in /home/timemory/build-docker-debug/libex_gotcha_lib.so
[10727/10727][gotcha.c:78] - Checking GNU hash for dlsym in /home/timemory/build-docker-debug/libex_gotcha_lib.so
[10727/10727][gotcha.c:90] - dlsym not found in /home/timemory/build-docker-debug/libex_gotcha_lib.so
[10727/10727][gotcha.c:69] - Searching for exported symbols in /home/timemory/build-docker-debug/libgotcha.so.1
[10727/10727][gotcha.c:78] - Checking GNU hash for dlsym in /home/timemory/build-docker-debug/libgotcha.so.1
[10727/10727][gotcha.c:90] - dlsym not found in /home/timemory/build-docker-debug/libgotcha.so.1
[10727/10727][gotcha.c:69] - Searching for exported symbols in /lib/x86_64-linux-gnu/libpthread.so.0
[10727/10727][gotcha.c:78] - Checking GNU hash for dlsym in /lib/x86_64-linux-gnu/libpthread.so.0
[10727/10727][gotcha.c:84] - Checking ELF hash for dlsym in /lib/x86_64-linux-gnu/libpthread.so.0
[10727/10727][gotcha.c:90] - dlsym not found in /lib/x86_64-linux-gnu/libpthread.so.0
[10727/10727][gotcha.c:69] - Searching for exported symbols in /usr/lib/x86_64-linux-gnu/libstdc++.so.6
[10727/10727][gotcha.c:78] - Checking GNU hash for dlsym in /usr/lib/x86_64-linux-gnu/libstdc++.so.6
[10727/10727][gotcha.c:90] - dlsym not found in /usr/lib/x86_64-linux-gnu/libstdc++.so.6
[10727/10727][gotcha.c:69] - Searching for exported symbols in /lib/x86_64-linux-gnu/libm.so.6
[10727/10727][gotcha.c:78] - Checking GNU hash for dlsym in /lib/x86_64-linux-gnu/libm.so.6
[10727/10727][gotcha.c:84] - Checking ELF hash for dlsym in /lib/x86_64-linux-gnu/libm.so.6
[10727/10727][gotcha.c:90] - dlsym not found in /lib/x86_64-linux-gnu/libm.so.6
[10727/10727][gotcha.c:69] - Searching for exported symbols in /lib/x86_64-linux-gnu/libgcc_s.so.1
[10727/10727][gotcha.c:78] - Checking GNU hash for dlsym in /lib/x86_64-linux-gnu/libgcc_s.so.1
[10727/10727][gotcha.c:90] - dlsym not found in /lib/x86_64-linux-gnu/libgcc_s.so.1
[10727/10727][gotcha.c:69] - Searching for exported symbols in /lib/x86_64-linux-gnu/libc.so.6
[10727/10727][gotcha.c:78] - Checking GNU hash for dlsym in /lib/x86_64-linux-gnu/libc.so.6
[10727/10727][gotcha.c:84] - Checking ELF hash for dlsym in /lib/x86_64-linux-gnu/libc.so.6
[10727/10727][gotcha.c:90] - dlsym not found in /lib/x86_64-linux-gnu/libc.so.6
[10727/10727][gotcha.c:69] - Searching for exported symbols in /lib64/ld-linux-x86-64.so.2
[10727/10727][gotcha.c:78] - Checking GNU hash for dlsym in /lib64/ld-linux-x86-64.so.2
[10727/10727][gotcha.c:84] - Checking ELF hash for dlsym in /lib64/ld-linux-x86-64.so.2
[10727/10727][gotcha.c:90] - dlsym not found in /lib64/ld-linux-x86-64.so.2
[10727/10727][gotcha.c:106] - Symbol dlsym was found in program
[10727/10727][gotcha.c:318] - Stashing dlsym in notfound_binding table to re-lookup on dlopens
[10727/10727][gotcha.c:325] - Symbol dlsym needs binding from application
[10727/10727][gotcha.c:262] - Searching all callsites for 2 bindings
[10727/10727][gotcha.c:219] - Setting library [EMPTY] GOT table from 0x55695ce44000 to +4096 to writeable
[10727/10727][gotcha.c:219] - Setting library /home/timemory/build-docker-debug/libex_gotcha_lib.so GOT table from 0x7ff2e9457000 to +24576 to writeable
[10727/10727][gotcha.c:219] - Setting library /home/timemory/build-docker-debug/libgotcha.so.1 GOT table from 0x7ff2e920b000 to +4096 to writeable
[10727/10727][gotcha.c:219] - Setting library /lib/x86_64-linux-gnu/libpthread.so.0 GOT table from 0x7ff2e8ffc000 to +4096 to writeable
[10727/10727][gotcha.c:219] - Setting library /usr/lib/x86_64-linux-gnu/libstdc++.so.6 GOT table from 0x7ff2e8ddc000 to +24576 to writeable
[10727/10727][gotcha.c:219] - Setting library /lib/x86_64-linux-gnu/libm.so.6 GOT table from 0x7ff2e8a01000 to +4096 to writeable
[10727/10727][gotcha.c:219] - Setting library /lib/x86_64-linux-gnu/libgcc_s.so.1 GOT table from 0x7ff2e8663000 to +4096 to writeable
[10727/10727][gotcha.c:219] - Setting library /lib/x86_64-linux-gnu/libc.so.6 GOT table from 0x7ff2e8446000 to +4096 to writeable
[10727/10727][gotcha.c:219] - Setting library /lib64/ld-linux-x86-64.so.2 GOT table from 0x7ff2e9682000 to +4096 to writeable
[10727/10727][gotcha.c:340] - Could not find bindings for 2 / 2 functions
[10727/10727][gotcha.c:276] - User called gotcha_wrap for tool exp with 1 bindings
	0: exp will map to 0x55695cb606c1
[10727/10727][gotcha.c:284] - Initializing 1 user binding entries to NULL
[10727/10727][gotcha.c:36] - Updating binding address pointer at 0x55695ce46d08 to (nil)
[10727/10727][tool.c:66] - Found no existing tool with name exp
[10727/10727][tool.c:82] - Created new tool exp
[10727/10727][gotcha.c:300] - Moved current_generation to 2 in gotcha_wrap
[10727/10727][gotcha.c:302] - Creating internal binding data structures and adding binding to tool
[10727/10727][tool.c:134] - Created new binding table of size 1 for tool exp
[10727/10727][gotcha.c:309] - Processing 1 bindings
[10727/10727][gotcha.c:143] - gotcha_rewrite_wrapper_orders for binding exp in tool exp of priority -1
[10727/10727][gotcha.c:150] - Adding new entry for exp to hash table
[10727/10727][gotcha.c:315] - Symbol exp needs lookup operation
[10727/10727][gotcha.c:56] - Looking up exported symbols for exp
[10727/10727][gotcha.c:69] - Searching for exported symbols in [EMPTY]
[10727/10727][gotcha.c:78] - Checking GNU hash for exp in [EMPTY]
[10727/10727][gotcha.c:90] - exp not found in [EMPTY]
[10727/10727][gotcha.c:65] - Skipping VDSO library at 0x7fffe2d5d000 with name linux-vdso.so.1
[10727/10727][gotcha.c:69] - Searching for exported symbols in /home/timemory/build-docker-debug/libex_gotcha_lib.so
[10727/10727][gotcha.c:78] - Checking GNU hash for exp in /home/timemory/build-docker-debug/libex_gotcha_lib.so
[10727/10727][gotcha.c:90] - exp not found in /home/timemory/build-docker-debug/libex_gotcha_lib.so
[10727/10727][gotcha.c:69] - Searching for exported symbols in /home/timemory/build-docker-debug/libgotcha.so.1
[10727/10727][gotcha.c:78] - Checking GNU hash for exp in /home/timemory/build-docker-debug/libgotcha.so.1
[10727/10727][gotcha.c:90] - exp not found in /home/timemory/build-docker-debug/libgotcha.so.1
[10727/10727][gotcha.c:69] - Searching for exported symbols in /lib/x86_64-linux-gnu/libpthread.so.0
[10727/10727][gotcha.c:78] - Checking GNU hash for exp in /lib/x86_64-linux-gnu/libpthread.so.0
[10727/10727][gotcha.c:84] - Checking ELF hash for exp in /lib/x86_64-linux-gnu/libpthread.so.0
[10727/10727][gotcha.c:90] - exp not found in /lib/x86_64-linux-gnu/libpthread.so.0
[10727/10727][gotcha.c:69] - Searching for exported symbols in /usr/lib/x86_64-linux-gnu/libstdc++.so.6
[10727/10727][gotcha.c:78] - Checking GNU hash for exp in /usr/lib/x86_64-linux-gnu/libstdc++.so.6
[10727/10727][gotcha.c:90] - exp not found in /usr/lib/x86_64-linux-gnu/libstdc++.so.6
[10727/10727][gotcha.c:69] - Searching for exported symbols in /lib/x86_64-linux-gnu/libm.so.6
[10727/10727][gotcha.c:78] - Checking GNU hash for exp in /lib/x86_64-linux-gnu/libm.so.6
[10727/10727][gotcha.c:100] - Symbol exp found in /lib/x86_64-linux-gnu/libm.so.6 at 0x7ff2e86731f0
[10727/10727][gotcha.c:46] - Updating binding address pointer at 0x55695e02cd18 to 0x7ff2e86731f0
[10727/10727][gotcha.c:325] - Symbol exp needs binding from application
[10727/10727][gotcha.c:262] - Searching all callsites for 1 bindings
[10727/10727][gotcha.c:194] - Remapped call to exp at 0x55695ce44dd8 in [EMPTY] to wrapper at 0x0x55695cb606c1
[10727/10727][gotcha.c:194] - Remapped call to exp at 0x7ff2e9458410 in /home/timemory/build-docker-debug/libex_gotcha_lib.so to wrapper at 0x0x55695cb606c1
[10727/10727][gotcha.c:343] - Gotcha wrap completed successfully

> [do_exp_work@'../examples/ex-gotcha/ex_gotcha_lib.cpp':61] ...

	[itr:  0]> float  expf(  2.000) =                7.389
	[itr:  1]> float  expf(  2.250) =               16.877
	[itr:  2]> float  expf(  2.750) =               32.519

	executing modified exp function :                2.000...	[itr:  0]> double exp (  2.000) =                7.389
	executing modified exp function :                2.250...	[itr:  1]> double exp (  2.250) =               16.877
	executing modified exp function :                2.750...	[itr:  2]> double exp (  2.750) =               32.519

[10727/10727][gotcha.c:276] - User called gotcha_wrap for tool exp with 1 bindings
	0: exp will map to 0x55695e02cd00
[10727/10727][gotcha.c:284] - Initializing 1 user binding entries to NULL
[10727/10727][gotcha.c:36] - Updating binding address pointer at 0x55695ce46d00 to (nil)
[10727/10727][gotcha.c:300] - Moved current_generation to 3 in gotcha_wrap
[10727/10727][gotcha.c:302] - Creating internal binding data structures and adding binding to tool
[10727/10727][tool.c:134] - Created new binding table of size 1 for tool exp
[10727/10727][gotcha.c:309] - Processing 1 bindings
[10727/10727][gotcha.c:143] - gotcha_rewrite_wrapper_orders for binding exp in tool exp of priority -1
[10727/10727][gotcha.c:150] - Adding new entry for exp to hash table
[10727/10727][gotcha.c:315] - Symbol exp needs lookup operation
[10727/10727][gotcha.c:56] - Looking up exported symbols for exp
[10727/10727][gotcha.c:69] - Searching for exported symbols in [EMPTY]
[10727/10727][gotcha.c:78] - Checking GNU hash for exp in [EMPTY]
[10727/10727][gotcha.c:90] - exp not found in [EMPTY]
[10727/10727][gotcha.c:65] - Skipping VDSO library at 0x7fffe2d5d000 with name linux-vdso.so.1
[10727/10727][gotcha.c:69] - Searching for exported symbols in /home/timemory/build-docker-debug/libex_gotcha_lib.so
[10727/10727][gotcha.c:78] - Checking GNU hash for exp in /home/timemory/build-docker-debug/libex_gotcha_lib.so
[10727/10727][gotcha.c:90] - exp not found in /home/timemory/build-docker-debug/libex_gotcha_lib.so
[10727/10727][gotcha.c:69] - Searching for exported symbols in /home/timemory/build-docker-debug/libgotcha.so.1
[10727/10727][gotcha.c:78] - Checking GNU hash for exp in /home/timemory/build-docker-debug/libgotcha.so.1
[10727/10727][gotcha.c:90] - exp not found in /home/timemory/build-docker-debug/libgotcha.so.1
[10727/10727][gotcha.c:69] - Searching for exported symbols in /lib/x86_64-linux-gnu/libpthread.so.0
[10727/10727][gotcha.c:78] - Checking GNU hash for exp in /lib/x86_64-linux-gnu/libpthread.so.0
[10727/10727][gotcha.c:84] - Checking ELF hash for exp in /lib/x86_64-linux-gnu/libpthread.so.0
[10727/10727][gotcha.c:90] - exp not found in /lib/x86_64-linux-gnu/libpthread.so.0
[10727/10727][gotcha.c:69] - Searching for exported symbols in /usr/lib/x86_64-linux-gnu/libstdc++.so.6
[10727/10727][gotcha.c:78] - Checking GNU hash for exp in /usr/lib/x86_64-linux-gnu/libstdc++.so.6
[10727/10727][gotcha.c:90] - exp not found in /usr/lib/x86_64-linux-gnu/libstdc++.so.6
[10727/10727][gotcha.c:69] - Searching for exported symbols in /lib/x86_64-linux-gnu/libm.so.6
[10727/10727][gotcha.c:78] - Checking GNU hash for exp in /lib/x86_64-linux-gnu/libm.so.6
[10727/10727][gotcha.c:100] - Symbol exp found in /lib/x86_64-linux-gnu/libm.so.6 at 0x7ff2e86731f0
[10727/10727][gotcha.c:46] - Updating binding address pointer at 0x55695e031ce8 to 0x7ff2e86731f0
[10727/10727][gotcha.c:325] - Symbol exp needs binding from application
[10727/10727][gotcha.c:262] - Searching all callsites for 1 bindings
[10727/10727][gotcha.c:194] - Remapped call to exp at 0x55695ce44dd8 in [EMPTY] to wrapper at 0x0x55695e02cd00
[10727/10727][gotcha.c:194] - Remapped call to exp at 0x7ff2e9458410 in /home/timemory/build-docker-debug/libex_gotcha_lib.so to wrapper at 0x0x55695e02cd00
[10727/10727][gotcha.c:343] - Gotcha wrap completed successfully
[0]> END SCOPED GOTCHA....

[0]> AFTER: exp(2.0) = 7.389056


[0]> BEFORE: exp(2.0) = 7.389056

[0]> BEGIN SCOPED GOTCHA....
[10727/10727][gotcha.c:276] - User called gotcha_wrap for tool exp with 1 bindings
	0: exp will map to 0x55695cb606c1
[10727/10727][gotcha.c:284] - Initializing 1 user binding entries to NULL
[10727/10727][gotcha.c:36] - Updating binding address pointer at 0x55695ce46d08 to (nil)
[10727/10727][gotcha.c:300] - Moved current_generation to 4 in gotcha_wrap
[10727/10727][gotcha.c:302] - Creating internal binding data structures and adding binding to tool
[10727/10727][tool.c:134] - Created new binding table of size 1 for tool exp
[10727/10727][gotcha.c:309] - Processing 1 bindings
[10727/10727][gotcha.c:143] - gotcha_rewrite_wrapper_orders for binding exp in tool exp of priority -1
[10727/10727][gotcha.c:177] - Inserting binding after tool exp
[10727/10727][gotcha.c:46] - Updating binding address pointer at 0x55695e02de48 to 0x7ff2e86731f0
[10727/10727][gotcha.c:46] - Updating binding address pointer at 0x55695e02de48 to 0x55695cb606c1
[10727/10727][gotcha.c:343] - Gotcha wrap completed successfully

> [do_exp_work@'../examples/ex-gotcha/ex_gotcha_lib.cpp':61] ...

	[itr:  0]> float  expf(  2.000) =                7.389
	[itr:  1]> float  expf(  2.250) =               16.877
	[itr:  2]> float  expf(  2.750) =               32.519


Program received signal SIGSEGV, Segmentation fault.
0x000055695e02cd00 in ?? ()
(gdb) bt
#0  0x000055695e02cd00 in ?? ()
#1  0x00007ff2e923b343 in ext::<lambda(double)>::operator()(double) const (__closure=0x7fffe2c26ff7, val=2) at ../examples/ex-gotcha/ex_gotcha_lib.cpp:69
#2  0x00007ff2e923b780 in ext::work<double, ext::do_exp_work(int)::<lambda(double)>, ext::do_exp_work(int)::<lambda(double, int)> >(const std::string &, int, ext::<lambda(double)> &&, ext::<lambda(double, int)> &&) (fname="exp", nitr=3, 
    func=..., incr=...) at ../examples/ex-gotcha/ex_gotcha_lib.cpp:47
#3  0x00007ff2e923b469 in ext::do_exp_work (nitr=3) at ../examples/ex-gotcha/ex_gotcha_lib.cpp:69
#4  0x000055695cb02168 in <lambda()>::operator()(void) const (__closure=0x7fffe2c29980) at ../examples/ex-gotcha/ex_gotcha.cpp:177
#5  0x000055695cb026df in main (argc=2, argv=0x7fffe2c29ae8) at ../examples/ex-gotcha/ex_gotcha.cpp:217

ppc64le port

Port gotcha to ppc64le. Off the top of my head, there's a different format for how the GOT table and function pointers are laid out that comes from the ABIv2 used by ppc64le. There may be other issues.

Work with libdl (dlopen/dlsym)

Right now if we see a series of actions like

gotcha_do_wrapping()
dlopen(...)
dlsym(...)

If the libdl calls return symbols that gotcha "has wrapped," we won't catch it.

Gotcha breaks OpenCL

I have a reproducer that breaks valid OpenCL applications when wrapping OpenCL functions with GOTCHA

Compile either with or without -DUSE_GOTCHA:

/* -*- c -*- */

#define _GNU_SOURCE
#include <unistd.h>
#include <stdlib.h>
#include <stdio.h>
#include <string.h>
#include <stdint.h>
#include <stdbool.h>
#include <math.h>

#ifdef __APPLE__
#  include <OpenCL/opencl.h>
#else
#  include <CL/cl.h>
#endif

#ifdef USE_GOTCHA

#include <gotcha/gotcha.h>

// We need a place to store the pointer to the function we've wrapped
static gotcha_wrappee_handle_t handle_clGetPlatformIDs;
static gotcha_wrappee_handle_t handle_clGetExtensionFunctionAddress;
static gotcha_wrappee_handle_t handle_clGetPlatformInfo;

static cl_int
doClGetPlatformIDs(cl_uint         numEntries,
                   cl_platform_id* platforms,
                   cl_uint*        numPlatforms)
{
    fprintf(stderr, "E clGetPlatformIDs(%u)\n", numEntries);

    typeof(&doClGetPlatformIDs) origClGetPlatformIDs = gotcha_get_wrappee(handle_clGetPlatformIDs);
    cl_int ret = origClGetPlatformIDs(numEntries, platforms, numPlatforms);

    fprintf(stderr, "E clGetPlatformIDs(%u) = %d\n", numEntries, ret);

    return ret;
}

static void*
doClGetExtensionFunctionAddress( const char* funcName )
{
    fprintf(stderr, "E clGetExtensionFunctionAddress(%s)\n", funcName);

    typeof(&doClGetExtensionFunctionAddress) origClGetExtensionFunctionAddress = gotcha_get_wrappee(handle_clGetExtensionFunctionAddress);
    void* ret = origClGetExtensionFunctionAddress( funcName );

    fprintf(stderr, "L clGetExtensionFunctionAddress(%s) = %p\n", funcName, ret);

    return ret;
}

static cl_int
doClGetPlatformInfo(cl_platform_id   platform,
                    cl_platform_info paramName,
                    size_t           paramValueSize,
                    void*            paramValue,
                    size_t*          paramValueSizeRet)
{
    fprintf(stderr, "E clGetPlatformInfo()\n");

    typeof(&doClGetPlatformInfo) origClGetPlatformInfo = gotcha_get_wrappee(handle_clGetPlatformInfo);
    cl_int ret = origClGetPlatformInfo(platform, paramName, paramValueSize, paramValue, paramValueSizeRet);

    fprintf(stderr, "L clGetPlatformInfo()\n");

    return ret;
}

struct gotcha_binding_t bindings[] = {
    { "clGetPlatformIDs", doClGetPlatformIDs, &handle_clGetPlatformIDs},
    { "clGetExtensionFunctionAddress", doClGetExtensionFunctionAddress, &handle_clGetExtensionFunctionAddress},
    { "clGetPlatformInfo", doClGetPlatformInfo, &handle_clGetPlatformInfo}
};

#endif

int
main(int ac, char *av[])
{
#ifdef USE_GOTCHA
    gotcha_wrap(bindings, 2, "ocl_test");
#endif

    cl_uint platform_count = 0;

    clGetPlatformIDs(0, NULL, &platform_count);

    if (platform_count == 0) {
        printf("No OpenCL platforms found!\n");
        return 127;
    } else {
        cl_platform_id* platforms = calloc(platform_count, sizeof(*platforms));
        clGetPlatformIDs(platform_count, platforms, NULL);

        printf("Platforms:\n");

        for (size_t i = 0; i < platform_count; i++) {
            char platform_name[256];
            clGetPlatformInfo(platforms[i], CL_PLATFORM_NAME, sizeof(platform_name), platform_name, NULL);
            printf(" %zu: %s\n", i, platform_name);
        }

        free(platforms);
    }

    return 0;
}

When running the non-wrapped version and setting a breakpoint at clGetExtensionFunctionAddress, the call chain looks like this:

#0  0x00007ffff7c64400 in clGetExtensionFunctionAddress () from /opt/rocm-6.0.2/lib/libamdocl64.so
#1  0x00007ffff7f8e532 in ?? () from /opt/rocm-6.0.2/lib/libOpenCL.so.1
#2  0x00007ffff7f9058c in ?? () from /opt/rocm-6.0.2/lib/libOpenCL.so.1
#3  0x00007ffff7d874df in __pthread_once_slow (once_control=0x7ffff7f940f8, init_routine=0x7ffff7f90420) at pthread_once.c:116
#4  0x00007ffff7f8eb95 in clGetPlatformIDs () from /opt/rocm-6.0.2/lib/libOpenCL.so.1
#5  0x000000000020190f in main ()

clGetExtensionFunctionAddress is called from inside libOpenCL.so.1 but will be dispatched to the AMD OpenCL ICD libamdocl64.so.

When doing the same with the wrapped binary:

#0  0x00007ffff7f8fc60 in clGetExtensionFunctionAddress () from /opt/rocm-6.0.2/lib/libOpenCL.so.1
#1  0x0000000000201c24 in doClGetExtensionFunctionAddress ()
#2  0x00007ffff7f8e532 in ?? () from /opt/rocm-6.0.2/lib/libOpenCL.so.1
#3  0x00007ffff7f9058c in ?? () from /opt/rocm-6.0.2/lib/libOpenCL.so.1
#4  0x00007ffff7d764df in __pthread_once_slow (once_control=0x7ffff7f940f8, init_routine=0x7ffff7f90420) at pthread_once.c:116
#5  0x00007ffff7f8eb95 in clGetPlatformIDs () from /opt/rocm-6.0.2/lib/libOpenCL.so.1
#6  0x0000000000201bb1 in doClGetPlatformIDs ()
#7  0x0000000000201d3d in main ()

the wrapper calls clGetExtensionFunctionAddress from libOpenCL.so.1 and not from libamdocl64.so. But because the OpenCL library is still in its initialization, clGetExtensionFunctionAddress tries to init itself again but that deadlocks because of the recursive call to pthread_once. Here is the backtrace when continuing:

#0  futex_wait (private=0, expected=1, futex_word=0x7ffff7f940f8) at ../sysdeps/nptl/futex-internal.h:141
#1  futex_wait_simple (private=0, expected=1, futex_word=0x7ffff7f940f8) at ../sysdeps/nptl/futex-internal.h:172
#2  __pthread_once_slow (once_control=0x7ffff7f940f8, init_routine=0x7ffff7f90420) at pthread_once.c:105
#3  0x00007ffff7f8fc7f in clGetExtensionFunctionAddress () from /opt/rocm-6.0.2/lib/libOpenCL.so.1
#4  0x0000000000201c24 in doClGetExtensionFunctionAddress ()
#5  0x00007ffff7f8e532 in ?? () from /opt/rocm-6.0.2/lib/libOpenCL.so.1
#6  0x00007ffff7f9058c in ?? () from /opt/rocm-6.0.2/lib/libOpenCL.so.1
#7  0x00007ffff7d764df in __pthread_once_slow (once_control=0x7ffff7f940f8, init_routine=0x7ffff7f90420) at pthread_once.c:116
#8  0x00007ffff7f8eb95 in clGetPlatformIDs () from /opt/rocm-6.0.2/lib/libOpenCL.so.1
#9  0x0000000000201bb1 in doClGetPlatformIDs ()
#10 0x0000000000201d3d in main ()

You will notice that the arguments to __pthread_once_slow are the same in level #2 and #7.

My current work around is to filter libOpenCL.so.

Intel compiler mkl wrapping dgemm_: undefined symbol: ompt_start_tool

This bug might be linked to #70 Intel OpenMP breaks Gotcha bug.
Intel version used: intel/19.0.3.199
Trying to wrap dgemm_ from mkl I got:

[...]
[44825/44825][gotcha.c:343] - Gotcha wrap completed successfully
Before wrappee dgemm call
./a.out: symbol lookup error: /opt/intel/compilers_and_libraries_2019.3.199/linux/compiler/lib/intel64/libiomp5.so: undefined symbol: ompt_start_tool

Basic Tool Stacking

To enhance what LD_PRELOAD offers, would be nice if we could have tools 1 and 2 both use GOTCHA, wrap the same symbols, and have them be stacked

For 0.1 we'll want this to work in the following way: if Tool1 wraps foo and then Tool2 wraps foo, Tool2 will see Tool1's foo as the "real" foo. This will lead to Tool2 calls Tool1 calls real_foo.

In the future, we'll want an interface for tool developers to say "I should happen before/after" other tools/tool categories.

Procedural C++ interface

We have a proposal for a functional interface ( #47 ), which could more aptly be thought of as a C++11 functional programming interface for Gotcha.

Independent of that, it would be good to have an interface that does the C things Gotcha does, but with a nicer interface which takes advantage of what C++ offers (RAII, use of nice object handles rather than string names for everything...)

PrgEnv for CORI users (documentation?)

Hi,

just be careful to run the target application in same environment as the one GOTCHA was compiled in.
Indeed the dependance to libimf.so, libsvml.so, libirng.so and libintlc.so will be broken by switching between INTEL and GNU environment for example.
$ ldd libgotcha.so.1
linux-vdso.so.1 (0x00002aaaaaad3000)
libz.so.1 => /lib64/libz.so.1 (0x00002aaaaacd3000)
libhugetlbfs.so => /usr/lib64/libhugetlbfs.so (0x00002aaaaaeea000)
librca.so.0 => /opt/cray/rca/default/lib64/librca.so.0 (0x00002aaaab120000)
libpthread.so.0 => /lib64/libpthread.so.0 (0x00002aaaab324000)
libimf.so => not found
libsvml.so => not found
libirng.so => not found
libm.so.6 => /lib64/libm.so.6 (0x00002aaaab542000)
libgcc_s.so.1 => /opt/gcc/8.2.0/snos/lib64/libgcc_s.so.1 (0x00002aaaab87a000)
libintlc.so.5 => not found
libc.so.6 => /lib64/libc.so.6 (0x00002aaaaba92000)
/lib64/ld-linux-x86-64.so.2 (0x00002aaaaaaab000)
libdl.so.2 => /lib64/libdl.so.2 (0x00002aaaabe4c000)

$ module swap PrgEnv-gnu PrgEnv-intel
$ ldd libgotcha.so.1
linux-vdso.so.1 (0x00002aaaaaad3000)
libz.so.1 => /lib64/libz.so.1 (0x00002aaaaacd3000)
libhugetlbfs.so => /usr/lib64/libhugetlbfs.so (0x00002aaaaaeea000)
librca.so.0 => /opt/cray/rca/default/lib64/librca.so.0 (0x00002aaaab120000)
libpthread.so.0 => /lib64/libpthread.so.0 (0x00002aaaab324000)
libimf.so => /opt/intel/compilers_and_libraries_2019.3.199/linux/compiler/lib/intel64/libimf.so (0x00002aaaab542000)
libsvml.so => /opt/intel/compilers_and_libraries_2019.3.199/linux/compiler/lib/intel64/libsvml.so (0x00002aaaabae2000)
libirng.so => /opt/intel/compilers_and_libraries_2019.3.199/linux/compiler/lib/intel64/libirng.so (0x00002aaaad486000)
libm.so.6 => /lib64/libm.so.6 (0x00002aaaad7f8000)
libgcc_s.so.1 => /opt/gcc/8.2.0/snos/lib64/libgcc_s.so.1 (0x00002aaaadb30000)
libintlc.so.5 => /opt/intel/compilers_and_libraries_2019.3.199/linux/compiler/lib/intel64/libintlc.so.5 (0x00002aaaadd48000)
libc.so.6 => /lib64/libc.so.6 (0x00002aaaadfba000)
/lib64/ld-linux-x86-64.so.2 (0x00002aaaaaaab000)
libdl.so.2 => /lib64/libdl.so.2 (0x00002aaaae374000)

Intel OpenMP breaks Gotcha

During the release process we discussed this issue with @daboehme, it was first reported in #40.

The problem is that Intel's runtimes use many mechanisms that break Gotcha, dlsym with RTLD_NEXT, and even (I believe) GOT rewriting. Fixing this could take multiple steps

Step 1) Function better with dlsym and RTLD_NEXT
Step 2) (If the problem still exists) Use some awareness when Intel OpenMP libraries are present. Perhaps, if Intel OpenMP is there, we wait until we see them do a rewrite before rewriting GOTs.

Closer to the time it comes to fix this I might ping some people at Intel, this seems like the kind of bug which we could iteratively cause for one another as we update code, I'd like to avoid that.

macOS does not have link.h

In file included from /opt/local/var/macports/build/_opt_PPCSnowLeopardPorts_devel_gotcha/gotcha/work/GOTCHA-1.0.4/src/translations.h:21,
                 from /opt/local/var/macports/build/_opt_PPCSnowLeopardPorts_devel_gotcha/gotcha/work/GOTCHA-1.0.4/src/gotcha.c:16:
/opt/local/var/macports/build/_opt_PPCSnowLeopardPorts_devel_gotcha/gotcha/work/GOTCHA-1.0.4/include/gotcha/gotcha.h:34:10: fatal error: link.h: No such file or directory
   34 | #include <link.h>
      |          ^~~~~~~~
compilation terminated.

See related: geodynamics/aspect#218

Adding C++ support

I plan on using this as a meta-issue from which we track C++ development in Gotcha.

First up is #46 : it would be good to have C++ niceness available to our users who would like it.

Some broader issues/philosophy on C++ in Gotcha:

  1. Never ever ever any C++ compiled code in the Gotcha library. Ever

  2. Name mangling is a thing in C++, we need to not get ourselves in trouble with it, and possibly want facilities to support it

  3. C++ objects have lifetimes, if they contain function pointers which Gotcha binds to and which then get deleted/freed, we are in trouble. Careful consideration of object lifetimes will be necessary for safe interfaces, especially considering things like "free" getting called after a gotcha binding to free gets freed

Potential issues wrapping malloc (indepdent of C++)

I was talking with @trws, and he mentioned that even if you link libc dynamically, sometimes g++ gets Very Clever and decides to inline malloc, or use a statically linked version, or otherwise be a jerk. If he is feeling generous, he could feel free to chime in, but as an action item we should test a malloc wrapping against a few gcc compiler versions to make sure we don't have hidden bugs.

Low priority, but needs to be documented

Gotcha segfaults in Caliper-annotated code when finalizing Python

Hi all, I got a report from @pramodk about a gotcha-related segfault when using Caliper here: LLNL/Caliper#529

It's using gotcha v1.0.4 that's shipped with Caliper v2.10. In the stack trace it looks like it's segfaulting when deleting thread data during Python finalization?:

#0  0x00007fffed8ec730 in do_lookup_x () from /lib64/ld-linux-x86-64.so.2
#1  0x00007fffed8ed09f in _dl_lookup_symbol_x () from /lib64/ld-linux-x86-64.so.2
#2  0x00007fffe9055d09 in do_sym () from /lib64/libc.so.6
#3  0x00007fffeb9280c4 in dlsym_doit () from /lib64/libdl.so.2
#4  0x00007fffed8f27d4 in _dl_catch_error () from /lib64/ld-linux-x86-64.so.2
#5  0x00007fffeb9285ad in _dlerror_run () from /lib64/libdl.so.2
#6  0x00007fffeb928118 in dlsym () from /lib64/libdl.so.2
#7  0x00007fffeb8b0938 in dlsym_wrapper () from /gpfs/bbp.cscs.ch/ssd/apps/bsd/2024-02-01/stage_applications/install_gcc-12.3.0-skylake/caliper-2.10.0-btrc4s/lib64/libcaliper.so.2
#8  0x00007fffeabb3dff in munmap (addr=0x7fffedad3000, len=16384) at reg_cache.c:723
#9  0x00007fffeb36d48f in tstate_delete_common (tstate=tstate@entry=0x7fffeb64cd78 <_PyRuntime+166328>, gilstate=<optimized out>) at Python/pystate.c:1105
#10 0x00007fffeb36e6f0 in _PyThreadState_Delete (check_current=0, tstate=0x7fffeb64cd78 <_PyRuntime+166328>) at Python/pystate.c:1119
#11 zapthreads (check_current=0, interp=0x7fffeb6329f8 <_PyRuntime+58936>) at Python/pystate.c:483
#12 PyInterpreterState_Delete (interp=0x7fffeb6329f8 <_PyRuntime+58936>) at Python/pystate.c:493
#13 0x00007fffeb36c455 in finalize_interp_delete (interp=<optimized out>) at Python/pylifecycle.c:1746
#14 Py_FinalizeEx () at Python/pylifecycle.c:1900
#15 Py_FinalizeEx () at Python/pylifecycle.c:1751
#16 0x00007fffed0623cc in nrnpython_start (b=0) at /gpfs/bbp.cscs.ch/home/kumbhar/tmp/caliper/spack-src/src/nrnpython/nrnpython.cpp:260
#17 0x00007fffec5208c9 in ivocmain_session (argc=2, argv=0x7ffffffef6f8, env=0x7ffffffef710, start_session=1) at /gpfs/bbp.cscs.ch/home/kumbhar/tmp/caliper/spack-src/src/ivoc/ivocmain.cpp:772
#18 0x00007fffec5203a8 in ivocmain (argc=2, argv=0x7ffffffef6f8, env=0x7ffffffef710) at /gpfs/bbp.cscs.ch/home/kumbhar/tmp/caliper/spack-src/src/ivoc/ivocmain.cpp:349
#19 0x0000000000402258 in main (argc=2, argv=0x7ffffffef6f8, env=0x7ffffffef710) at /gpfs/bbp.cscs.ch/home/kumbhar/tmp/caliper/spack-src/src/ivoc/nrnmain.cpp:71

Any suggestions on how to debug this?

Document internal structures

Currently figuring out the binding_ref_t and binding_t pipelines.

There are lines like

 user_binding = ref->binding->user_binding + ref->index;

Which can be a bit inscrutable. Game to do Doxygen for that, or should I figure it out and document it myself?

Transient C interface

Users will need the ability to have interfaces for "temporary" wrappings, or wrappings that only happen in certain scopes. The C interface for this may be resolved through #49 , but on closing that issue we should see whether we feel able to do temporary wrappings in C, and if not add anything needed for that purpose

Breaks when actually using libc

@mplegendre ,

With the new libc_wrappers.h


[  2%] Building C object src/CMakeFiles/gotcha.dir/gotcha_utils.c.o
In file included from /g/g0/dzpolia/src/GOTCHA/src/gotcha_utils.c:17:
/g/g0/dzpolia/src/GOTCHA/src/libc_wrappers.h:40:35: error: "(" may not appear in macro parameter list
make[2]: *** [src/CMakeFiles/gotcha.dir/gotcha_utils.c.o] Error 1
make[1]: *** [src/CMakeFiles/gotcha.dir/all] Error 2
make: *** [all] Error 2
@dzpolia:vim ../src/libc_wrappers.h
@dzpolia:git checkout -- ../src/libc_wrappers.h
@dzpolia:cmake -DGOTCHA_ENABLE_TESTS=ON -DCMAKE_C_FLAGS="-DGOTCHA_USE_LIBC" ..
-- Configuring done
-- Generating done
-- Build files have been written to: GOTCHA/build
@dzpolia:make
Scanning dependencies of target gotcha
[  2%] Building C object src/CMakeFiles/gotcha.dir/gotcha_utils.c.o
[  5%] Building C object src/CMakeFiles/gotcha.dir/gotcha.c.o
[  7%] Building C object src/CMakeFiles/gotcha.dir/gotcha_auxv.c.o
GOTCHA/src/gotcha_auxv.c: In function 'parse_auxv_contents':
GOTCHA/src/gotcha_auxv.c:48: error: expected expression before ')' token
GOTCHA/src/gotcha_auxv.c: In function 'get_vdso_from_maps':
GOTCHA/src/gotcha_auxv.c:228: error: expected expression before ')' token
make[2]: *** [src/CMakeFiles/gotcha.dir/gotcha_auxv.c.o] Error 1
make[1]: *** [src/CMakeFiles/gotcha.dir/all] Error 2
make: *** [all] Error 2

I'll start knocking that out tomorrow

Different behavior using GCC or ICC

Hello,

I am using gotcha to capture exp math function calls (thanks for the tool by the way ;) ),
but even if the exp calls appears in both g++ and icpc compiled binary (I looked into with objdump -D), the former is correctly intercepted whereas the latter is not ...

Would you have any explanation of this difference? They are both compiled in O0.
Thanks

Test printf

Once we merge in #8 , we'll need to also test printf and then we should be fully covered. I am very tentatively assigning that to @mplegendre , though it could be reassigned as necessary :)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.