Comments (2)
yonghong-song
commented on May 28, 2024
The current memleak implementation only supports C functions like malloc/calloc/mmap/memalign/free/munmap etc. If you are interested, you could contribute to add C++ support.
from bcc.
Bojun-Seo
commented on May 28, 2024
The cause of this issue is that Linux kernel cannot unwind the stack frame of operator new function.
Kernel uses rbp registers to unwind the stack.
You can check that on file arch/x86/events/core.c in Linux kernel, at line 2875
2858 void
2859 perf_callchain_user(struct perf_callchain_entry_ctx *entry, struct pt_regs *regs)
2860 {
2861 struct stack_frame frame;
2862 const struct stack_frame __user *fp;
2863
2864 if (perf_guest_state()) {
2865 /* TODO: We don't support guest os callchain now */
2866 return;
2867 }
2868
2869 /*
2870 * We don't know what to do with VM86 stacks.. ignore them for now.
2871 */
2872 if (regs->flags & (X86_VM_MASK | PERF_EFLAGS_VM))
2873 return;
2874
2875 fp = (void __user *)regs->bp;
2876
2877 perf_callchain_store(entry, regs->ip);
2878But operator new function doesn't push and pop rbp register on its prologue and epilogue.
_Znwm is a mangled name of operator new.
00000000000ae970 <_Znwm@@GLIBCXX_3.4>:
ae970: f3 0f 1e fa endbr64
ae974: 48 85 ff test %rdi,%rdi
ae977: b8 01 00 00 00 mov $0x1,%eax
ae97c: 53 push %rbx
ae97d: 48 0f 45 c7 cmovne %rdi,%rax
ae981: 48 89 c3 mov %rax,%rbx
ae984: 48 89 df mov %rbx,%rdi
ae987: e8 44 07 ff ff call 9f0d0 <malloc@plt>
ae98c: 48 85 c0 test %rax,%rax
ae98f: 74 02 je ae993 <_Znwm@@GLIBCXX_3.4+0x23>
ae991: 5b pop %rbx
ae992: c3 ret
ae993: e8 88 07 ff ff call 9f120 <_ZSt15get_new_handlerv@plt>
ae998: 48 85 c0 test %rax,%rax
ae99b: 0f 84 dd 3d ff ff je a277e <__cxa_throw_bad_array_new_length@@CXXABI_1.3.8+0x132>
ae9a1: ff d0 call *%rax
ae9a3: eb df jmp ae984 <_Znwm@@GLIBCXX_3.4+0x14>
ae9a5: 66 2e 0f 1f 84 00 00 cs nopw 0x0(%rax,%rax,1)
ae9ac: 00 00 00
ae9af: 90 nop
The first backrace is acquired by pc register.
And others use stack and rbp to unwind the stack.
Since operator new function doesn't touch rbp register.
The value inside rbp register is same as if it is currently in the function that calls operator new.
For example, imagine the call chain looks like this: main -> foo -> operator new -> malloc
As memleak saves stack backtrace information on uretprobe of malloc function.
So current pc is inside operator new function. And the rbp value is same as when the pc register is inside foo function.
perf_callchain_user function acquires top stack by using pc register.
And it uses stack and rbp to unwind others, and at this point, it thinks foo is the top frame.
So it doesn't print top frame because top frame is already printed by using pc.
I found that following feature could resolve this issue.
#4463
I patched memleak like followings, on that branch.
(It seems complicated but it's not. Almost every change is to change skel string to obj)
diff --git a/libbpf-tools/memleak.bpf.c b/libbpf-tools/memleak.bpf.c
index cb13fdd8..9d7d7da3 100644
--- a/libbpf-tools/memleak.bpf.c
+++ b/libbpf-tools/memleak.bpf.c
@@ -7,6 +7,7 @@
#include "maps.bpf.h"
#include "memleak.h"
#include "core_fixes.bpf.h"
+#include "unwind.bpf.h"
const volatile size_t min_size = 0;
const volatile size_t max_size = -1;
@@ -122,7 +123,8 @@ static int gen_alloc_exit2(void *ctx, u64 address)
if (address != 0) {
info.timestamp_ns = bpf_ktime_get_ns();
- info.stack_id = bpf_get_stackid(ctx, &stack_traces, stack_flags);
+ //info.stack_id = bpf_get_stackid(ctx, &stack_traces, stack_flags);
+ info.stack_id = uw_get_stackid();
bpf_map_update_elem(&allocs, &address, &info, BPF_ANY);
diff --git a/libbpf-tools/memleak.c b/libbpf-tools/memleak.c
index a2c7d1cd..bab2b9e0 100644
--- a/libbpf-tools/memleak.c
+++ b/libbpf-tools/memleak.c
@@ -23,6 +23,7 @@
#include "memleak.h"
#include "memleak.skel.h"
#include "trace_helpers.h"
+#include "unwind_helpers.h"
#ifdef USE_BLAZESYM
#include "blazesym.h"
@@ -86,38 +87,38 @@ struct allocation {
struct allocation_node* allocations;
};
-#define __ATTACH_UPROBE(skel, sym_name, prog_name, is_retprobe) \
+#define __ATTACH_UPROBE(obj, sym_name, prog_name, is_retprobe) \
do { \
LIBBPF_OPTS(bpf_uprobe_opts, uprobe_opts, \
.func_name = #sym_name, \
.retprobe = is_retprobe); \
- skel->links.prog_name = bpf_program__attach_uprobe_opts( \
- skel->progs.prog_name, \
+ obj->links.prog_name = bpf_program__attach_uprobe_opts( \
+ obj->progs.prog_name, \
env.pid, \
env.object, \
0, \
&uprobe_opts); \
} while (false)
-#define __CHECK_PROGRAM(skel, prog_name) \
+#define __CHECK_PROGRAM(obj, prog_name) \
do { \
- if (!skel->links.prog_name) { \
+ if (!obj->links.prog_name) { \
perror("no program attached for " #prog_name); \
return -errno; \
} \
} while (false)
-#define __ATTACH_UPROBE_CHECKED(skel, sym_name, prog_name, is_retprobe) \
+#define __ATTACH_UPROBE_CHECKED(obj, sym_name, prog_name, is_retprobe) \
do { \
- __ATTACH_UPROBE(skel, sym_name, prog_name, is_retprobe); \
- __CHECK_PROGRAM(skel, prog_name); \
+ __ATTACH_UPROBE(obj, sym_name, prog_name, is_retprobe); \
+ __CHECK_PROGRAM(obj, prog_name); \
} while (false)
-#define ATTACH_UPROBE(skel, sym_name, prog_name) __ATTACH_UPROBE(skel, sym_name, prog_name, false)
-#define ATTACH_URETPROBE(skel, sym_name, prog_name) __ATTACH_UPROBE(skel, sym_name, prog_name, true)
+#define ATTACH_UPROBE(obj, sym_name, prog_name) __ATTACH_UPROBE(obj, sym_name, prog_name, false)
+#define ATTACH_URETPROBE(obj, sym_name, prog_name) __ATTACH_UPROBE(obj, sym_name, prog_name, true)
-#define ATTACH_UPROBE_CHECKED(skel, sym_name, prog_name) __ATTACH_UPROBE_CHECKED(skel, sym_name, prog_name, false)
-#define ATTACH_URETPROBE_CHECKED(skel, sym_name, prog_name) __ATTACH_UPROBE_CHECKED(skel, sym_name, prog_name, true)
+#define ATTACH_UPROBE_CHECKED(obj, sym_name, prog_name) __ATTACH_UPROBE_CHECKED(obj, sym_name, prog_name, false)
+#define ATTACH_URETPROBE_CHECKED(obj, sym_name, prog_name) __ATTACH_UPROBE_CHECKED(obj, sym_name, prog_name, true)
static void sig_handler(int signo);
@@ -147,11 +148,11 @@ static int print_outstanding_allocs(int allocs_fd, int stack_traces_fd);
static int print_outstanding_combined_allocs(int combined_allocs_fd, int stack_traces_fd);
static bool has_kernel_node_tracepoints();
-static void disable_kernel_node_tracepoints(struct memleak_bpf *skel);
-static void disable_kernel_percpu_tracepoints(struct memleak_bpf *skel);
-static void disable_kernel_tracepoints(struct memleak_bpf *skel);
+static void disable_kernel_node_tracepoints(struct memleak_bpf *obj);
+static void disable_kernel_percpu_tracepoints(struct memleak_bpf *obj);
+static void disable_kernel_tracepoints(struct memleak_bpf *obj);
-static int attach_uprobes(struct memleak_bpf *skel);
+static int attach_uprobes(struct memleak_bpf *obj);
const char *argp_program_version = "memleak 0.1";
const char *argp_program_bug_address =
@@ -229,7 +230,7 @@ static const char default_object[] = "libc.so.6";
int main(int argc, char *argv[])
{
int ret = 0;
- struct memleak_bpf *skel = NULL;
+ struct memleak_bpf *obj = NULL;
static const struct argp argp = {
.options = argp_options,
@@ -331,51 +332,52 @@ int main(int argc, char *argv[])
libbpf_set_print(libbpf_print_fn);
- skel = memleak_bpf__open();
- if (!skel) {
+ obj = memleak_bpf__open();
+ if (!obj) {
fprintf(stderr, "failed to open bpf object\n");
ret = 1;
goto cleanup;
}
- skel->rodata->min_size = env.min_size;
- skel->rodata->max_size = env.max_size;
- skel->rodata->page_size = env.page_size;
- skel->rodata->sample_rate = env.sample_rate;
- skel->rodata->trace_all = env.trace_all;
- skel->rodata->stack_flags = env.kernel_trace ? 0 : BPF_F_USER_STACK;
- skel->rodata->wa_missing_free = env.wa_missing_free;
+ obj->rodata->min_size = env.min_size;
+ obj->rodata->max_size = env.max_size;
+ obj->rodata->page_size = env.page_size;
+ obj->rodata->sample_rate = env.sample_rate;
+ obj->rodata->trace_all = env.trace_all;
+ obj->rodata->stack_flags = env.kernel_trace ? 0 : BPF_F_USER_STACK;
+ obj->rodata->wa_missing_free = env.wa_missing_free;
- bpf_map__set_value_size(skel->maps.stack_traces,
+ bpf_map__set_value_size(obj->maps.stack_traces,
env.perf_max_stack_depth * sizeof(unsigned long));
- bpf_map__set_max_entries(skel->maps.stack_traces, env.stack_map_max_entries);
+ bpf_map__set_max_entries(obj->maps.stack_traces, env.stack_map_max_entries);
+ UW_INIT(obj, 128, 10240);
// disable kernel tracepoints based on settings or availability
if (env.kernel_trace) {
if (!has_kernel_node_tracepoints())
- disable_kernel_node_tracepoints(skel);
+ disable_kernel_node_tracepoints(obj);
if (!env.percpu)
- disable_kernel_percpu_tracepoints(skel);
+ disable_kernel_percpu_tracepoints(obj);
} else {
- disable_kernel_tracepoints(skel);
+ disable_kernel_tracepoints(obj);
}
- ret = memleak_bpf__load(skel);
+ ret = memleak_bpf__load(obj);
if (ret) {
fprintf(stderr, "failed to load bpf object\n");
goto cleanup;
}
- const int allocs_fd = bpf_map__fd(skel->maps.allocs);
- const int combined_allocs_fd = bpf_map__fd(skel->maps.combined_allocs);
- const int stack_traces_fd = bpf_map__fd(skel->maps.stack_traces);
+ const int allocs_fd = bpf_map__fd(obj->maps.allocs);
+ const int combined_allocs_fd = bpf_map__fd(obj->maps.combined_allocs);
+ const int stack_traces_fd = bpf_map__fd(obj->maps.stack_traces);
// if userspace oriented, attach upbrobes
if (!env.kernel_trace) {
- ret = attach_uprobes(skel);
+ ret = attach_uprobes(obj);
if (ret) {
fprintf(stderr, "failed to attach uprobes\n");
@@ -383,7 +385,7 @@ int main(int argc, char *argv[])
}
}
- ret = memleak_bpf__attach(skel);
+ ret = memleak_bpf__attach(obj);
if (ret) {
fprintf(stderr, "failed to attach bpf program(s)\n");
@@ -476,7 +478,7 @@ cleanup:
if (ksyms)
ksyms__free(ksyms);
#endif
- memleak_bpf__destroy(skel);
+ memleak_bpf__destroy(obj);
free(allocs);
free(stack);
@@ -786,14 +788,7 @@ int print_stack_frames(struct allocation *allocs, size_t nr_allocs, int stack_tr
}
}
- if (bpf_map_lookup_elem(stack_traces_fd, &alloc->stack_id, stack)) {
- if (errno == ENOENT)
- continue;
-
- perror("failed to lookup stack trace");
-
- return -errno;
- }
+ uw_map_lookup_elem(&alloc->stack_id, env.pid, stack, 32);
(*print_stack_frames_func)();
}
@@ -1004,68 +999,68 @@ bool has_kernel_node_tracepoints()
tracepoint_exists("kmem", "kmem_cache_alloc_node");
}
-void disable_kernel_node_tracepoints(struct memleak_bpf *skel)
+void disable_kernel_node_tracepoints(struct memleak_bpf *obj)
{
- bpf_program__set_autoload(skel->progs.memleak__kmalloc_node, false);
- bpf_program__set_autoload(skel->progs.memleak__kmem_cache_alloc_node, false);
+ bpf_program__set_autoload(obj->progs.memleak__kmalloc_node, false);
+ bpf_program__set_autoload(obj->progs.memleak__kmem_cache_alloc_node, false);
}
-void disable_kernel_percpu_tracepoints(struct memleak_bpf *skel)
+void disable_kernel_percpu_tracepoints(struct memleak_bpf *obj)
{
- bpf_program__set_autoload(skel->progs.memleak__percpu_alloc_percpu, false);
- bpf_program__set_autoload(skel->progs.memleak__percpu_free_percpu, false);
+ bpf_program__set_autoload(obj->progs.memleak__percpu_alloc_percpu, false);
+ bpf_program__set_autoload(obj->progs.memleak__percpu_free_percpu, false);
}
-void disable_kernel_tracepoints(struct memleak_bpf *skel)
+void disable_kernel_tracepoints(struct memleak_bpf *obj)
{
- bpf_program__set_autoload(skel->progs.memleak__kmalloc, false);
- bpf_program__set_autoload(skel->progs.memleak__kmalloc_node, false);
- bpf_program__set_autoload(skel->progs.memleak__kfree, false);
- bpf_program__set_autoload(skel->progs.memleak__kmem_cache_alloc, false);
- bpf_program__set_autoload(skel->progs.memleak__kmem_cache_alloc_node, false);
- bpf_program__set_autoload(skel->progs.memleak__kmem_cache_free, false);
- bpf_program__set_autoload(skel->progs.memleak__mm_page_alloc, false);
- bpf_program__set_autoload(skel->progs.memleak__mm_page_free, false);
- bpf_program__set_autoload(skel->progs.memleak__percpu_alloc_percpu, false);
- bpf_program__set_autoload(skel->progs.memleak__percpu_free_percpu, false);
+ bpf_program__set_autoload(obj->progs.memleak__kmalloc, false);
+ bpf_program__set_autoload(obj->progs.memleak__kmalloc_node, false);
+ bpf_program__set_autoload(obj->progs.memleak__kfree, false);
+ bpf_program__set_autoload(obj->progs.memleak__kmem_cache_alloc, false);
+ bpf_program__set_autoload(obj->progs.memleak__kmem_cache_alloc_node, false);
+ bpf_program__set_autoload(obj->progs.memleak__kmem_cache_free, false);
+ bpf_program__set_autoload(obj->progs.memleak__mm_page_alloc, false);
+ bpf_program__set_autoload(obj->progs.memleak__mm_page_free, false);
+ bpf_program__set_autoload(obj->progs.memleak__percpu_alloc_percpu, false);
+ bpf_program__set_autoload(obj->progs.memleak__percpu_free_percpu, false);
}
-int attach_uprobes(struct memleak_bpf *skel)
+int attach_uprobes(struct memleak_bpf *obj)
{
- ATTACH_UPROBE_CHECKED(skel, malloc, malloc_enter);
- ATTACH_URETPROBE_CHECKED(skel, malloc, malloc_exit);
+ ATTACH_UPROBE_CHECKED(obj, malloc, malloc_enter);
+ ATTACH_URETPROBE_CHECKED(obj, malloc, malloc_exit);
- ATTACH_UPROBE_CHECKED(skel, calloc, calloc_enter);
- ATTACH_URETPROBE_CHECKED(skel, calloc, calloc_exit);
+ ATTACH_UPROBE_CHECKED(obj, calloc, calloc_enter);
+ ATTACH_URETPROBE_CHECKED(obj, calloc, calloc_exit);
- ATTACH_UPROBE_CHECKED(skel, realloc, realloc_enter);
- ATTACH_URETPROBE_CHECKED(skel, realloc, realloc_exit);
+ ATTACH_UPROBE_CHECKED(obj, realloc, realloc_enter);
+ ATTACH_URETPROBE_CHECKED(obj, realloc, realloc_exit);
- ATTACH_UPROBE_CHECKED(skel, mmap, mmap_enter);
- ATTACH_URETPROBE_CHECKED(skel, mmap, mmap_exit);
+ ATTACH_UPROBE_CHECKED(obj, mmap, mmap_enter);
+ ATTACH_URETPROBE_CHECKED(obj, mmap, mmap_exit);
- ATTACH_UPROBE_CHECKED(skel, posix_memalign, posix_memalign_enter);
- ATTACH_URETPROBE_CHECKED(skel, posix_memalign, posix_memalign_exit);
+ ATTACH_UPROBE_CHECKED(obj, posix_memalign, posix_memalign_enter);
+ ATTACH_URETPROBE_CHECKED(obj, posix_memalign, posix_memalign_exit);
- ATTACH_UPROBE_CHECKED(skel, memalign, memalign_enter);
- ATTACH_URETPROBE_CHECKED(skel, memalign, memalign_exit);
+ ATTACH_UPROBE_CHECKED(obj, memalign, memalign_enter);
+ ATTACH_URETPROBE_CHECKED(obj, memalign, memalign_exit);
- ATTACH_UPROBE_CHECKED(skel, free, free_enter);
- ATTACH_UPROBE_CHECKED(skel, munmap, munmap_enter);
+ ATTACH_UPROBE_CHECKED(obj, free, free_enter);
+ ATTACH_UPROBE_CHECKED(obj, munmap, munmap_enter);
// the following probes are intentinally allowed to fail attachment
// deprecated in libc.so bionic
- ATTACH_UPROBE(skel, valloc, valloc_enter);
- ATTACH_URETPROBE(skel, valloc, valloc_exit);
+ ATTACH_UPROBE(obj, valloc, valloc_enter);
+ ATTACH_URETPROBE(obj, valloc, valloc_exit);
// deprecated in libc.so bionic
- ATTACH_UPROBE(skel, pvalloc, pvalloc_enter);
- ATTACH_URETPROBE(skel, pvalloc, pvalloc_exit);
+ ATTACH_UPROBE(obj, pvalloc, pvalloc_enter);
+ ATTACH_URETPROBE(obj, pvalloc, pvalloc_exit);
// added in C11
- ATTACH_UPROBE(skel, aligned_alloc, aligned_alloc_enter);
- ATTACH_URETPROBE(skel, aligned_alloc, aligned_alloc_exit);
+ ATTACH_UPROBE(obj, aligned_alloc, aligned_alloc_enter);
+ ATTACH_URETPROBE(obj, aligned_alloc, aligned_alloc_exit);
return 0;Now memleak report prints foo function.
root@ubuntu-Standard-PC-i440FX-PIIX-1996:~# ./memleak -p 8081
using default object: libc.so.6
using page size: 4096
tracing kernel: false
libbpf: Error in bpf_create_map_xattr(uw_stacks):Invalid argument(-22). Retrying
without BTF.
Tracing outstanding memory allocs... Hit Ctrl-C to end
^C[1:43:4] Top 1 stacks with outstanding allocations:
4 bytes in 1 allocations from stack
get_entries, err: 0
0 [<00007f348d6e1dad>] _Znwm+0x1d [/usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.19]
1 [<000055863a200798>] _Z3foov+0xe [/home/worker/a.out]
2 [<000055863a2007de>] main+0x44 [/home/worker/a.out]
3 [<00007f348d2dfec5>] __libc_start_main+0xf5 [/lib/x86_64-linux-gnu/libc-2.19.so]
4 [<000055863a2006aa>] _start+0x2a [/home/worker/a.out]
done
root@ubuntu-Standard-PC-i440FX-PIIX-1996:~#
In this test program
#include <chrono>
#include <thread>
int* foo() { return new int; }
int main(int argc, char* argv[]) {
while (true) {
auto p = foo();
std::this_thread::sleep_for(std::chrono::seconds(5));
}
return 0;
}from bcc.
Related Issues (20)
- can't deny access to a specific file
- can't deny access to a specific file
- no matching member function for call to 'replace'
- fatal error: clang/Basic/FileManager.h: No such file or directory
- ImportError: cannot import name 'BPF' from 'bcc' (unknown location)
- python-bcc works only if clang-libs Installed, fails with only clang16-libs Installed
- libbpf javagc fails when no PID is provided
- bpflist doesn't show attached probes
- Build fail from source HOT 4
- Fail building from source HOT 1
- attach_raw_socket function does not work well on Qualcomm modem HOT 1
- Build bcc from source code failed HOT 1
- Unable to obtain the complete function call stack of `__memset_avx2_erms`.
- Backported probe_user_read* probe_kernel_read* are not used by bcc HOT 3
- some libbpf-tools tooling / libbpf seems broken on thinlto-compiled kernel
- Not sure I understand the full issue or suggestion here. As it is now, we're just using standard gcc logic when compiling. HOT 1
- `bcc_procutils_which_so(libname, pid)` finds library unused by the process
- BCC source setup make error on Debian Bookworm HOT 1
- attach to specific address of user space spplication in c
- Using funclatency.py to trace bpf_map_lookup_elem
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from bcc.