unable to run in aarch64 about libbpf-bootstrap HOT 5 CLOSED

Hello,

I was trying to prepare libbpf-bootstrap on aarch64 using BTF and CORE. When I cloned the GIT, I realized that the bbpftool bin was built for x86. Hence I rebilt the “bpftool” from the kernel tree (v 5.9.0) on aarch64 base board. When I try to move the bpftool to my final board, it missed the shared library (libbfd-2.30-system.so & libopcodes-2.30-system.so),

Might be a good idea to try to get a static build of bpftool with those libraries compiled in. And have bpftool pre-compiled in libbpf-bootstrap for most popular architectures. Would make this less painful for sure.

Which I copied from the base board to make the build through. Also I update the ARCH in the libbpf-bootstrap-master/src/Makefile to ARCH := $(shell uname -m | sed 's/x86_64/x86/aarch64') to consider aarch64

This doesn't sound right, it doesn't look like a valid sed invocations. What was the output of uname -m, when you ran it in your terminal? If it's already aarch64 you should have been fine without this step.

When I try to run, I get the following errors. I am not sure the steps I follow are right or am I missing something.

Any hint would be of great help.
Thankyou for your support.

Tools used:
Clang version: clang version 10.0.1 Target: aarch64-fsl-linux, Thread model: posix, InstalledDir: /usr/bin
Kernel version: 5.4.69
Pahole version used at kernel build: V1.9

v1.9 is too old. You need at least 1.16 to get BTF information properly generated. You need BTF for CO-RE. You can check like this:

$ readelf -S ~/linux-build/default/vmlinux  | grep -A1 BTF
  [13] .BTF              PROGBITS         ffffffff826d75b8  018d75b8
       00000000004b4a4d  0000000000000000   A       0     0     1

If you see .BTF with non-zero size, you should have a good BTF data.

libbpf: sec '.reltp/syscalls/sys_enter_write': collecting relocation for section(2) 'tp/syscalls/sys_enter_write'
libbpf: sec '.reltp/syscalls/sys_enter_write': relo #0: insn #2 against 'my_pid'
libbpf: prog 'handle_tp': found data map 0 (minimal_.bss, sec 4, off 0) for insn 2
libbpf: map 'minimal_.bss': created successfully, fd=4
Segmentation fault

This shouldn't happen. Can you get a stack trace for this? It's not supposed to crash in any case.

~/libbpf-bootstrap-master/src# ./bootstrap -v
libbpf: loading object 'bootstrap_bpf' from buffer

libbpf: prog 'handle_exit': found map 1 (rb, sec 6, off 32) for insn #35
libbpf: loading kernel BTF '/sys/kernel/btf/vmlinux': 0

This suggests you do have valid BTF, but then maybe your pahole version is not really 1.9?

libbpf: map 'exec_start': created successfully, fd=4
libbpf: map 'rb': failed to create: Invalid argument(-22)

This means that your kernel doesn't support BPF ringbuf yet. You can try switching to perf buffer, it's available on earlier kernel versions. See https://nakryiko.com/posts/bpf-ringbuf/ for examples of using perf buffer instead of BPF ringbuf.

While generation the “bpftool”, clang-bpf-co-re: [ OFF ] was there, I am not sure either it could cause the problem

This is fine, it's CO-RE for internal bpftool needs. It will work without it for your case just fine, don't worry.

from libbpf-bootstrap.

Might be a good idea to try to get a static build of bpftool with those libraries compiled in. And have bpftool pre-compiled in libbpf-bootstrap for most popular architectures. Would make this less painful for sure.

Yes, you are right. I should make the tool as statically linked to avoid copying the shared objects.

Which I copied from the base board to make the build through. Also I update the ARCH in the libbpf-bootstrap-master/src/Makefile to ARCH := $(shell uname -m | sed 's/x86_64/x86/aarch64') to consider aarch64

This doesn't sound right, it doesn't look like a valid sed invocations. What was the output of uname -m, when you ran it in your terminal? If it's already aarch64 you should have been fine without this step.

Yes, you are right, I don't need the aarch64 as per below response

# uname -m
aarch64

When I try to run, I get the following errors. I am not sure the steps I follow are right or am I missing something.
Any hint would be of great help.
Thankyou for your support.
Tools used:
Clang version: clang version 10.0.1 Target: aarch64-fsl-linux, Thread model: posix, InstalledDir: /usr/bin
Kernel version: 5.4.69
Pahole version used at kernel build: V1.9

v1.9 is too old. You need at least 1.16 to get BTF information properly generated. You need BTF for CO-RE. You can check like this:

Sorry for the typo. I did the source build on V1.19

~/libbpf-bootstrap-master/src#pahole --version
v1.19

$ readelf -S ~/linux-build/default/vmlinux  | grep -A1 BTF
  [13] .BTF              PROGBITS         ffffffff826d75b8  018d75b8
       00000000004b4a4d  0000000000000000   A       0     0     1

If you see .BTF with non-zero size, you should have a good BTF data.

Here is the output

#readelf -S vmlinux | grep -A1 BTF
[11] .BTF PROGBITS ffffffc010e874a0 00e174a0
000000000022e31b 0000000000000000 A 0 0 1

libbpf: sec '.reltp/syscalls/sys_enter_write': collecting relocation for section(2) 'tp/syscalls/sys_enter_write'
libbpf: sec '.reltp/syscalls/sys_enter_write': relo #0: insn #2 against 'my_pid'
libbpf: prog 'handle_tp': found data map 0 (minimal_.bss, sec 4, off 0) for insn 2
libbpf: map 'minimal_.bss': created successfully, fd=4
Segmentation fault

This shouldn't happen. Can you get a stack trace for this? It's not supposed to crash in any case.

Backtrace with GDB:

~/libbpf-bootstrap-master/src# gdb gnucrash
GNU gdb (GDB) 8.3.1
Copyright (C) 2019 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "aarch64-fsl-linux".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
gnucrash: No such file or directory.
(gdb) pwd
Working directory /home/root/libbpf-bootstrap-master/src.
(gdb) file ./minimal
Reading symbols from ./minimal...
(gdb) run
Starting program: /home/root/libbpf-bootstrap-master/src/minimal
libbpf: loading object 'minimal_bpf' from buffer
libbpf: elf: section(2) tp/syscalls/sys_enter_write, size 192, link 0, flags 6, type=1
libbpf: sec 'tp/syscalls/sys_enter_write': found program 'handle_tp' at insn offset 0 (0 bytes), code size 24 insns (192 bytes)
libbpf: elf: section(3) license, size 13, link 0, flags 3, type=1
libbpf: license of minimal_bpf is Dual BSD/GPL
libbpf: elf: section(4) .bss, size 4, link 0, flags 3, type=8
libbpf: elf: section(5) .rodata.str1.1, size 28, link 0, flags 32, type=1
libbpf: elf: skipping unrecognized data section(5) .rodata.str1.1
libbpf: elf: section(6) .BTF, size 490, link 0, flags 0, type=1
libbpf: elf: section(7) .BTF.ext, size 160, link 0, flags 0, type=1
libbpf: elf: section(8) .symtab, size 168, link 13, flags 0, type=2
libbpf: elf: section(9) .reltp/syscalls/sys_enter_write, size 16, link 8, flags 0, type=9
libbpf: looking for externs among 7 symbols...
libbpf: collected 0 externs total
libbpf: map 'minimal_.bss' (global data): at sec_idx 4, offset 0, flags 400.
libbpf: map 0 is "minimal_.bss"
libbpf: sec '.reltp/syscalls/sys_enter_write': collecting relocation for section(2) 'tp/syscalls/sys_enter_write'
libbpf: sec '.reltp/syscalls/sys_enter_write': relo #0: insn #2 against 'my_pid'
libbpf: prog 'handle_tp': found data map 0 (minimal_.bss, sec 4, off 0) for insn 2
libbpf: map 'minimal_.bss': created successfully, fd=4

Program received signal SIGSEGV, Segmentation fault.
main (argc=1, argv=0x7ffffffc28) at minimal.c:46
46              skel->bss->my_pid = getpid();
(gdb) bt full
#0  main (argc=1, argv=0x7ffffffc28) at minimal.c:46
        skel = 0x55555912a0
        err = 85
(gdb)
(gdb) file ./bootstrap
Reading symbols from ./bootstrap...
(gdb) run
Starting program: /home/root/libbpf-bootstrap-master/src/bootstrap
libbpf: map 'rb': failed to create: Invalid argument(-22)
libbpf: failed to load object 'bootstrap_bpf'
libbpf: failed to load BPF skeleton 'bootstrap_bpf': -22
Failed to load and verify BPF skeleton
[Inferior 1 (process 25759) exited with code 026]
(gdb)

just checked the dmesg error during the invocation of ./minimal and found the below notice:

~/libbpf-bootstrap-master/src#dmesg
[...]
[ 2122.825324] **********************************************************
[ 2122.825328] **   NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE   **
[ 2122.825332] **                                                      **
[ 2122.825335] ** trace_printk() being used. Allocating extra memory.  **
[ 2122.825339] **                                                      **
[ 2122.825342] ** This means that this is a DEBUG kernel and it is     **
[ 2122.825346] ** unsafe for production use.                           **
[ 2122.825349] **                                                      **
[ 2122.825352] ** If you see this message and you are not debugging    **
[ 2122.825355] ** the kernel, report this immediately to your vendor!  **
[ 2122.825359] **                                                      **
[ 2122.825362] **   NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE   **
[ 2122.825365] **********************************************************

This means that your kernel doesn't support BPF ringbuf yet. You can try switching to perf buffer, it's available on earlier kernel versions. See https://nakryiko.com/posts/bpf-ringbuf/ for examples of using perf buffer instead of BPF ringbuf.

Thankyou.
I am going through your post and taking support of your examples to port into perf buffer.
Will post it as soon as I succeed (coz I am learning by doing )

While generation the “bpftool”, clang-bpf-co-re: [ OFF ] was there, I am not sure either it could cause the problem

This is fine, it's CO-RE for internal bpftool needs. It will work without it for your case just fine, don't worry.

Thanks for the clarification.

from libbpf-bootstrap.

Program received signal SIGSEGV, Segmentation fault.
main (argc=1, argv=0x7ffffffc28) at minimal.c:46
46 skel->bss->my_pid = getpid();

Oh, right, skel->bss is NULL if kernel doesn't support memory-mapping BPF_MAP_TYPE_ARRAY, which backs global variables. So unfortunately you have too old kernel to read or update global variables from user-space after skeleton was loaded. But guess what, you can still initialize it before you load the skeleton, you'll just need to split the __open_and_load() into separate __open() and __load() and do exactly the same initialization before __load(). See how bootstrap is doing that.

This is a not-so-obvious graceful degradation technique which still allows to initialize variables, even if kernel doesn't support the necessary memory-mapping. Sorry that it's not very obvious.

just checked the dmesg error during the invocation of ./minimal and found the below notice:

That's fine, it's a scary-looking message that's printed if you are using bpf_printk() in your BPF program, because it's not supposed to be used in production. Nothing is breaking, don't worry.

Thankyou.
I am going through your post and taking support of your examples to port into perf buffer.
Will post it as soon as I succeed (coz I am learning by doing )

Glad I could help.

from libbpf-bootstrap.

Oh, right, skel->bss is NULL if kernel doesn't support memory-mapping BPF_MAP_TYPE_ARRAY, which backs global variables. So unfortunately you have too old kernel to read or update global variables from user-space after skeleton was loaded. But guess what, you can still initialize it before you load the skeleton, you'll just need to split the __open_and_load() into separate __open() and __load() and do exactly the same initialization before __load(). See how bootstrap is doing that.

Kernel version I use is 5.4.69
I modified minimal.c as proposed (__open(), init, __load()) and it works fine now. Thankyou.
https://github.com/BhaskaranVinithaTR/libbpf-bootstrap/blob/master/src/minimal.c
Would it be ok if I create a pull request ?

just checked the dmesg error during the invocation of ./minimal and found the below notice:

That's fine, it's a scary-looking message that's printed if you are using bpf_printk() in your BPF program, because it's not supposed to be used in production. Nothing is breaking, don't worry.

ok

from libbpf-bootstrap.

Yes, please send a PR. I wanted to demonstrate open_and_load() as a short-cut, but we can do it in a separate example, no big deal.

from libbpf-bootstrap.