Miscalulation of offset with consecutive bitfields about bcc HOT 9 CLOSED

Yes, bcc won't use vmlinux.h and libbpf-tools is using vmlinux.h.

from bcc.

the full reproducer can be found here, and below is the LLVM IR of the program for completeness. As far as I can tell the IR looks correct.

define dso_local i32 @kprobe__netif_rx(ptr noundef %0) local_unnamed_addr #0 section ".bpf.fn.kprobe__netif_rx" !dbg !67 {
  %2 = alloca %struct.event_t, align 8, !DIAssignID !104
  %3 = getelementptr inbounds %struct.pt_regs, ptr %0, i64 0, i32 14, !dbg !108
  %4 = load i64, ptr %3, align 8, !dbg !108, !tbaa !109
  %5 = inttoptr i64 %4 to ptr, !dbg !114
  %6 = getelementptr inbounds %struct.my_struct, ptr %5, i64 0, i32 1, !dbg !115
  %7 = ptrtoint ptr %6 to i64, !dbg !116
  %8 = sub nsw i64 %7, %4, !dbg !117
  store i64 %8, ptr %2, align 8, !dbg !118, !tbaa !119, !DIAssignID !121
  %9 = getelementptr inbounds %struct.event_t, ptr %2, i64 0, i32 1, !dbg !122
  store i64 24, ptr %9, align 8, !dbg !123, !tbaa !124, !DIAssignID !125
  ...
}

from bcc.

@shunghsiyu thanks for reporting. This is not a compiler bug but rather it is an unfortunate situation with the interaction between x86 IR and BPF backend. Note that currently, currently, the original bpf program is compiled first with x86 arch. The reason is we need x86 headers in compiling bpf programs. Once IR is generated, the x86 IR will feed into bpf backend. In certain situations, this may cause the problem. The following is a simple example to demonstrate:

$ cat t.c
struct my_struct {
        long: 64;
        long: 64;
        long: 64;
        void *ptr;
};

struct event_t {
        long offset0;
        long offset1;
        long offset2;
};
void bar(void *);

int foo(void *p1) {
        struct event_t event = {};
        void *p2 = 0;

        event.offset1 = ((long)&((struct my_struct*)p1)->ptr) - (long)p1;
        event.offset2 = ((long)&((struct my_struct*)p2)->ptr) - (long)p2;
        bar(&event);
        return 0;
}

The compilation steps:

  clang -O2 -S -emit-llvm t.c -Xclang -disable-llvm-passes -o t.ll
  llc -march=bpf -O2 t.ll

With llvm17, the llvm with x86 target has %struct.my_struct = type { i192, ptr }.
With llvm18, we have %struct.my_struct = type { [24 x i8], ptr }.

Since BPF does not have i192, it rounds up to alignment 16 and this caused a problem.

The fix is rather simple, do not use bitfield and proper alignments will be the same for both x86 and bpf.

from bcc.

Thanks for the explanation. So if I understand correctly the reason that offset1 and offset2 differs is because they came from different sources?

offset2 == 24 came from LLVM x86 IR directly, because the (long)&((struct my_struct *)0)->ptr can be easily inferred to be offset calculation by the x86 IR and transformed into a constant. The using the x86 alignment is used here and thus i192 is aligned to 8.

On the other hand offset1 == 32 came from LLVM BPF backend, because x86 IR cannot infer that ((long)&((struct my_struct*)p1)->ptr) - (long)p1 is offset calculation (because -disable-llvm-passes maybe?), therefore compiles it literally into a sequence of instructions involving sub. The BPF backend was then able to transform such sequence into a constant, but there it uses BPF alignment where i192 is aligned to 16 (since f35dae07).

Is the above correct?

from bcc.

Your interpretation largely correct. I didn't dig out in llvm why for i192 type BPF backend put an alignment of 16 though. This probably related to how to handle i<> -> alignment in the arch-specific string.

from bcc.

I didn't dig out in llvm why for i192 type BPF backend put an alignment of 16 though

I think I read on the LLVM's Phabricator archive that "the alignment of the largest defined type (i128:128) will be chosen as the alignment" for types that are not explicitly defined in data layout.

from bcc.

I'm still trying to figure out from the original bug report why consecutive bitfields are used in the first place, and whether it is strictly necessary.

Let's close this issue for now, and I'll reopen if needed. Thanks!

from bcc.

I'm still trying to figure out from the original bug report why consecutive bitfields are used in the first place, and whether it is strictly necessary.

The consecutive bitfields came from vmlinux.h generated by bpftool, which had more long :64 padding than usual because the original BTF has been processed by gen min_core_btf first.

From the existence of libbpf-tools I would guess that bcc is not meant to be used with vmlinux.h, rather it should stick to the kernel header that are produced as part of the kernel build process.

But so far I haven't found reference that confirm or deny the above guess. @yonghong-song do yon know if this is the case? It'd be nice to get an explicit statement on this subject. Thanks!

from bcc.

FWIW we end up working around this by passing the -ffine-grained-bitfield-accesses flag to Clang

from bcc.