Comments (19)
vincentmli
commented on May 26, 2024
it works on 5.5 kernel
[root@centos-dev pwru]# ./pwru --filter-dst-ip=10.169.72.236 --filter-dst-port=8472 --filter-proto=udp --output-tuple
2021/10/20 14:19:13 Attaching kprobes...
1060 / 1060 [--------------------------------------------------------------------------------------------------] 100.00% 29 p/s
Attached (ignored 0)
2021/10/20 14:19:50 Listening for events..
SKB PROCESS FUNC TIMESTAMP
0xffff9a1407361b00 [ping] ip_local_out 979036123709 10.169.72.233:48805->10.169.72.236:8472(udp)
0xffff9a1407361b00 [ping] __ip_local_out 979036131143 10.169.72.233:48805->10.169.72.236:8472(udp)
0xffff9a1407361b00 [ping] nf_hook_slow 979036136262 10.169.72.233:48805->10.169.72.236:8472(udp)
0xffff9a1407361b00 [ping] ip_output 979036595661 10.169.72.233:48805->10.169.72.236:8472(udp)
0xffff9a1407361b00 [ping] nf_hook_slow 979036611100 10.169.72.233:48805->10.169.72.236:8472(udp)
0xffff9a1407361b00 [ping] ip_finish_output 979037170778 10.169.72.233:48805->10.169.72.236:8472(udp)
0xffff9a1407361b00 [ping] __ip_finish_output 979037174425 10.169.72.233:48805->10.169.72.236:8472(udp)
0xffff9a1407361b00 [ping] ip_finish_output2 979037176770 10.169.72.233:48805->10.169.72.236:8472(udp)
0xffff9a1407361b00 [ping] neigh_resolve_output 979037179575 10.169.72.233:48805->10.169.72.236:8472(udp)
0xffff9a1407361b00 [ping] __neigh_event_send 979037181829 10.169.72.233:48805->10.169.72.236:8472(udp)
0xffff9a1407361b00 [ping] eth_header 979037184534 10.169.72.233:48805->10.169.72.236:8472(udp)
0xffff9a1407361b00 [ping] skb_push 979037186678 10.169.72.233:48805->10.169.72.236:8472(udp)
0xffff9a1407361b00 [ping] dev_queue_xmit 979037189303 10.169.72.233:48805->10.169.72.236:8472(udp)
0xffff9a1407361b00 [ping] __dev_queue_xmit 979037191497 10.169.72.233:48805->10.169.72.236:8472(udp)
0xffff9a1407361b00 [ping] netdev_core_pick_tx 979037193712 10.169.72.233:48805->10.169.72.236:8472(udp)
0xffff9a1407361b00 [ping] netdev_pick_tx 979037197088 10.169.72.233:48805->10.169.72.236:8472(udp)
0xffff9a1407361b00 [ping] __skb_get_hash 979037199463 10.169.72.233:48805->10.169.72.236:8472(udp)
0xffff9a1407361b00 [ping] sch_direct_xmit 979037205394 10.169.72.233:48805->10.169.72.236:8472(udp)
0xffff9a1407361b00 [ping] validate_xmit_skb_list 979037207919 10.169.72.233:48805->10.169.72.236:8472(udp)
0xffff9a1407361b00 [ping] validate_xmit_skb 979037209902 10.169.72.233:48805->10.169.72.236:8472(udp)
0xffff9a1407361b00 [ping] netif_skb_features 979037212076 10.169.72.233:48805->10.169.72.236:8472(udp)
0xffff9a1407361b00 [ping] skb_network_protocol 979037214080 10.169.72.233:48805->10.169.72.236:8472(udp)
0xffff9a1407361b00 [ping] validate_xmit_xfrm 979037216424 10.169.72.233:48805->10.169.72.236:8472(udp)
0xffff9a1407361b00 [ping] dev_hard_start_xmit 979037218649 10.169.72.233:48805->10.169.72.236:8472(udp)
0xffff9a1407361b00 [containerd-shim] __dev_kfree_skb_any 979037351640 10.169.72.233:48805->10.169.72.236:8472(udp)
0xffff9a1407361b00 [containerd-shim] consume_skb 979037359014 10.169.72.233:48805->10.169.72.236:8472(udp)
0xffff9a1407361b00 [containerd-shim] skb_release_all 979037361258 10.169.72.233:48805->10.169.72.236:8472(udp)
0xffff9a1407361b00 [containerd-shim] skb_release_head_state 979037363262 10.169.72.233:48805->10.169.72.236:8472(udp)
0xffff9a1407361b00 [containerd-shim] sock_wfree 979037365517 10.169.72.233:48805->10.169.72.236:8472(udp)
0xffff9a1407361b00 [containerd-shim] skb_release_data 979037368672 10.169.72.233:48805->10.169.72.236:8472(udp)
0xffff9a1407361b00 [containerd-shim] skb_free_head 979037370867 10.169.72.233:48805->10.169.72.236:8472(udp)
0xffff9a1407361b00 [containerd-shim] kfree_skbmem 979037373521 10.169.72.233:48805->10.169.72.236:8472(udp)
0xffff9a13af5e6e00 [ksoftirqd/6] ip_local_out 984083409078 10.169.72.233:33754->10.169.72.236:8472(udp)
0xffff9a13af5e6e00 [ksoftirqd/6] __ip_local_out 984083419898 10.169.72.233:33754->10.169.72.236:8472(udp)
0xffff9a13af5e6e00 [ksoftirqd/6] nf_hook_slow 984083422523 10.169.72.233:33754->10.169.72.236:8472(udp)
0xffff9a13af5e6e00 [ksoftirqd/6] ip_output 984083845844 10.169.72.233:33754->10.169.72.236:8472(udp)
0xffff9a13af5e6e00 [ksoftirqd/6] nf_hook_slow 984083852967 10.169.72.233:33754->10.169.72.236:8472(udp)
0xffff9a13af5e6e00 [ksoftirqd/6] ip_finish_output 984084177621 10.169.72.233:33754->10.169.72.236:8472(udp)
0xffff9a13af5e6e00 [ksoftirqd/6] __ip_finish_output 984084183542 10.169.72.233:33754->10.169.72.236:8472(udp)
0xffff9a13af5e6e00 [ksoftirqd/6] ip_finish_output2 984084186678 10.169.72.233:33754->10.169.72.236:8472(udp)
0xffff9a13af5e6e00 [ksoftirqd/6] neigh_resolve_output 984084191337 10.169.72.233:33754->10.169.72.236:8472(udp)
0xffff9a13af5e6e00 [ksoftirqd/6] eth_header 984084194212 10.169.72.233:33754->10.169.72.236:8472(udp)
0xffff9a13af5e6e00 [ksoftirqd/6] skb_push 984084196467 10.169.72.233:33754->10.169.72.236:8472(udp)
0xffff9a13af5e6e00 [ksoftirqd/6] dev_queue_xmit 984084198721 10.169.72.233:33754->10.169.72.236:8472(udp)
0xffff9a13af5e6e00 [ksoftirqd/6] __dev_queue_xmit 984084200695 10.169.72.233:33754->10.169.72.236:8472(udp)
0xffff9a13af5e6e00 [ksoftirqd/6] netdev_core_pick_tx 984084203079 10.169.72.233:33754->10.169.72.236:8472(udp)
0xffff9a13af5e6e00 [ksoftirqd/6] netdev_pick_tx 984084206535 10.169.72.233:33754->10.169.72.236:8472(udp)
0xffff9a13af5e6e00 [ksoftirqd/6] __skb_get_hash 984084209481 10.169.72.233:33754->10.169.72.236:8472(udp)
0xffff9a13af5e6e00 [ksoftirqd/6] sch_direct_xmit 984084216775 10.169.72.233:33754->10.169.72.236:8472(udp)
0xffff9a13af5e6e00 [ksoftirqd/6] validate_xmit_skb_list 984084219189 10.169.72.233:33754->10.169.72.236:8472(udp)
0xffff9a13af5e6e00 [ksoftirqd/6] validate_xmit_skb 984084221554 10.169.72.233:33754->10.169.72.236:8472(udp)
0xffff9a13af5e6e00 [ksoftirqd/6] netif_skb_features 984084223718 10.169.72.233:33754->10.169.72.236:8472(udp)
0xffff9a13af5e6e00 [ksoftirqd/6] skb_network_protocol 984084225792 10.169.72.233:33754->10.169.72.236:8472(udp)
0xffff9a13af5e6e00 [ksoftirqd/6] validate_xmit_xfrm 984084228216 10.169.72.233:33754->10.169.72.236:8472(udp)
0xffff9a13af5e6e00 [ksoftirqd/6] dev_hard_start_xmit 984084230371 10.169.72.233:33754->10.169.72.236:8472(udp)
0xffff9a13af5e6e00 [<empty>] __dev_kfree_skb_any 984084258243 10.169.72.233:33754->10.169.72.236:8472(udp)
0xffff9a13af5e6e00 [<empty>] consume_skb 984084261850 10.169.72.233:33754->10.169.72.236:8472(udp)
0xffff9a13af5e6e00 [<empty>] skb_release_all 984084265387 10.169.72.233:33754->10.169.72.236:8472(udp)
0xffff9a13af5e6e00 [<empty>] skb_release_head_state 984084267831 10.169.72.233:33754->10.169.72.236:8472(udp)
0xffff9a13af5e6e00 [<empty>] skb_release_data 984084270717 10.169.72.233:33754->10.169.72.236:8472(udp)
0xffff9a13af5e6e00 [<empty>] skb_free_head 984084273452 10.169.72.233:33754->10.169.72.236:8472(udp)
0xffff9a13af5e6e00 [<empty>] kfree_skbmem 984084277480 10.169.72.233:33754->10.169.72.236:8472(udp)
from pwru.
aditighag
commented on May 26, 2024
; mark = BPF_CORE_READ(skb, mark);
38: (b7) r2 = 4
39: (85) call unknown#113
invalid func unknown#113
The bpf_core_read.h that defined BPF_CORE_READ was added in 5.5 - https://elixir.bootlin.com/linux/v5.5/source/tools/lib/bpf/bpf_core_read.h#L117. We'll need to use bpf_probe_read for kernels <5.5.
from pwru.
vincentmli
commented on May 26, 2024
; mark = BPF_CORE_READ(skb, mark); 38: (b7) r2 = 4 39: (85) call unknown#113 invalid func unknown#113The
bpf_core_read.hthat definedBPF_CORE_READwas added in 5.5 - https://elixir.bootlin.com/linux/v5.5/source/tools/lib/bpf/bpf_core_read.h#L117. We'll need to usebpf_probe_readfor kernels <5.5.
ok, that sounds good, my issue happens to be in 5.4, I can't think of what tool I can use to trouble shoot the issue, hope pwru could help here.
from pwru.
duanjiong
commented on May 26, 2024
; mark = BPF_CORE_READ(skb, mark); 38: (b7) r2 = 4 39: (85) call unknown#113 invalid func unknown#113The
bpf_core_read.hthat definedBPF_CORE_READwas added in 5.5 - https://elixir.bootlin.com/linux/v5.5/source/tools/lib/bpf/bpf_core_read.h#L117. We'll need to usebpf_probe_readfor kernels <5.5.
In this case we should update the readme, because it says kernel version 5.3
from pwru.
brb
commented on May 26, 2024
@vincentmli For your debugging you could revert 00de303 and build the tool yourself (please refer to README.md how to do that). Let me know if you have problems with this.
I think for older kernels we could rely on bpf_probe_read() and __sk_buff instead (UPDATE: the latter seems to be not available for kprobes. However, it's safe to assume that the offset / size of the relevant sk_buff fields does not change on <5.5).
from pwru.
vincentmli
commented on May 26, 2024
@vincentmli For your debugging you could revert 00de303 and build the tool yourself (please refer to README.md how to do that). Let me know if you have problems with this.
I think for older kernels we could rely on
bpf_probe_read()and__sk_buffinstead (UPDATE: the latter seems to be not available for kprobes. However, it's safe to assume that the offset / size of the relevantsk_bufffields does not change on <5.5).
@brb thanks, git revert has some conflicts so I manually changed the code, it works on 5.4, FYI, I got different output for my issue, do you see any problem there :) cilium/cilium#17528 (comment)
from pwru.
zhangbo1882
commented on May 26, 2024
add a PR to fix it. #27
from pwru.
brb
commented on May 26, 2024
@vincentmli Just stumbled into the issue again, as I am able to run on 5.4 kernel (Ubuntu 20.04).
39: (85) call unknown#113 means that the following function was compiled out on your kernel:
static long (*bpf_probe_read_kernel)(void *dst, __u32 size, const void *unsafe_ptr) = (void *) 113;
Could you attach your kernel configuration and bpftool feature output?
from pwru.
vincentmli
commented on May 26, 2024
@brb I attached bpftool feature and default ubuntu 5.4 kernel config, yes, it would be really nice to run pwru on default ubuntu 5.4 :)
bpftool-feature.txt
config-5.4.0-117-generic.txt
from pwru.
brb
commented on May 26, 2024
@vincentmli Thanks. Interesting, you might be running into the lockdown issues (iovisor/bcc#2565). I am running on the following:
vagrant@vagrant:~$ uname -a
Linux vagrant 5.4.0-110-generic #124-Ubuntu SMP Thu Apr 14 19:46:19 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
vagrant@vagrant:~$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 20.04.4 LTS
Release: 20.04
Codename: focal
Could you try running pwru and then attaching the dmesg output?
from pwru.
vincentmli
commented on May 26, 2024
# pwru version
2022/06/20 15:14:58 Loading objects: field KprobeSkb1: program kprobe_skb_1: load program: invalid argument: ; int kprobe_skb_1(struct pt_regs *ctx) {
.........
39: (85) call unknown#113
invalid func unknown#113
processed 39 insns (limit 1000000) max_states_per_insn 0 total_states 1 peak_states 1 mark_read 1
by the way, i tried to re-build pwru with most recent master branch, I got error
[root@centos-dev pwru]# make
go generate
Generating for amd64
# github.com/cilium/ebpf
vendor/github.com/cilium/ebpf/marshalers.go:102:10: undefined: unsafe.Slice
main_amd64.go:5: running "go": exit status 2
make: *** [Makefile:15: pwru] Error 1
from pwru.
vincentmli
commented on May 26, 2024
also fyi, https://github.com/ehids/ecapture and cilium tetragon runs fine on the same ubuntu
from pwru.
vincentmli
commented on May 26, 2024
also fyi, https://github.com/ehids/ecapture and cilium tetragon runs fine on the same ubuntu
I guess these two project not involving (*bpf_probe_read_kernel)
from pwru.
vincentmli
commented on May 26, 2024
so far I am unable to find evidence that my ubuntu VM is in lockdown mode or not after reading through online resources :)
from pwru.
vincentmli
commented on May 26, 2024
[root@centos-dev pwru]# make go generate Generating for amd64 # github.com/cilium/ebpf vendor/github.com/cilium/ebpf/marshalers.go:102:10: undefined: unsafe.Slice main_amd64.go:5: running "go": exit status 2 make: *** [Makefile:15: pwru] Error 1
I need to upgrade golang to 1.18.3 and above issue is resolved
from pwru.
vincentmli
commented on May 26, 2024
@brb the issue is resolved after I build most recent pwru from mater branch, it might be because I am using an old pwru on this new installed ubuntu 20.04
from pwru.
brb
commented on May 26, 2024
the issue is resolved after I build most recent pwru from mater branch
Do you mean that pwru is able to run on your machine with 5.4 kernel?
from pwru.
vincentmli
commented on May 26, 2024
Do you mean that
pwruis able to run on your machine with 5.4 kernel?
correct
from pwru.
brb
commented on May 26, 2024
Cool, then closing this issue!
from pwru.
Related Issues (20)
- IPv6 filtering is broken
- Release script fails to create group HOT 1
- Add large GHA runner to replace self-hosted
- How to uninstall all probes after a ssh disconnection? HOT 1
- IPv6 L4 Protocol HOT 1
- Enable kprobe-multi backend for kernel modules tracing (--kmods)
- Add datetime to the output HOT 2
- a few minutes delay to complete the pwru multi kprobe attachment HOT 16
- IPv6 filter-src-ip/filter-dst-ip does not filter packets HOT 2
- no BTF found for kernel HOT 2
- pwru fails to run if compiled on machine with more CPUs than target HOT 2
- Long start time on bpf-next (6.4-rc3) HOT 5
- Track SKB clones HOT 6
- Failed to --output-skb on Ubuntu 23.04 HOT 4
- kprobe_pwru.c:set_tuple() reads wrong L4 for ESP on some kprobes HOT 2
- "make release" fails: ln: failed to create symbolic link './clang': File exists
- release.yaml: Cross compile libpcap.a for arm64
- Update vendored cilium/ebpf
- Use clang for libpcap and CGO compilation HOT 5
- Can this watch arp request? HOT 3
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from pwru.