Comments (9)
from #80
It is well known that cgo has bad performance when calling c code, and even worse when calling go callbacks from c (see, for example, https://about.sourcegraph.com/go/gophercon-2018-adventures-in-cgo-performance/).
This is actually not a problem for most of the use cases of libbpfgo, where we just need to load a program and attach it, or update a map, as these operations are not that frequent. It may become a problem, however, when we need to poll for events coming from the kernel through one of the perf/ring buffers, as these are much more frequent.
Suggestion: for buffers polling (either perf or ring buffers) let's implement the logic in pure go. These functions can be added as an alternative API for the already existing functions, and will offer high performance where needed.
Specifically, these are the libbpf functions that we will need to implement: C.perf_buffer__poll() and C.ring_buffer_poll() (both are called by the PerfBuffer/RingBuffer poll() function)
from libbpfgo.
After performing some local tests, it seems that cgo is not the bottleneck of tracee, but the printer.
Below is a pprof output where I used a very noisy event (sched_switch) and the gob printer.
The conclusion is the same for other workloads (e.g. using default event set on an idle system) and other printers (table/json) - the printer path is always worse than that of cgo.
from libbpfgo.
With table printer:
from libbpfgo.
What does this mean for this issue? Isn't it still something we should do?
from libbpfgo.
I still need to find a way to compare a prototype I have with pure go implementation to the current cgo implementation.
With pprof I can only see the bottleneck, but can't quantitatively compare between the two implementation.
Any suggestion for how to do that?
from libbpfgo.
The performance of c to go calls has improved in recent go versions: golang/go#42469 (comment)
If there is no strong evidence that this is still an issue for libbpfgo, we may probably close this one for now
from libbpfgo.
The performance of c to go calls has improved in recent go versions: golang/go#42469 (comment)
If there is no strong evidence that this is still an issue for libbpfgo, we may probably close this one for now
It's also important to note that since we also pass pointers around in our cgo code, that can be much more safely and efficiently done using Cgo Handles as of go 1.17. https://pkg.go.dev/runtime/cgo#Handle
from libbpfgo.
Hey @yanivagman
I saw issue #80 and wondered what did was your plan there? By implementing the polling in pure go you mean just call the epoll_wait from go instead of cgo? Or did you talk about implementing more function inside the perf_buffer_poll?
In my project Im handling performance issues that are caused mainly by the perf_buffer_poll cgo implemention and Im trying to find a solution
As i understand, unless we can somehow trigger the perf callback in pure go there is still going to be a massive cpu consumptions as c-to-go is the most expensive directive in cgo
from libbpfgo.
Hey @yanivagman I saw issue #80 and wondered what did was your plan there? By implementing the polling in pure go you mean just call the epoll_wait from go instead of cgo? Or did you talk about implementing more function inside the perf_buffer_poll? In my project Im handling performance issues that are caused mainly by the perf_buffer_poll cgo implemention and Im trying to find a solution
As i understand, unless we can somehow trigger the perf callback in pure go there is still going to be a massive cpu consumptions as c-to-go is the most expensive directive in cgo
Hi @guyarb,
To implement the polling in pure go, the following changes are required:
- Change InitPerfBuff function by removing the usage of C.init_perf_buf. This function then creates a new perf buffer per each cpu in the system (using perf_event_open()) and mmap()s it to memory. The new fds can then be added to epoll.
- Change the Start function to call a pure go polling function. This function should then wait for events on the epoll fd, and for each ring buffer that got new events, read the received data from it.
A reference implementation is Cilium's ebpf library (that is written in pure go).
When playing with this code and trying to measure differences using pprof, I didn't see major improvements, as described above. It might be that pprof is not the right tool for this task.
In your project, how did you find that the performance issues were caused mainly by the cgo perf_buffer_poll call?
from libbpfgo.
Related Issues (20)
- Split BPFMap API into managed and unmanaged
- Split bpf module, progs, maps into own files
- Wrap C struct field handling
- Split bpf module and program into own files
- LinkType enum needs to align libbpf values
- attach xdp error
- Check type cast results
- [Error] in running go user space code HOT 1
- OSInfo singleton HOT 3
- LIBBPF_MAJOR_VERSION missing HOT 2
- Compiling a user space application that uses libbpfgo as dependency fails HOT 1
- Error compiling go programme HOT 2
- Incorrect parsing of IoUringRequestFlag
- Incompatability between libbpf and libbpfgo HOT 8
- TcHook.Detach() always raises the `failed to detach tc hook: invalid argument` error. HOT 7
- TcHook.Query() always raises the `failed to query tc hook: invalid argument` error. HOT 3
- cookie support HOT 2
- Some tests fail on ARM64 HOT 1
- can show an interaction example between ebpf c and golang that uses ring buffer? HOT 1
- Static builds with newer libelf require `-l zstd` HOT 3
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from libbpfgo.