Comments (10)
edeNFed
commented on July 24, 2024
2
@damemi I found a way to allocate memory without the launcher, will open a PR that includes the new mechanism soon.
from opentelemetry-go-instrumentation.
edeNFed
commented on July 24, 2024
1
Sure, will open a PR shortly
from opentelemetry-go-instrumentation.
damemi
commented on July 24, 2024
ping @edeNFed I realize this is probably not a bug, but something I'm doing wrong. do you know what this issue could be?
from opentelemetry-go-instrumentation.
edeNFed
commented on July 24, 2024
I'll try to explain why we need this map and how it is created.
-
Trace ids and span ids are generated in eBPF (if not propagated from the incoming request). In order to add the generated values to the request headers we need to be able to read them from within the Go application, which obviously can't access memory in kernel space. The way we solve this is by allocating off-heap memory on the Go application (1Mb if I am not mistaken) which the eBPF program writes into. This allocation happens only once - when the application starts.
-
You can create this memory area in two ways: A. via code - invoke the
mmapsyscall and the instrumentation agent will find it. B. via launcher - a small program that I wrote that allocates this map area and then starts the target application. Unfortunately, I didn't have time to release it as open source yet (hopefully will happen next week). For now, you can use the docker imagekeyval/launcher:v0.1and use it like here
Hope that makes sense, please let me know if it's still not clear.
from opentelemetry-go-instrumentation.
damemi
commented on July 24, 2024
Thanks @edeNFed, that's very helpful! So is it correct that right now, this auto-instrumentation requires either code changes (invoking mmap) or to use your launcher to work?
It would be great if you could contribute the code/dockerfile for the launcher to this repo, so we can package something users can run out of the box.
from opentelemetry-go-instrumentation.
dineshg13
commented on July 24, 2024
@edeNFed as suggested on slack, i started to use https://github.com/keyval-dev/launcher , to launch the binary . Am now getting a new error
{"level":"info","ts":1670773134.3289542,"caller":"cli/main.go:37","msg":"starting Go OpenTelemetry Agent ..."}
{"level":"info","ts":1670773134.3289938,"caller":"opentelemetry/controller.go:107","msg":"Establishing connection to OpenTelemetry collector ..."}
{"level":"info","ts":1670773136.3361652,"caller":"process/discover.go:57","msg":"found process","pid":112920}
{"level":"info","ts":1670773136.3364372,"caller":"process/analyze.go:88","msg":"found addr of keyval map","addr":139924897234944}
{"level":"info","ts":1670773136.3405275,"caller":"process/analyze.go:151","msg":"found relevant function for instrumentation","function":"google.golang.org/grpc/internal/transport.(*http2Client).createHeaderFields","returns":3}
{"level":"info","ts":1670773136.340637,"caller":"process/analyze.go:151","msg":"found relevant function for instrumentation","function":"google.golang.org/grpc.(*ClientConn).Invoke","returns":2}
{"level":"info","ts":1670773136.3408015,"caller":"process/analyze.go:151","msg":"found relevant function for instrumentation","function":"google.golang.org/grpc.(*Server).handleStream","returns":5}
{"level":"info","ts":1670773136.340816,"caller":"cli/main.go:79","msg":"target process analysis completed","pid":112920,"go_version":"1.19.1","dependencies":{"github.com/golang/protobuf":"v1.5.2","github/dineshg13/goplay/libs/logging":"v0.0.0-00010101000000-000000000000","github/dineshg13/goplay/protos":"v0.0.0-00010101000000-000000000000","go.uber.org/atomic":"v1.7.0","go.uber.org/multierr":"v1.6.0","go.uber.org/zap":"v1.24.0","golang.org/x/net":"v0.0.0-20220722155237-a158d28d115b","golang.org/x/sys":"v0.0.0-20220722155257-8c9f86f7a55f","golang.org/x/text":"v0.4.0","google.golang.org/genproto":"v0.0.0-20200526211855-cb27e3aa2013","google.golang.org/grpc":"v1.49.0","google.golang.org/protobuf":"v1.28.1"},"total_functions_found":3}
{"level":"info","ts":1670773136.341011,"caller":"cli/main.go:85","msg":"invoking instrumentors"}
{"level":"info","ts":1670773136.3461616,"logger":"allocator","caller":"allocator/allocator_linux.go:34","msg":"Loading allocator","start_addr":139924897234944,"end_addr":139924897247232}
{"level":"info","ts":1670773136.3470068,"caller":"instrumentors/runner.go:83","msg":"loading instrumentor","name":"google.golang.org/grpc"}
{"level":"info","ts":1670773136.3480148,"caller":"inject/injector.go:82","msg":"Injecting variables","vars":{"clientconn_target_ptr_pos":24,"end_addr":139924897247232,"is_registers_abi":true,"start_addr":139924897234944,"total_cpus":32}}
{"level":"error","ts":1670773136.3515298,"caller":"instrumentors/runner.go:86","msg":"error while loading instrumentors, cleaning up","name":"google.golang.org/grpc","error":"field UprobeHttp2ClientCreateHeaderFields: program uprobe_Http2Client_CreateHeaderFields: load program: permission denied: R1 type=ctx expected=fp\n0: R1=ctx(off=0,imm=0) R10=fp0\n; int uprobe_Http2Client_CreateHeaderFields(struct pt_regs *ctx)\n0: (bf) r6 = r1 ; R1=ctx(off=0,imm=0) R6_w=ctx(off=0,imm=0)\n1: (b7) r1 = 0 ; R1_w=0\n; struct go_slice slice = {};\n2: (7b) *(u64 *)(r10 -96) = r1 ; R1_w=P0 R10=fp0 fp-96_w=00000000\n3: (7b) *(u64 *)(r10 -104) = r1 ; R1_w=P0 R10=fp0 fp-104_w=00000000\n; if (is_registers_abi)\n4: (18) r1 = 0xffff9c5890049510 ; R1_w=map_value(off=0,ks=4,vs=48,imm=0)\n6: (71) r2 = *(u8 *)(r1 +0) ; R1_w=map_value(off=0,ks=4,vs=48,imm=0) R2_w=scalar(umax=255,var_off=(0x0; 0xff))\n; if (is_registers_abi)\n7: (7b) *(u64 *)(r10 -448) = r6 ; R6_w=ctx(off=0,imm=0) R10=fp0 fp-448_w=ctx\n8: (15) if r2 == 0x0 goto pc+15 ; R2_w=scalar(umax=255,var_off=(0x0; 0xff))\n; slice.array = (void *)ctx->rax;\n9: (79) r1 = *(u64 *)(r6 +80) ; R1_w=scalar() R6_w=ctx(off=0,imm=0)\n; slice.array = (void *)ctx->rax;\n10: (7b) *(u64 *)(r10 -104) = r1 ; R1_w=scalar() R10=fp0 fp-104_w=mmmmmmmm\n; slice.len = (s32)ctx->rbx;\n11: (7```
from opentelemetry-go-instrumentation.
edeNFed
commented on July 24, 2024
@dineshg13 Can you please give more details about the kernel version that you are using? it will help me to try to reproduce it (output of running uname -r will be great)
from opentelemetry-go-instrumentation.
dineshg13
commented on July 24, 2024
@edeNFed My kernel version 5.19.0-26-generic . This is Ubuntu 22.04 with all updates installed.
from opentelemetry-go-instrumentation.
damemi
commented on July 24, 2024
@edeNFed just checking, were you planning to donate the launcher code as well? I think that's the final piece of this
from opentelemetry-go-instrumentation.
mariomac
commented on July 24, 2024
This PR should help if you are not instrumenting GRPC services: #32
from opentelemetry-go-instrumentation.
Related Issues (20)
- Remove support for Go versions below 1.17 HOT 3
- Reduce pointers usage in TargetDetails
- Refactor `getGoDetails()` to use Go `buildinfo` package
- Warning: cast to smaller integer type HOT 1
- autoinstrumentation-go crashes with "offset not found: net/http.response:status (1.22.0)" error HOT 4
- Add log verbosity configuration and move the log in controller.go to debug level HOT 7
- License Scan/Remediation Questions HOT 1
- Cross processes context propagation not working in the getting started doc
- Define beta release goals HOT 3
- Ctrl-C signal not handled while searching for the target PID
- auto instrument failed: offset not found: net/http.response:status HOT 8
- Separate e2e fixture generation and validation
- Audit and update `database/sql` instrumentation to comply with SQL semantic conventions
- Fix offset generation cache HOT 2
- Add feature gate for unstable/experimental libraries HOT 1
- Remove either `helm repo add` or `git clone` in fixture command
- How to set the ratio of the sampler
- bpf error log:failed to write to userspace
- Add an opt-in environment variable for experimental instrumentation HOT 22
- Correctly set `IsRecording` from default OTel default global impl
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from opentelemetry-go-instrumentation.