grantseltzer / weaver Goto Github PK

Trace Go program execution with uprobes and eBPF

License: Mozilla Public License 2.0

Makefile 2.39% Go 95.90% Shell 1.71%

ebpf weaver bcc linux golang go trace tracing

weaver's Introduction

Weaver

PLEASE READ! - I am currently refactoring Weaver to use libbpf instead of bcc which would include various other major improvements. If you're currently using weaver please be aware that features/bug fixes are being held off until the major refactor occurs. This will be tracked in the branch "refactor"

Weaver is a CLI tool that allows you to trace Go programs in order to inspect what values are passed to specified functions. It leverages eBPF attached to uprobes.

Quick Start

There are two modes of operation, one that uses a 'functions file', and one that extracts a symbol table from a passed binary and filters by Go packages. More information on functionality in docs.

Functions file

Take the following example program:

test_prog.go

package main

//go:noinline
func test_function(int, [2]int) {}

//go:noinline
func other_test_function(rune, int64) {}

func main() {
	test_function(3, [2]int{1, 2})
	other_test_function('a', 33)
}

Let's say we want to know what values are passed to test_function and other_test_function whenever the program is run. Once the program is compiled (make) we just have to create a file which specifies each function to trace:

functions_to_trace.txt

main.test_function(int, [2]int)
main.other_test_function(rune, int64)

Notice that we have to specify the parameter data types. (You can use weaver --types to see what data types are supported.)

Now we can call weaver like so:

sudo weaver -f /path/to/functions_to_trace.txt /path/to/test-prog-binary

Weaver will then sit idle without any output until test-prog is run and the test_function and other_test_function functions are called. This will also work on an already running Go Program.

{"functionName":"main.other_test_function","args":[{"type":"RUNE","value":"a"},{"type":"INT64","value":"33"}],"procInfo":{"pid":43300,"ppid":42754,"comm":"test-prog-binar"}}
{"functionName":"main.test_function","args":[{"type":"INT","value":"3"},{"type":"INT_ARRAY","value":"1, 2"}],"procInfo":{"pid":43300,"ppid":42754,"comm":"test-prog-binar"}}

Package mode

For the same example Go program as above, you can choose to not specify a functions file. The command would like like this:

sudo weaver /path/to/test-prog-binary

This will default to only tracing functions in the main package, however you can use the --packages flag to specify a comma seperated list of packages (typially of the form github.com/x/y)

Output does include argument vlaues in this mode.

{"functionName":"main.main","procInfo":{"pid":44411,"ppid":42754,"comm":"test-prog-binar"}}
{"functionName":"main.test_function","procInfo":{"pid":44411,"ppid":42754,"comm":"test-prog-binar"}}

Note on supported types

Currently weaver supports basic data types but getting support for user defined types is a high priority. Getting following types defined are a work in progress:

user/stdlib defined structs
user/stdlib defined interfaces

System Dependencies

bcc / bcc-devel
linux kernel version > 4.14 (please make bug reports if your kernel version doesn't work)

Build

make will compile the weaver binary to bin/weaver (It also creates the smoke test binary and print-stack utility)

Can't build? Please make an issue!

Roadmap

Check issues for tasks currently being tracked. Please open bug reports, i'm sure there are plenty :-)

Short term goals include:

Testing
Output options
Inspecting binaries for parameter data types instead of specifying them with a functions file
CI/CD infrastructre

image modified version of art by Ashley McNamara (license) based on art by Renee French.

weaver's People

Contributors

Stargazers

Watchers

Forkers

isgasho backwardn rrreeezzz amasser wangyanphp wanggancheng ocnix gerifield syllogy th3w4y yago-123 hushi55 dwh110 cwww3 13shivam lyedc zhaohaoru 5l1v3r1 xieguochao

weaver's Issues

Add the ability to read in the value, or address of strings

Currently strings aren't a supported data type (just haven't gotten to it yet). Go stores a pointer to the string on the stack, which needs to be read in as a long (8 bytes) and then de-referenced. That may be able to happen in Go instead of in eBPF, but probably not.

Fix flaky unit testcase in TestReadFunctionsFile

Some of tests occasionally fail because the ordering of the slice is off:

    --- FAIL: TestReadFunctionsFile/empty_line_at_begining_of_file (0.00s)
        functions_file_test.go:122: 
            	Error Trace:	functions_file_test.go:122
            	Error:      	Not equal: 
            	            	expected: []main.functionTraceContext{main.functionTraceContext{binaryName:"", FunctionName:"main.foobar", HasArguments:true, Arguments:[]main.argument{main.argument{CType:"long", goType:1, StartingOffset:8, VariableName:"argument1", PrintfFormat:"%ld", TypeSize:8, ArrayLength:0}, main.argument{CType:"long", goType:5, StartingOffset:16, VariableName:"argument2", PrintfFormat:"%ld", TypeSize:8, ArrayLength:0}}}, main.functionTraceContext{binaryName:"", FunctionName:"main.buzbaz", HasArguments:true, Arguments:[]main.argument{main.argument{CType:"char", goType:13, StartingOffset:8, VariableName:"argument1", PrintfFormat:"%t", TypeSize:1, ArrayLength:0}, main.argument{CType:"int", goType:16, StartingOffset:12, VariableName:"argument2", PrintfFormat:"%c", TypeSize:4, ArrayLength:0}}}}
            	            	actual  : []main.functionTraceContext{main.functionTraceContext{binaryName:"", FunctionName:"main.buzbaz", HasArguments:true, Arguments:[]main.argument{main.argument{CType:"char", goType:13, StartingOffset:8, VariableName:"argument1", PrintfFormat:"%t", TypeSize:1, ArrayLength:0}, main.argument{CType:"int", goType:16, StartingOffset:12, VariableName:"argument2", PrintfFormat:"%c", TypeSize:4, ArrayLength:0}}}, main.functionTraceContext{binaryName:"", FunctionName:"main.foobar", HasArguments:true, Arguments:[]main.argument{main.argument{CType:"long", goType:1, StartingOffset:8, VariableName:"argument1", PrintfFormat:"%ld", TypeSize:8, ArrayLength:0}, main.argument{CType:"long", goType:5, StartingOffset:16, VariableName:"argument2", PrintfFormat:"%ld", TypeSize:8, ArrayLength:0}}}}
            	            	
            	            	Diff:
            	            	--- Expected
            	            	+++ Actual
            	            	@@ -1,2 +1,27 @@
            	            	 ([]main.functionTraceContext) (len=2) {
            	            	+ (main.functionTraceContext) {
            	            	+  binaryName: (string) "",
            	            	+  FunctionName: (string) (len=11) "main.buzbaz",
            	            	+  HasArguments: (bool) true,
            	            	+  Arguments: ([]main.argument) (len=2) {
            	            	+   (main.argument) {
            	            	+    CType: (string) (len=4) "char",
            	            	+    goType: (main.goType) 13,
            	            	+    StartingOffset: (int) 8,
            	            	+    VariableName: (string) (len=9) "argument1",
            	            	+    PrintfFormat: (string) (len=2) "%t",
            	            	+    TypeSize: (int) 1,
            	            	+    ArrayLength: (int) 0
            	            	+   },
            	            	+   (main.argument) {
            	            	+    CType: (string) (len=3) "int",
            	            	+    goType: (main.goType) 16,
            	            	+    StartingOffset: (int) 12,
            	            	+    VariableName: (string) (len=9) "argument2",
            	            	+    PrintfFormat: (string) (len=2) "%c",
            	            	+    TypeSize: (int) 4,
            	            	+    ArrayLength: (int) 0
            	            	+   }
            	            	+  }
            	            	+ },
            	            	  (main.functionTraceContext) {
            	            	@@ -25,27 +50,2 @@
            	            	   }
            	            	- },
            	            	- (main.functionTraceContext) {
            	            	-  binaryName: (string) "",
            	            	-  FunctionName: (string) (len=11) "main.buzbaz",
            	            	-  HasArguments: (bool) true,
            	            	-  Arguments: ([]main.argument) (len=2) {
            	            	-   (main.argument) {
            	            	-    CType: (string) (len=4) "char",
            	            	-    goType: (main.goType) 13,
            	            	-    StartingOffset: (int) 8,
            	            	-    VariableName: (string) (len=9) "argument1",
            	            	-    PrintfFormat: (string) (len=2) "%t",
            	            	-    TypeSize: (int) 1,
            	            	-    ArrayLength: (int) 0
            	            	-   },
            	            	-   (main.argument) {
            	            	-    CType: (string) (len=3) "int",
            	            	-    goType: (main.goType) 16,
            	            	-    StartingOffset: (int) 12,
            	            	-    VariableName: (string) (len=9) "argument2",
            	            	-    PrintfFormat: (string) (len=2) "%c",
            	            	-    TypeSize: (int) 4,
            	            	-    ArrayLength: (int) 0
            	            	-   }
            	            	-  }
            	            	  }
            	Test:       	TestReadFunctionsFile/empty_line_at_begining_of_file
        functions_file_test.go:124: Test failed.

Create a parser for go source code which reads function names and parameter data types to generate a functions file

This may be a good place to start: http://notes.eatonphil.com/interpreting-go.html

Change JSON to default instead of table output format

The table is really ugly, not particularly happy with the package we're using.

Add slice/array support

Deep pointer inspection

It would be great for users to be able to specify pointer parameters (i.e. *int or **int) and to have the option to get the underlying value of that pointer.

Add compile option for granting weaver binary CAP_SYS_ADMIN

Loading ebpf requires CAP_SYS_ADMIN or to be run as root.

[feature request] Create a helper/tester tool or library for inspectig the full contents of stack

When trying to determine how the Go stack is organized when entering into a function I had an eBPF program that would print out the stack one byte at a time for a Go program with various arrangements of argument data types.

Creating a general solution would be great for testing different versions of Go/system compatibility

	        int i;
		char y;
		for (i = 0; i < 50; i++) {
			char *x = (char *)decSP+i; 
			bpf_probe_read(&y, sizeof(y), x);
			bpf_trace_printk("%d\n", y);
		}

Add more detail to list of dependencies/system compatibility

Minimum types to index when parsing DWARF

dwarf.TagCompileUnit
dwarf.TagSubprogram
dwarf.TagVariable
dwarf.TagTag(0)
dwarf.TagFormalParameter
dwarf.TagLexDwarfBlock
dwarf.TagInlinedSubroutine
dwarf.TagConstant
dwarf.TagUnspecifiedType
dwarf.TagPointerType
dwarf.TagBaseType
dwarf.TagStructType
dwarf.TagMember
dwarf.TagTypedef
dwarf.TagSubroutineType
dwarf.TagArrayType
dwarf.TagSubrangeType

weaver does not error out if a binary path is not specified

Add support for rune's

rune's are just int32's except with different stringf modifiers

Output should include PID, PPID, COMM, and other process context information

bcc gives most of this for free with bpf_get_current_task

[bug] `make` fail because encounter “not enough arguments in call to (_C2func_bpf_attach_uprobe)”

Describe the bug
When I wanted to make this project according to the guidance of README, I encountered the following error，

Environment

kernel version: linux 5.10.134-14.1.al8.x86_64
linux release: centOS 8.x

How to fix
I refer to the current code of gobpf, I think a parameter should be added at the end of line 261 of vendor/github.com/iovisor/gobpf/bcc/module.go like this:

// - res, err := C.bpf_attach_uprobe(C.int(fd), attachType, evNameCS, binaryPathCS, (C.uint64_t)(addr), (C.pid_t)(pid))
res, err := C.bpf_attach_uprobe(C.int(fd), attachType, evNameCS, binaryPathCS, (C.uint64_t)(addr), (C.pid_t)(pid), 0)

`weaver --types` output should be in logical order

This output order doesn't make much sense:

[*] ./bin/oster -t    
UINT
RUNE
INT
UINT64
FLOAT64
STRING
UINT16
INT32
INT64
UINT32
BYTE
STRUCT
POINTER
INT16
UINT8
FLOAT32
BOOL
INT8

Go 1.17 will support register based arguments instead of the stack (feature flagged)

Will certainly still be retrievable but will need to be feature flagged. I believe the go version string is passed into the binary debug information too so could auto-detect.

golang/go#40724

Add struct value inspection

If a struct is a parameter we'd like to be able to print the individual field values instead of a pointer to the struct for example.

Weaver does not error out if specified binary doesn't exist

Support map parameters

At first can just print the addr of it, but later can add logic for printing out the contents of the map

Strip null's from end of strings

Currently when tracing strings a lot of padded null's are appended to the end of strings:

[*] sudo ./bin/oster --function-to-trace='main.test_function(int, string)' ./bin/test-prog
{"Type":"INT","Value":"3"}
{"Type":"STRING","Value":"hello\u0000\u0000\u0000\u0000\u0000\u0000\u0000"}

Let's remove those after treating the user space byte array as little endian.

Add a flag for output streams (stdout/stderr/file)

Add the ability to trace multiple functions (or perhaps a package) at a time

Investigate using cilium/eBPF package to replace bcc

This might make weaver a lot more portable: https://github.com/cilium/ebpf

Add flag for changing output format

For example, adding all parameters from a single call into one JSON object and spitting that out.

More robust integration testing

The smoke-test breaks pretty easily. We should have go tests that actually execute weaver and do typed assertions for checking output.

This'll make it really easy to port to new systems too.

Add the ability to let users specify method receiver to be traced

Sequence numbering/ordering for function calls

Currently there's no order for function calls, just whenever they come out of the kernel. It'd be useful to associate invocations in a map and order them.

Add support for inspecting interface parameters on the stack

Oster needs to support multiple (or many) functions at a time

Currently having oster support only a single function isn't terribly useful. Being able to trace functions far down a call stack would be a really useful debugging tool.

Add a check to see if uprobe is already attached to symbol

Multiple uprobes cause duplicates of output

Proposal: Is it possible to inspect the data types of function, or method, symbols in binaries?

Setup CI/CD Infrastructure for integration testing

Create mode of operation where instead of functions file, just trace the function calls without param types

We can just inspect the symbol table and attach uprobes to every function in a list of a specified packages (default: just main), and have it print out just when the function is called. If #8 is confirmed then this would just be the default mode of operation.

Functions files might still be useful for people who want fine grained control of what's traced

Run build script which creates a docker image containing all dependencies and the oster binary
Have a run script which emulates the oster CLI except under the hood runs oster inside the container with the correct permissions for accessing the kernel facilities

Also can have the run script run the build script if it hasn't been run yet.

Add the ability to specify a PID instead of a binary path for tracing

Use arch and OS build constraints

Weaver is built for x86_64 linux platforms. It would be great in the future to support ARM, but for now it'd make sense to use build constraints