Git Product home page Git Product logo

wachy's Introduction

wachy

Logo


  A dynamic tracing profiler for Linux


Wachy provides a UI for interactive eBPF-based userspace performance debugging. For an overview, see the website: https://rubrikinc.github.io/wachy/. For background see the introductory blog post.

For more details see the guide.

Compatibility

Wachy requires:

  1. Linux 4.6 or later kernel
  2. Traced binary should be in a compiled language, and have debugging symbols

1 is due to availability of certain eBPF features, and 2 is due to the techniques used by wachy (eBPF uprobes and address to line number mappings from debugging symbols). Wachy also supports C++ symbol demangling - it has mostly been tested with C++ binaries. If you'd like demangling support for a new compiled language, please open an issue (note: despite being compiled, Go does not play well with eBPF). If you have ideas on how to do something similar on other platforms or with other unsupported languages, I'm interested in hearing it!

Wachy also currently only supports x86-64 binaries. If you are interested in other architectures, please open an issue.

Install

Download the latest version from the Releases page.

Wachy relies on bpftrace and the following shared libraries to run: libgcc_s, libncursesw. On ubuntu some of these may be installed by default, but to install them all you can run

sudo apt install bpftrace libgcc1 libncursesw5

If you see strange characters in the TUI, ensure your LANG is set correctly, e.g. before starting wachy, run

export LANG=en_US.UTF-8

Compiling

If you want to build wachy from source, it requires the following development packages: libiberty, ncurses, cmake. On ubuntu you can install them with

sudo apt install libiberty-dev libncurses5-dev libncursesw5-dev cmake

You also need Rust installed.

Then build with cargo

cargo build --release
target/release/wachy --help

Contribution

Unless you explicitly state otherwise, any contribution intentionally submitted for inclusion in the work by you, as defined in the Apache-2.0 license, shall be dual licensed as per LICENSE.md, without any additional terms or conditions.

Contributions to this project must be accompanied by a Contributor License Agreement. We use https://cla-assistant.io to automate this process.

wachy's People

Contributors

cfsnyder avatar mgaurav avatar viveksjain avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

wachy's Issues

CPU architecture independent design

It is nice to have dynamic user space profiling support. Instead of resolving dependencies, another approach that could be considered is to have a light-weight agent running on-board the system streaming the data and have an external server handle the UI part, that could be web-based as well. Regular ftrace based perf tool itself can generate CPU data reports at function level, though not continuously. When combined with the debug symbol resolution and external web server/UI and possibly adding process ID based tracing, that would work out well across CPU architectures.

Unguarded `&self.name_to_symbol.get(&function).unwrap()` can lead to panic

src/program.rs has

    pub fn get_symbol(&self, function: FunctionName) -> &SymbolInfo {
        &self.name_to_symbol.get(&function).unwrap()
    }

which leads to:

Error: Panic! [v0.1.0-alpha.6]
Cause: called Option::unwrap() on a None value
Location: src/program.rs:404:45.

when called (by pressing Enter) on a line without symbols available.

support on aarch64-linux-android

Hi, I found the great tool, and I want to use it on android device.

I installed the android ndk, and then I built wachy:

export CC_aarch64_linux_android=$HOME/Android/toolchains/arm64/bin/aarch64-linux-android30-clang
export AR_aarch64_linux_android=$HOME/Android/toolchains/arm64/bin/aarch64-linux-android-ar
export CARGO_TARGET_AARCH64_LINUX_ANDROID_LINKER=$HOME/Android/toolchains/arm64/bin/aarch64-linux-android30-clang
cargo build --target aarch64-linux-android --release

I got the error:

  TARGET = Some("aarch64-linux-android")
  HOST = Some("x86_64-unknown-linux-gnu")
  CC_aarch64-linux-android = None
  CC_aarch64_linux_android = Some("/home/ripper/Android/toolchains/arm64/bin/aarch64-linux-android30-clang")
  CFLAGS_aarch64-linux-android = None
  CFLAGS_aarch64_linux_android = None
  TARGET_CFLAGS = None
  CFLAGS = None
  CRATE_CC_NO_DEFAULTS = None
  DEBUG = Some("false")

  --- stderr
  /home/ripper/wachy/target/aarch64-linux-android/release/build/ncurses-905dd36bad61d0d6/out/chtype_size.c:6:10: fatal error: 'ncurses.h' file not found
  #include <ncurses.h>
           ^~~~~~~~~~~
  1 error generated.
  thread 'main' panicked at 'assertion failed: command.status().expect(\"compilation failed\").success()', /home/ripper/.cargo/registry/src/github.com-1ecc6299db9ec823/ncurses-5.101.0/build.rs:105:5
  note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
warning: build failed, waiting for other jobs to finish...



I'm a newer to rust, don't know how to solve the dependency.

Scroll to the traced function in the initial view

For me the function traced is initially always one line out of sight as there are two header lines in the window:

Latency [╷ Frequency╷ [^]

─────────┴──────────┴─────

so scrolling one line more gets it in sight:

diff --git a/src/views.rs b/src/views.rs
index 5e60819..9d2bc77 100644
--- a/src/views.rs
+++ b/src/views.rs
@@ -182,10 +182,10 @@ pub fn set_source_view(
     }
     // Set this twice - once before to prevent out of bounds, second time to
     // ensure the table scrolls to the right place.
-    sview.set_selected_row(selected_line as usize - 1);
+    sview.set_selected_row(selected_line as usize - 2);
     sview.set_items(items);
     // TODO is second time necessary?
-    sview.set_selected_row(selected_line as usize - 1);
+    sview.set_selected_row(selected_line as usize - 2);
 }
 
 pub type FooterView = PaddedView<Layer<TextView>>;

/DLange

Offset outside the function bounds

Hey, great tool!

I'm seeing error messages like these whenever I try to trace a line with "x": (the initial tracepoint on main works)

Error: bpftrace command 'BEGIN { @start_time = nsecs; @depth[-1] = 0; @matched_retfilters[-1] = 0; } uprobe:/home/max/projects/uni/avatar/build/imgui_ros/devel/lib/imgui_ros/gui:main /@depth[tid] == 0/ { @start147[tid] = nsecs; @depth[tid] = 1; } uretprobe:/home/max/projects/uni/avatar/build/imgui_ros/devel/lib/imgui_ros/gui:main /@depth[tid] == 1/ { @duration_tmp147[tid] += (nsecs - @start147[tid]); $duration = @duration_tmp147[tid]; @count_tmp147[tid] += 1; delete(@start147[tid]); @depth[tid] = 0; if (@matched_retfilters[tid] == 0) { @duration171 += @duration_tmp171[tid]; @count171 += @count_tmp171[tid]; @duration147 += @duration_tmp147[tid]; @count147 += @count_tmp147[tid]; } delete(@duration_tmp171[tid]); delete(@count_tmp171[tid]); delete(@duration_tmp147[tid]); delete(@count_tmp147[tid]); delete(@matched_retfilters[tid]); } uprobe:/home/max/projects/uni/avatar/build/imgui_ros/devel/lib/imgui_ros/gui:main+1011 /@depth[tid] == 1/ { @start171[tid] = nsecs; } uprobe:/home/max/projects/uni/avatar/build/imgui_ros/devel/lib/imgui_ros/gui:main+1016 /@depth[tid] == 1 && @start171[tid]/ { @duration_tmp171[tid] += (nsecs - @start171[tid]); @count_tmp171[tid] += 1; delete(@start171[tid]); } interval:s:1 { printf("{\"time\": %d, \"lines\": {", (nsecs - @start_time) / 1000000000); printf("\"171\": [%lld, %lld], ", @duration171, @count171); printf("\"147\": [%lld, %lld]", @duration147, @count147); printf("}}\n"); }' failed, status: ExitStatus(ExitStatus(65280)), stderr:
ERROR: Offset outside the function bounds ('_ZZ4mainENKUlvE_clEv' size is 303)

Versions:

bpftrace v0.14.0-93-g625a
wachy 0.1.0-alpha.6

Do you have an idea what is going on here / what information I can provide?
The source code of the "gui" executable is here: https://github.com/xqms/imgui_ros. But I'll try to create a minimal example that triggers this problem.

Make search view provide more than 8 results (scrolling)

There is some confusion using the SEARCH_VIEW_HEIGHT variable to limit amount of the search results. Clearing that up allows a much more useful search view:

diff --git a/src/views.rs b/src/views.rs
index 7298cbd..2c6ee05 100644
--- a/src/views.rs
+++ b/src/views.rs
@@ -215,6 +215,7 @@ pub type SearchView = ResizedView<Dialog>;
 
 const SEARCH_VIEW_WIDTH: usize = 70;
 const SEARCH_VIEW_HEIGHT: usize = 8;
+const SEARCH_VIEW_MAX_COUNT: usize = 50;
 
 /// `title` must be unique (it is used in the name of the view). Parameters of
 /// `edit_search_fn` are search view name, search string, and (max) number of
@@ -253,11 +254,11 @@ where
             .min_width(SEARCH_VIEW_WIDTH - 2), // ScrollView adds 2 character border
     )
     .scroll_x(true)
-    .fixed_size((SEARCH_VIEW_WIDTH, 8));
+    .fixed_size((SEARCH_VIEW_WIDTH, SEARCH_VIEW_HEIGHT));
 
     let update_edit_view = move |siv: &mut Cursive, search: &str, _| {
         // TODO we should add more results and allow scrolling?
-        edit_search_fn(siv, &name, search, SEARCH_VIEW_HEIGHT);
+        edit_search_fn(siv, &name, search, SEARCH_VIEW_MAX_COUNT);
     };
     let edit_view = EditView::new()
         .filler(" ")

Greetings!
/DLange

Question: Will this work with languages that don't use the system stack pointer register?

This looks really cool! Sorry for the dumb question but I was wondering if this would work with compiled languages that don't make use of the native stack pointer register? In particular I'm looking at GHC haskell which afaiu compiles most everything into tail calls jmps, and doesn't actually use the native stack pointer register for haskell code. It does support dwarf debug symbols however.

Thanks for open-sourcing this!

bpftrace syntax generated gives error on Debian testing or sid

wachy --version
wachy 0.1.0-alpha.5

bpftrace --version
bpftrace v0.14.1

wachy generates a eBPF program containing @duration_tmp139[tid] += nsecs - @start139[tid]; which causes:

stdin:1:274-275: ERROR: syntax error, unexpected -, expecting }

(the character count of course varies)

This code works on earlier bpftrace versions, e.g. tested with v0.9.4.

Changing the offending code segment to @duration_tmp139[tid] += (nsecs - @start139[tid]); makes it work again with bpftrace v0.14.1 as found in Debian testing and sid at the moment.

/DLange (@debian.org)

Wachy panics trying to load Rust binary.

Context

toby@foo:~/src/wachy$ rustc -V
rustc 1.62.0 (a8314ef7d 2022-06-27)

toby@foo:~/src/wachy$ target/release/wachy -V
wachy 0.1.0-alpha.6

toby@foo:~/src/wachy$ bpftrace -V
bpftrace v0.14.0

toby@foo:+/src/wachy$ file ~/src/vector/target/release/vector
/home/toby/src/vector/target/release/vector: ELF 64-bit LSB pie executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, BuildID[sha1]=85a6094a204866a55d1ce1d43ef267036612ea0f, for GNU/Linux 3.2.0, with debug_info, not stripped

toby@foo:+/src/wachy$ WACHY_LOG=trace sudo -E target/release/wachy ~/src/vector/target/release/vector "vector::internal_telemetry::allocations::Producer<_,_,T>::write"
Error: Panic! [v0.1.0-alpha.6]
Cause: panicked at 'called `Option::unwrap()` on a `None` value', src/controller.rs:443:53

No matter whether or not I run as sudo, older versions of bpftrace (tried v0.11.x), it seems to throw this panic. It's a bit hard to grok what it's doing at that location so I didn't have any great immediate ideas for debugging further, given that it seems to find the symbol.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.