tanelpoder / 0xtools Goto Github PK

View Code? Open in Web Editor NEW

567.0 33.0 89.0 5.84 MB

0x.Tools: Always-on Profiling for Production Systems

Home Page: https://0x.tools

License: GNU General Public License v2.0

Python 61.29% C 35.58% Makefile 0.46% Shell 2.67%

linux performance troubleshooting

0xtools's Introduction

0x.Tools: Always-on Profiling for Production Systems

0x.tools is a set of open-source utilities for analyzing application performance on Linux. It has a goal of deployment simplicity and minimal dependencies, to reduce friction of systematic troubleshooting. There’s no need to upgrade the OS, install kernel modules, heavy monitoring frameworks, Java agents or databases. These tools also work on over-decade-old Linux kernels, like version 2.6.18 from 14 years ago.

0x.tools allow you to measure individual thread level activity, like thread sleep states, currently executing system calls and kernel wait locations. Additionally, you can drill down into CPU usage of any thread or the system as a whole. You can be systematic in your troubleshooting - no need for guessing or clever metric-voodoo tricks with traditional system-level statistics.

An example of one of the tools psn is here:

$ sudo psn -p "mysqld|kwork" -G syscall,wchan

Linux Process Snapper v0.14 by Tanel Poder [https://0x.tools]
Sampling /proc/syscall, stat, wchan for 5 seconds... finished.


=== Active Threads ========================================================================================

 samples | avg_threads | comm          | state                  | syscall   | wchan                        
-----------------------------------------------------------------------------------------------------------
      25 |        3.12 | (mysqld)      | Disk (Uninterruptible) | fsync     | _xfs_log_force_lsn
      16 |        2.00 | (mysqld)      | Running (ON CPU)       | [running] | 0                            
      14 |        1.75 | (mysqld)      | Disk (Uninterruptible) | pwrite64  | call_rwsem_down_write_failed
       8 |        1.00 | (mysqld)      | Disk (Uninterruptible) | fsync     | submit_bio_wait              
       4 |        0.50 | (mysqld)      | Disk (Uninterruptible) | pread64   | io_schedule                  
       4 |        0.50 | (mysqld)      | Disk (Uninterruptible) | pwrite64  | io_schedule                  
       3 |        0.38 | (mysqld)      | Disk (Uninterruptible) | pread64   | 0                            
       3 |        0.38 | (mysqld)      | Running (ON CPU)       | [running] | io_schedule                  
       3 |        0.38 | (mysqld)      | Running (ON CPU)       | pread64   | 0                            
       2 |        0.25 | (mysqld)      | Disk (Uninterruptible) | [running] | 0                            
       1 |        0.12 | (kworker/*:*) | Running (ON CPU)       | read      | worker_thread                
       1 |        0.12 | (mysqld)      | Disk (Uninterruptible) | fsync     | io_schedule                  
       1 |        0.12 | (mysqld)      | Disk (Uninterruptible) | futex     | call_rwsem_down_write_failed 
       1 |        0.12 | (mysqld)      | Disk (Uninterruptible) | poll      | 0                            
       1 |        0.12 | (mysqld)      | Disk (Uninterruptible) | pwrite64  | _xfs_log_force_lsn           
       1 |        0.12 | (mysqld)      | Running (ON CPU)       | fsync     | submit_bio_wait              
       1 |        0.12 | (mysqld)      | Running (ON CPU)       | futex     | futex_wait_queue_me

Usage info and more details here:

https://0x.tools

Twitter:

https://twitter.com/0xtools

Author:

https://tanelpoder.com

0xtools's People

Contributors

Stargazers

Watchers

Forkers

oliveirafv biyiklioglu magmongoing goryszewskig kgtdbx kobe24shou javicane hishale ramujagini ktaranov samseongdong-hasu shabbirmdshaik hyee bakink praneethbalmuri lcdsantos1310 manoj21kothamasu bolugum jampaniuday eubide jonesadirimo ramdba143 cybernetics jkstill lefred mltdlvrs yago-123 thomasdale tcdale sjanulonoks mvandewouw aebm timurakhmadeev bsjerps karlarao xeronith hilkinr skunwar javikalsan dalai-srikant dubislv afu5956 dlotterman bharathm libing0526 crubbo yongganus thatvishalsingh bu3ny xcnix hungrybirder hotlib crazysz smudkey hiranp majidshabani kamusis ymhctt eternalerrors s905060 vitich daqingyun narendratreddi cljidkx techiewarrior benbkim icodein ygemici nizarakbarm c-hernandez0 cyberflamego wadia wenjian80 leonlee dbsid sysadreq cemberk xmas25 ojehle wfcadd light2cloud radtek parvez99 suchoudh jac0bwang huangchenzhao wanghenshui

0xtools's Issues

unistd_64.h not found

I am running this image in VirtualBox. Runs fine on another machine with VirtualBox but Ubuntu image is 4.15.0-47-generic there and not 46. Python versions are same on both but have not confirmed if same packages available on both.

Any ideas?

epost@arcshellvm:/media/sf_temp/arcshell/global/bin$ python --version
Python 2.7.15rc1

epost@arcshellvm:/media/sf_temp/arcshell/global/bin$ uname -a
Linux arcshellvm 4.15.0-46-generic #49-Ubuntu SMP Wed Feb 6 09:33:07 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

epost@arcshellvm:/media/sf_temp/arcshell/global/bin$ psn
Traceback (most recent call last):
  File "/media/sf_temp/arcshell/global/bin/psn", line 25, in <module>
    import proc, report
  File "/media/sf_temp/arcshell/global/bin/proc.py", line 276, in <module>
    syscall_id_to_name = get_system_call_names()
  File "/media/sf_temp/arcshell/global/bin/proc.py", line 273, in get_system_call_names
    raise 'unistd_64.h not found'
TypeError: exceptions must be old-style classes or derived from BaseException, not str

ubuntu 18 error while running make

make
gcc -I include -Wall -o bin/xcapture src/xcapture.c
src/xcapture.c: In function ‘main’:
src/xcapture.c:392:41: warning: ‘%s’ directive writing up to 255 bytes into a region of size 94 [-Wformat-overflow=]
sprintf(dirpath, "/proc/%s", pde->d_name);
^~
src/xcapture.c:392:17: note: ‘sprintf’ output between 7 and 262 bytes into a destination of size 100
sprintf(dirpath, "/proc/%s", pde->d_name);
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
src/xcapture.c:401:49: warning: ‘%s’ directive writing up to 255 bytes into a region of size 94 [-Wformat-overflow=]
sprintf(dirpath, "/proc/%s/task", pde->d_name);
^~
src/xcapture.c:401:25: note: ‘sprintf’ output between 12 and 267 bytes into a destination of size 100
sprintf(dirpath, "/proc/%s/task", pde->d_name);
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
root@sql3:/home/moustafa_ahmed/0xtools# lsb_release
No LSB modules are available.
root@sql3:/home/moustafa_ahmed/0xtools# cat /etc/issue
Ubuntu 18.04.5 LTS \n \l

Improve schedlat to report thread/task sched latency

Currently schedlat shows only the process (thread group leader, PID) latency.

Use /proc/PID/task/TID/schedstat instead, to allow reporting a single thread's latency

support wrapping/narrower output (especially when kstack is used)

for now the workaround is to run ./psn ... > output.txt and use the text editors left-right scrolling capabilities, if the output with kstack is too wide. Or use a terminal that can scroll horizontally too (the horizontal-scrolling terminal that I use is this: https://github.com/software-jessies-org/jessies/wiki/Terminator)

Not working on Fedora 35

Hi,

I'm getting the following error while trying to use any of the tools:

[root@fedora ~]# git clone https://github.com/tanelpoder/0xtools
Cloning into '0xtools'...
remote: Enumerating objects: 293, done.
remote: Counting objects: 100% (210/210), done.
remote: Compressing objects: 100% (125/125), done.
remote: Total 293 (delta 113), reused 157 (delta 73), pack-reused 83
Receiving objects: 100% (293/293), 133.78 KiB | 1.21 MiB/s, done.
Resolving deltas: 100% (154/154), done.

[root@fedora 0xtools]# make
gcc -I include -Wall -o bin/xcapture src/xcapture.c
[root@fedora 0xtools]# make install 
install -m 0755 bin/xcapture /usr/bin/xcapture
install -m 0755 bin/psn /usr/bin/psn
install -m 0755 bin/schedlat /usr/bin/schedlat
install -m 0755 -d /usr/lib/0xtools
install -m 0644 lib/0xtools/proc.py /usr/lib/0xtools/proc.py
install -m 0644 lib/0xtools/psnreport.py /usr/lib/0xtools/psnreport.py
install -m 0644 lib/0xtools/argparse.py /usr/lib/0xtools/argparse.py

[root@fedora ~]# sudo psn -p "mysqld|kwork" -G syscall,wchan
Traceback (most recent call last):
  File "/usr/bin/psn", line 113, in <module>
    args.sources = [s for s in proc.all_sources if s.name in args.sources.split(',')]
AttributeError: module 'proc' has no attribute 'all_sources'

[root@fedora ~]# uname -a
Linux fedora 5.15.12-200.fc35.x86_64 #1 SMP Wed Dec 29 15:03:38 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
[root@fedora ~]# which python
/usr/bin/python
[root@fedora ~]# python --version
Python 3.10.1

missing proc

hi, ive used this utility before, many times -its great -thanks

i downloaded the current version, getting this message - im sure it somthing stupid imdoing or not, but cant figure it out

/tmp/psnapper-master/bin# ./psn
Traceback (most recent call last):
File "./psn", line 31, in
import proc, psnreport
ImportError: No module named proc
root@linux4:/tmp/psnapper-master/bin#

KeyError getting syscall_id

$ sudo ./bin/psn -G syscall,wchan

Linux Process Snapper v1.1.0 by Tanel Poder [https://0x.tools]
Sampling /proc/syscall, wchan, stat for 5 seconds...
Traceback (most recent call last):
  File "./bin/psn", line 375, in <module>
    main()
  File "./bin/psn", line 301, in main
    task_samples = [s.sample(event_time, pid, task) for s in sources.keys() if s.task_level == True]
  File "/home/delijn/0xtools/lib/0xtools/proc.py", line 108, in sample
    return [create_row_sample(rs) for rs in raw_samples]
  File "/home/delijn/0xtools/lib/0xtools/proc.py", line 104, in create_row_sample
    r =  [event_time, pid, task] + [convert(full_sample[idx]) for idx, convert in self.schema_extract]
  File "/home/delijn/0xtools/lib/0xtools/proc.py", line 369, in <lambda>
    ('syscall',    str,  0, lambda sn: syscall_id_to_name[sn]),  # convert syscall_id via unistd_64.h into call name
KeyError: '1879048192'

Pstool snapshot

It's better to replace % to format

https://github.com/tanelpoder/psnapper/blob/1682a1acccf98448805b4ad08686c524d742488c/proc.py#L368

For example:
return ['{} {}'.format(depth, stack_line) for depth, stack_line in enumerate(fh.readlines()[::-1])]

Run process by psn itself

It would be great if you are will add opportunity for psn to run some processes itself.

cmdline

Hi Tanel,
Is the cmdline option supposed to include the full command line of a process? It looks like I am getting the process name (or path to it) instead of /proc/< pid >/cmdline

RHEL 7.9

unistd_64.h

psn
Traceback (most recent call last):
File "/usr/bin/psn", line 47, in
import proc, psnreport
File "/usr/lib/0xtools/proc.py", line 331, in
syscall_id_to_name = get_system_call_names()
File "/usr/lib/0xtools/proc.py", line 328, in get_system_call_names
raise Exception('unistd_64.h not found in' + ' or '.join(unistd_64_paths) + '.\n You may need to "yum install kernel-headers" or "apt-get install libc6-dev"\n until this dependency is removed in a newer pSnapper version')
Exception: unistd_64.h not found in/usr/include/asm/unistd_64.h or /usr/include/x86_64-linux-gnu/asm/unistd_64.h or /usr/include/asm-x86_64/unistd.h or /usr/lib/0xtools/syscall_64_5.15.0.h or /usr/lib/0xtools/syscall_64.h.
You may need to "yum install kernel-headers" or "apt-get install libc6-dev"
until this dependency is removed in a newer pSnapper version

I'm on 5.15.0-1033-azure #40~20.04.1-Ubuntu SMP Tue Jan 24 16:06:12 UTC 2023 aarch64 aarch64 aarch64 GNU/Linux

after installed libc6-dev, the problem is still happening

sudo apt-get install libc6-dev
Reading package lists... Done
Building dependency tree
Reading state information... Done
libc6-dev is already the newest version (2.31-0ubuntu9.9).
0 upgraded, 0 newly installed, 0 to remove and 7 not upgraded.

Enhancement: Addition of checksum to go with a release

This is a really useful suite of tools - watching the development with interest.

A useful enhancement (and a requirement for some users) is to have a checksum (MD5 or SHA) to use as an indicator that the release content is uncorrupted.

Here's an example of this from the utPLSQL project:

https://github.com/utPLSQL/utPLSQL/releases/tag/v3.1.10

Is this something that could be added to future releases please?

Thanks again,
Mat.

add command line option to sample processes of a user

Fix missing newlines when formatting python3

Currently python3 gives this (python2 has some newlines for readability):

$ sudo psn -r -p 111525 -a -G syscall,wchan
Linux Process Snapper v1.1.0 by Tanel Poder [https://0x.tools]
Sampling /proc/stat, syscall, wchan for 5 seconds...
finished.
=== Active Threads ===========================================================================
 samples | avg_threads | comm   | state                 | syscall         | wchan             
----------------------------------------------------------------------------------------------
     100 |        1.00 | (perl) | (zombie)              | [kernel_thread] | 0                 
     100 |        1.00 | (perl) | Sleep (Interruptible) | nanosleep       | hrtimer_nanosleep 
samples: 100
(expected: 100)
total processes: 2, threads: 2
runtime: 5.00, measure time: 0.13

Python2:

$ sudo psn -r -p 111525 -a -G syscall,wchan

Linux Process Snapper v1.1.0 by Tanel Poder [https://0x.tools]
Sampling /proc/stat, syscall, wchan for 5 seconds... finished.


=== Active Threads ===========================================================================

 samples | avg_threads | comm   | state                 | syscall         | wchan             
----------------------------------------------------------------------------------------------
     100 |        1.00 | (perl) | (zombie)              | [kernel_thread] | 0                 
     100 |        1.00 | (perl) | Sleep (Interruptible) | nanosleep       | hrtimer_nanosleep 


samples: 100 (expected: 100)
total processes: 2, threads: 2
runtime: 5.00, measure time: 0.13

KeyError on Rocky 9 when running psn

reproduce steps

yum install -y git make gcc python procps
git clone https://github.com/tanelpoder/0xtools
cd 0xtools/
make && make install

[root@tc-tikv-0 0xtools]# psn -p 1 -G syscall,wchan,filename
Linux Process Snapper v1.2.3 by Tanel Poder [https://0x.tools]
Sampling /proc/stat, syscall, wchan for 5 seconds...
Traceback (most recent call last):
  File "/usr/bin/psn", line 375, in <module>
    main()
  File "/usr/bin/psn", line 301, in main
    task_samples = [s.sample(event_time, pid, task) for s in sources.keys() if s.task_level == True]
  File "/usr/bin/psn", line 301, in <listcomp>
    task_samples = [s.sample(event_time, pid, task) for s in sources.keys() if s.task_level == True]
  File "/usr/lib/0xtools/psnproc.py", line 109, in sample
    return [create_row_sample(rs) for rs in raw_samples]
  File "/usr/lib/0xtools/psnproc.py", line 109, in <listcomp>
    return [create_row_sample(rs) for rs in raw_samples]
  File "/usr/lib/0xtools/psnproc.py", line 105, in create_row_sample
    r =  [event_time, pid, task] + [convert(full_sample[idx]) for idx, convert in self.schema_extract]
  File "/usr/lib/0xtools/psnproc.py", line 105, in <listcomp>
    r =  [event_time, pid, task] + [convert(full_sample[idx]) for idx, convert in self.schema_extract]
  File "/usr/lib/0xtools/psnproc.py", line 383, in <lambda>
    ('syscall',    str,  0, lambda sn: syscall_id_to_name[sn]),  # convert syscall_id via unistd_64.h into call name
KeyError: '45'

[root@tc-tikv-0 0xtools]# cat /etc/redhat-release 
Rocky Linux release 9.1 (Blue Onyx)

bundle unistd.h with psn

This way no need to install or copy this file even on minimal Linux installs that don't have it. it's not completely trivial as different platforms (and different major kernel versions apparently) the syscall number<->name mappings are different. So might be easy to just bundle x86_64's mappings for "recent" kernels - and still use the separate unistd.h file if it's there

Add tid field to psn

Using pid,tid for listing PID/thread ID is more consistent naming than pid,task.

Add tid into the output of psn --list too.

Bundle argparse.py with psnapper

This is needed for older python versions where argparse is not bundled by default.

Apparenlty it's tested even on python 2.3 so can be safely bundled for convenience.

https://pypi.org/project/argparse/

Modify Makefile to better accommodate cross compilation

Thanks for creating this valuable tool.

Can I suggest some minor changes to the Makefile that will make it much easier to cross compile your useful tool?

These will more easily allow people to specify an alternate compiler (CC) and installation destination (PREFIX)

--- Makefile.orig       2022-09-23 12:06:14.876630117 +1000
+++ Makefile    2022-09-23 12:09:03.834063588 +1000
@@ -1,8 +1,8 @@
-CC=gcc
+CC ?= gcc
 PREFIX ?= /usr

 # build
-CFLAGS=-I include -Wall
+CFLAGS ?= -Wall

 # debuginfo included
 CFLAGS_DEBUG=-I include -ggdb -Wall
@@ -11,7 +11,7 @@
 CFLAGS_DEBUG0=-I include -ggdb -O0

 all:
-       $(CC) $(CFLAGS) -o bin/xcapture src/xcapture.c
+       $(CC) $(CFLAGS) -I include -o bin/xcapture src/xcapture.c

 debug:
        $(CC) $(CFLAGS_DEBUG) -o bin/xcapture src/xcapture.c
@@ -20,6 +20,7 @@
        $(CC) $(CFLAGS_DEBUG0) -o bin/xcapture src/xcapture.c

 install:
+       install -m 0755 -d ${PREFIX}/bin
        install -m 0755 bin/xcapture ${PREFIX}/bin/xcapture
        install -m 0755 bin/psn ${PREFIX}/bin/psn
        install -m 0755 bin/schedlat ${PREFIX}/bin/schedlat

psn v1.2.3 release may show wrong syscall names on x86_64 due to looking them up from a wrong/template file

unistd.h files on ARM/aarch64 work somewhat differently from x86_64... Fixed in the repo already:

0b1d25c

Will release 1.2.4 with the fix soon.

add epoll_pwait() to the list of single-fd syscalls

so that psn would print out the filename used

tanelpoder / 0xtools Goto Github PK

0xtools's Introduction

0x.Tools: Always-on Profiling for Production Systems

0xtools's People

Contributors

Stargazers

Watchers

Forkers

0xtools's Issues

Recommend Projects

Recommend Topics

Recommend Org