hayamiz / perfmonger Goto Github PK

License: GNU General Public License v3.0

Emacs Lisp 0.01% Ruby 13.79% Makefile 0.55% Go 16.26% CSS 0.04% HTML 1.55% JavaScript 66.21% Shell 1.35% Dockerfile 0.23%

performance-analysis cpu monitor-cpu iops storage performance-metrics

perfmonger's Introduction

PerfMonger

PerfMonger is a system performance monitor which enables high-resolution and holistic performance measurement with the programmer friendly interface.

High-resolution: sub-second level monitoring is possible!
Holistic performance measurement: monitoring CPU, Disk I/O, Network all at once.
Programmer friendly: PerfMonger speaks monitoring results in JSON format, which makes later performance analysis much easier (ex. jq).

CAUTION: PerfMonger is still in early stage, so there may be a drastic change in the future. Do not use it for critical jobs

Target platform

GNU/Linux
Mac OS X (experimental support)

How to installation

gem install perfmonger

You need gnuplot 4.6.0 or later build with cairo terminals for plotting measurement data with perfmonger plot command.

Build from source

You need Ruby 2.2 or later, and Go 1.8 or later to build perfmonger.

bundle
rake build

Getting started

Basic usage of PerfMonger is:

Run perfmonger record to record performance information logs
Run perfmonger play to show performance information logs in JSON format

perfmonger play repatedly prints records of system performance including CPU usages, disk usages, and network usages. One line includes only one record, so you can easily process output records with jq, or any scripting languages like Ruby, Python, and etc.

Pretty-printed structure of a record is as follows:

{"time": 1500043743.504, # timestamp in unix epoch time
  "cpu": {               # CPU usages
    "num_core": 2,         # the number of cores
    "all": {               # aggregated CPU usage (max = num_core * 100%)
      "usr": 50.0,
      "sys": 50.0,
      "idle": 100.0,
      ...
    },
    "cores": [             # Usage of each CPU core
      {
        "usr": 25.0,
        "sys": 25.0,
        "idle": 50.0,
        ...
      },
      {
        "usr": 25.0,
        "sys": 25.0,
        "idle": 50.0,
        ...
      }
    ]
  },
  "disk": {             # Disk usages
    "devices": ["sda"],   # List of disk devices
    "sda": {              # Usage of device 'sda'
      "riops": 10.0,        # The number of read I/O per second
      "wiops": 20.0,        # The number of write I/O per second
      "rkbyteps": 80.0,     # Read transfer rate in KB/s
      "wkbyteps": 160.0,    # Write transfer rate in KB/s
      ...
    }
    "total": {           # Aggregated usage of all devices
      "riops": 10.0,
      "wiops": 20.0,
      "rkbyteps": 80.0,
      "wkbyteps": 160.0,
      ...
    }
  }
}

Typical use cases

`perfmonger live`: live performance monitoring

$ perfmonger live

Monitor IO performance of /dev/sda for each 0.1 second

$ perfmonger record -i 0.1 -d sda

Monitor CPU usage for each 0.1 second

$ perfmonger record -i 0.1

Monitor CPU usage and IO performance of /dev/sda, sdb for each 0.1 second

$ perfmonger record -i 0.1 -d sda -d sdb

perfmonger's People

Contributors

Stargazers

Watchers

Forkers

matsuu lurdan limitusus

perfmonger's Issues

Command line option for displaying version number

I want something like this:

$ perfmonger --version
PerfMonger version 0.3.1 (20XX-XX-XX)

[record] Per-process CPU usage (and etc) recording

Progress bar for summary subcommand

Summary takes >100 seconds for large perfmonger log files.
Show progress bar for such files.

[plot] Stacked CPU core usage layout instead of current matrix layout

Gzipped output for record subcommand

streaming gzip output?
gzip just after finishing recording session?

--list-device option

[server] Display hostname

Display hostname on HTML5 realtime monitor.

Use a hostname given by command line argument, or try to obtain it automatically.

[record] Trigger option

perfmonger-record starts recording just after a trigger condition is fired.

trigger can be:

the first launch of the specified program
the first fork from the specified process
specified time

or something elese.

[summary] ASCII art for CPU usage

Use ANSI color or something like that.

[record] adaptive interval backoff

[plot] Clean up temporary files correctly

[summary] Use pager for terminal output

[server] CPU usage graph

[record] /proc/interrupts

[fingerprint] Speed up fingerprint subcommand

In a server with a lot of drives, fingerprint subcommand takes so long time with low CPU usage. There must exist some opportunities for speed up.

[server] Skip rendering of graphs that are not on screen

[server] No graph update without 'record' options

$ perfmonger server

This command should draw a CPU graph but no graph update because of load failure of /faucet

[plot] plot size option

Compact output in summary subcommand

[record] Timeout option

If timeout option is specified, perfmonger-record exits after the specified time has passed.

$ perfmonger record --interval 0.1 --timeout 10 -d sda
{"time": ..., "ioinfo": {... } ... }
(... 10 seconds ...)
== Summary ==
...

Resolve logical volume name

If /dev/mapper/VolGroup00-LogVol00 or /dev/VolGroup00/LogVol00 is passed to perfmonger record via -d option, convert it to corresponding device name dm-x.

This is done by following steps:

Build device name dictionary by reading /proc/partitions
Parse input and identify the path of logical volume
stat(3) the volume and get major/minor number of the volume
Look up the dictionary build in step 1. with major/minor number

lsblk
fdisk -l
LVM(vgscan, lvscan, pvscan)
hdparm?

x[i] ... i-th sample value of IOPS
t[i] ... beginning time of i-th sample
N ... # of samples

Currently avg. IOPS is calculated as a average value of sampled IOPS.

avg IOPS = (x[0] + x[1] + ... + x[N-1]) / N

But actually, avg. IOPS needs to be calculated by this way.

avg IOPS = (x[1] * (t[1] - t[0]) + x[2] * (t[2] - t[1]) + ...  + x[N-1] * (t[N-1] - t[N-2])) / (t[N-1] - t[0])

Check exit status of background perfmonger-recorder

perfmonger record --kill does not check the background perfmonger-recorder process has exited or not, and fail if the process does not exit correctly.

Also, perfmonger record --kill should fail if there is no background recorder process.

Gzipped data file support

record subcommand: gzipped output stream (maybe gzipped file as default for future version?)
play subcommand: handle gzipped and plain input stream
- file extension based approach (*.pgr is plain, *pgr.gz is gzipped)
- transparent file handling approach (check magic number first, then switch plain/gzip read)