Comments (21)
That should probably go under https://github.com/prometheus/procfs
from node_exporter.
@wrouesnel what is in the /proc/net/rpc file? Does not exist on my system.
from node_exporter.
It's NFS operation stats. Shows counters for requests.
from node_exporter.
@wrouesnel thx
from node_exporter.
Oh yeah I'd like that 👍
from node_exporter.
In addition to this, parsing /proc/self/mountstats
would allow you to get per-mountpoint stats with some useful data (operation rate, RPC backlog, retransmits, average round-trip and execution times, etc.) and counts of each unique operation. The catch is that the format wasn't designed to be easy to parse as can be seen by http://git.linux-nfs.org/?p=steved/nfs-utils.git;a=blob_plain;f=tools/mountstats/mountstats.py;hb=refs/heads/master
from node_exporter.
How much overlap is there between /proc/net/rpc/
and /proc/self/mountstats
? It seems like on atleast 3.2 the rpc directory has a number of stats, and they seem to be in a more machine-readable format.
from node_exporter.
@SuperQ It's trading hassle for granularity. /proc/net/rpc/nfs
reports global stats for the entire NFS client while /proc/self/mountstats
includes more details and breaks it down for each mount but the format wasn't designed to be easily parsed. I work in an environment with a bunch of different NFS servers and mounts so the ability to break the metrics down by mount would be really useful but I recognize that it's a lot more work.
Here's an example with a 4.6-series kernel:
/proc/net/rpc/nfs
:
net 0 0 0 0
rpc 105 0 105
proc3 22 0 6 0 86 3 0 2 0 0 0 0 0 0 0 0 0 0 3 0 2 1 0
proc4 59 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
/proc/self/mountstats
:
device sysfs mounted on /sys with fstype sysfs
device proc mounted on /proc with fstype proc
device devtmpfs mounted on /dev with fstype devtmpfs
…
device server.example.org:/path/to/export mounted on /mnt/test with fstype nfs statvers=1.1
opts: rw,vers=3,rsize=65536,wsize=65536,namlen=255,acregmin=3,acregmax=60,acdirmin=30,acdirmax=60,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=1.2.3.4,mountvers=3,mountport=4046,mountproto=udp,local_lock=none
age: 78
caps: caps=0x3fc7,wtmult=4096,dtsize=4096,bsize=0,namlen=255
sec: flavor=1,pseudoflavor=1
events: 6 306 0 0 7 86 395 0 0 2 0 14 2 0 4 0 0 6 0 0 0 0 0 0 0 0 0
bytes: 143698 0 0 0 23907 0 6 0
RPC iostats version: 1.0 p/v: 100003/3 (nfs)
xprt: tcp 812 1 1 0 65 105 105 0 105 0 2 0 0
per-op statistics
NULL: 0 0 0 0 0 0 0 0
GETATTR: 6 6 0 840 672 0 4 4
SETATTR: 0 0 0 0 0 0 0 0
LOOKUP: 86 86 0 13424 10120 0 75 78
ACCESS: 3 3 0 432 360 0 6 6
READLINK: 0 0 0 0 0 0 0 0
READ: 2 2 0 304 24164 0 18 18
WRITE: 0 0 0 0 0 0 0 0
CREATE: 0 0 0 0 0 0 0 0
MKDIR: 0 0 0 0 0 0 0 0
SYMLINK: 0 0 0 0 0 0 0 0
MKNOD: 0 0 0 0 0 0 0 0
REMOVE: 0 0 0 0 0 0 0 0
RMDIR: 0 0 0 0 0 0 0 0
RENAME: 0 0 0 0 0 0 0 0
LINK: 0 0 0 0 0 0 0 0
READDIR: 0 0 0 0 0 0 0 0
READDIRPLUS: 3 3 0 492 8028 0 12 12
FSSTAT: 0 0 0 0 0 0 0 0
FSINFO: 2 2 0 272 328 0 1 1
PATHCONF: 1 1 0 136 140 0 0 0
COMMIT: 0 0 0 0 0 0 0 0
from node_exporter.
Thanks for the summary. I agree, more hassle. I think the per-mount and other details are worth it. We already parse some fairly annoying outputs like /proc/mdstat
.
It would be nice to reference some kernel docs on what all of the fields mean.
from node_exporter.
It definitely seems to be under-documented. proc(5)
has the basic structure but there's not a ton of detail (e.g. http://linux.die.net/man/5/proc) so I've been referring to the mountstats.py file in http://git.linux-nfs.org/?p=steved/nfs-utils.git;a=blob;f=tools/mountstats/mountstats.py;hb=HEAD
from node_exporter.
It looks like most use of /proc/self/mountstats
is through the mountstats.py file in nfs-utils
, which is somewhat unfortunately hard to work with. I tossed together a quick proof of concept in a fork which modifies the mountstats Python code to make it easier to get structured values out and has a rudimentary hack to generate something plausibly compatible with textfile:
https://github.com/acdha/nfs-utils/compare/machine-readable-output
With that, I get something like this output:
https://gist.github.com/acdha/57825d3c99e5dd60acbfa7f71bc09c2c
Currently thinking that each operation should be a label rather than a separate metric but otherwise wondering what might be worth changing before porting it into Go.
from node_exporter.
Just coming back around to thinking about this one. Anyone volunteers to implement this parser in go? :-D
from node_exporter.
I'd be happy to take a shot at this.
from node_exporter.
To clarify, is this issue for tracking NFS client, server, or both statistics?
from node_exporter.
I think we are interested in adding /proc/self/mountstats
for tracking NFS client stats first.
from node_exporter.
Cool. I'll look into writing a general purpose mountstats
package, and once I've made some progress, I'll make a PR to vendor it in and read out some data for Prometheus metrics.
from node_exporter.
PR submitted: prometheus/procfs#30.
from node_exporter.
Now that the above has been merged, I notice there is another PR open for NFS stats: #360.
I'd be happy to work on a PR using the new procfs code, but I want to make sure I don't duplicate work.
from node_exporter.
@mdlayher I think having both the rpc
and the mountstats
view is a good idea.
from node_exporter.
Can this get closed, now that #360 and #367 are merged?
from node_exporter.
Yup, fixed!
from node_exporter.
Related Issues (20)
- openwrt : ramips/mt7621 run node_exporter failed HOT 3
- Missing file system metrics for /dev/shm HOT 4
- Exporter may freeze if network file system is unavailable HOT 4
- fatal error: concurrent map writes in github.com/prometheus/node_exporter v1.7.0 HOT 2
- New patch release HOT 2
- runit collector reports invalid (outdated) service state if runsv (supervisor process) is not running HOT 3
- thermal_zone collector stuck on Jetson Orin Nano HOT 1
- Bug: collector.arp.netlink is super slow to retrieve metrics HOT 2
- caller=stdlib.go caller='error encoding and sending metric family:write tcp 10.10.10.10:9100' msg=">192.168.3.12:47158:write:broken pipe" HOT 2
- Question about the procfs package
- NewNodeCollector all collectors disabled
- Node exporter have high memory usage in some nodes HOT 4
- node_exporter cannot parse drbd 8.4.11 stat file HOT 4
- several collectors fail on OpenBSD 7.5 HOT 1
- AWS ECS implementation
- Have you considered adding support for ebpf? HOT 3
- Qdisc collector does not expose queues with a parent
- PFSense node_exporter is not emitting `node_uname_info` metric HOT 2
- Question about diskIOSaturationThreshold
- Add Support for GPU Clock Frequencies (sclk and mclk) in hwmon Collector
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from node_exporter.