Git Product home page Git Product logo

filestat_exporter's Introduction

File statistics exporter

CircleCI Docker Pulls GitHub All Releases Go Report Card

Prometheus exporter gathering metrics about file size, modification time and other statistics.

Quickstart

Pre-built binaries are available on the GitHub release page.

Usage

Configure target files on command line, passing glob patterns in parameters

./filestat_exporter '*'

Optional flags:

  • -config.file <yaml>: The path to the configuration file (use "none" to disable).
  • -log.level <level>: Logging level [debug, info, warn, error]. (default: info)
  • -version: Print the version of the exporter and exit.
  • -web.listen-address <port>: Address to listen on for web interface and telemetry. (default: 9943)
  • -web.telemetry-path <URL path>: Path under which to expose metrics. (default: /metrics)
  • -web.config <web file>: Path to config yaml file that can enable TLS or authentication.
  • -web.systemd-socket: Flag to use systemd socket activation listeners instead of port listeners (Linux only).
  • -path.cwd <path>: Change working directory of path pattern collection.
  • -metric.crc32: Generate CRC32 hash metric of files.
  • -metric.nb_lines: Generate line number metric of files.

The exporter can read a config file in yaml format (filestat.yaml by default).

exporter:
  # Optional network parameters
  listen_address: ':9943'
  #metrics_path: /metrics
  
  # Optional working directory - overridden by parameter '-path.cwd'
  working_directory: "/path/to/my/project"
  # Default enable/disable of metrics - overridden if not set by parameter '-metric.*'
  enable_crc32_metric: true
  # enable_nb_line_metric: false
  # list of patterns to apply - metrics can be enable/disabled for each group
  files:
    - patterns: ["*.html","assets/*.css","scripts/*.js"]
    - patterns: ["data/*.csv"]
      enable_nb_line_metric: true
    - patterns: ["archives/*.tar.gz"]
      enable_crc32_metric: false
      enable_nb_line_metric: false

Note: if a file is matched by a pattern more than once, only the first match's config is used

Glob pattern format

Glob pattern uses the implementation of bmatcuk/doublestar project:

  • Doublestar (**) can be used to match directories recursively. It should appear surrounded by path separators such as /**/.
  • Usual Glob syntax is still supported.

Exported Metrics

Metric Description Labels
file_glob_match_number Number of files matching pattern pattern
file_stat_size_bytes Size of file in bytes path
file_stat_modif_time_seconds Last modification time of file in epoch time path
file_content_hash_crc32 (*) CRC32 hash of file content path
file_content_line_number (*) Number of lines in file path

Note: metrics with (*) are only provided if configured

Building and running

Prerequisites:

  • Go compiler - currently version v1.13
  • Linux with make installed
  • Essential build environment for dependencies

Building

go get github.com/michael-doubez/filestat_exporter
cd ${GOPATH-$HOME/go}/src/github.com/michael-doubez/filestat_exporter
make
./filestat_exporter <flags> <file glob patterns ...>

To see all available configuration flags:

./filestat_exporter -h

The Makefile provides several targets:

  • make check: Running checks and tests
  • make run: Run exporter from go
  • make version: Print current version
  • make build: Build exporter without checks

Cross compiled distribution

To build all distribustion packages

make dist

To build a specific os/architecture package

make dist-<os>-<archi>
make dist-linux-amd64
...

Using Docker

The filestat_exporter is designed to monitor files on the host system.

Try it out in minutes on Katakoda docker playground:

# create local file
docker container run --rm -d -v ~/my_files:/my_files --name my_files bash -c 'echo "Hello world" > /my_files/sample.txt'
# launch exporter watching the files
docker run -d -p 9943:9943 --name=filestats -v ~/my_files:/data mdoubez/filestat_exporter -path.cwd /data '*'
# see file metrics
curl -s docker:9943/metrics | grep file_

TLS and basic authentication

Filestat Exporter supports TLS and basic authentication. This enables better control of the various HTTP endpoints.

To use TLS and/or basic authentication, you need to pass a configuration file using the --web.config.file parameter. The format of the file is described in the exporter-toolkit repository.

License

Apache License 2.0, see LICENSE.

filestat_exporter's People

Contributors

crooks avatar dependabot[bot] avatar lucian-vanghele avatar michael-doubez avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar

filestat_exporter's Issues

Too many stats are cast to floats

crc32, unix time and other metrics should stay int so they can be compared, instead the output coes out as float. size in megabytes instead of bytes is less of an issue, but it would be nice to have it consistent.

Outdated docker hub version

Hi!

The docker hub image is still v0.3.5 whereas the v0.3.6 has already been released here.
Is there a possibility that you could update the docker hub image to v0.3.6 as well?

Windows 386 version

I was trying to build the dist-windows-386 but the package is removed immediatly and building just with dist only builds amd64 versions.

Sparse Files Monitoring

Hiya,

Appreciate the filestat exporter.

Would it be possible to add sparse file detection and its actual byte sizing?

Thanks
A

The 0.3.6 version starts to report an error

The 0.3.6 version starts to report an error

OS Version: CentOS Linux release 7.6.1810 (Core)

/usr/local/prometheus/filestat_exporter: /lib64/libc.so.6: version GLIBC_2.32' not found (required by /usr/local/prometheus/filestat_exporter) /usr/local/prometheus/filestat_exporter: /lib64/libc.so.6: version GLIBC_2.34' not found (required by /usr/local/prometheus/filestat_exporter)
error

Pattern to search for files in directory and its subdirectories is not possible

We want to be able to list all files matching a certain pattern in a directory or subdirectories of that directory. This is not possible in the current situation. Something like ** which works for Python globs would be a nice feature. For example when using "test/**/*.txt" as a pattern, this would look for all .txt files in the test directory and all its subdirectories.

Availability of the file to be checked.

Hello , encountered the following problem, there are 3 files test.txt here is my filestat.yaml

exporter:
  listen_address: ':9943'
  metrics_path: /metrics

  working_directory: "/app"
  # Default enable/disable of metrics - overridden if not set by parameter '-metric.*'
  enable_crc32_metric: true
  # enable_nb_line_metric: false
  # list of patterns to apply - metrics can be enable/disabled for each group
  files:
    - patterns: ["test1.txt","test2.txt","test23.txt"]
      enable_nb_line_metric: true
      enable_crc32_metric: true

The received metrics look like this and when changing the content of files hash change, but when deleting a file, for example test1.txt metrics do not change in any way. Also if you recreate the file test1.txt. the metrics will not change in any way.

HELP file_content_hash_crc32 CRC32 hash of file content using the IEEE polynomial
TYPE file_content_hash_crc32 gauge
file_content_hash_crc32{path="test1.txt"} 3.85946783e+09
file_content_hash_crc32{path="test2.txt"} 2.657284769e+09
file_content_hash_crc32{path="test23.txt"} 7.73236185e+08

I would be grateful to find a solution.

Monitoring of directories

Is there any way to monitor directories with this exporter??
I tried following patterns but it didn't work:

  • .
  • *
  • **
  • /**/

Pattern Order Affects File Counts

This is one of those edge case things that might not affect many instances. By the way this exporter is a lifesaver for a very specific problem I have been dealing with.

Problem: The physical order of the patterns in the config file seems to affect the count.

Example: yaml

...
  files:
  - patterns:
    -  "/main_dir/*"
    -  "/main_dir/Troubled_file*"
...

Result:
main_dir count will be accurate, but the specific file pattern within main_dir are grossly under counted. While ls -l /main_dir/Troubled_file* | wc -l would show around 500 of the specific files exporter would show ~0-1 sometimes 10.

Fix: yaml

...
  files:
  - patterns:
    -  "/main_dir/Troubled_file*"
    -  "/main_dir/*"
...

Result fix:
main_dir file count and specific file pattern within main_dir seem to be more accurate.

Probably not worth a code change, but maybe a note in the documentation.

File owner information

Hi,

Appreciate the filestat exporter.

Would it be possible to get owner of the file?

Thanks
Ravi

Monitornig of file backup

Is there a way to monitor file backup availabe for last 24 hrs.
Since i was trying with

time() - file_stat_modif_time_seconds{instance="server:63387"} < 86400
  • This expression will list the files which are avaliable in last 24 hours. But how can i alert which all files are not available in last 24 hrs.

Slash missing

It seems a final slash is missing on all the output:

filestat.yml:

exporter:
  #! Working directory of exporter
  working_directory: "/dir1/dir2/dir3/"

  #! Network parameters
  #listen_address: ':9943'
  #metrics_path: /metrics

  #! Uncomment one of the following to enable default config
  #enable_crc32_metric: true
  #enable_nb_line_metric: true

  # list of patterns to apply
  files:
    - patterns: ["*/*/.ERROR"]
    - patterns: ["*/*/*.ERROR"]
    - patterns: ["*/*/*/*.ERROR"]

Metric output:

file_glob_match_number{pattern="*/*/*.ERROR"} 294
file_glob_match_number{pattern="*/*/*/*.ERROR"} 36
file_glob_match_number{pattern="*/*/.ERROR"} 0
file_stat_size_bytes{path="dir/dir/dir20201208164827.xml.ERROR"} 171
file_stat_size_bytes{path="dir/dir/dir20201208165023.xml.ERROR"} 17

The files are actually located at:

/dir1/dir2/dir3/dir/dir/dir/20201208164827.xml.ERROR
/dir1/dir2/dir3/dir/dir/dir/20201208165023.xml.ERROR

Between the filename and the last directory a slash seems missing.
Is this expected, an issue with configuration, a bug or me not understanding something?

Thanks!

Make it possible to enable/disable specific metrics in configuration file

I'm using this exporter mostly to keep an eye on the number of files in a specific folder, and it works beautifully.

However, I don't need to have the file size of modified date to be recorded for every file that appears in a folder.

For folders that are used heavily this generates a huge amount of metrics generated that are never used and put a some strain on our Prometheus installation. It might be useful to have the ability to enable/disable specific metrics from being generated for use cases like this.

Command Substitution

Nice exporter!
Is it possible to have shell expansions (command substitution) in the pattern of the config file?
I have tried but not got it to work.
If it isn't possible today, could you implement it?

For example to catch a file with todays date:

/tmp/*_`date +%Y%m%d`.log

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.