In internal ref https://secure.helpscout.net/conversation/2175262282/30980?folderId=6397333 it was mentioned it would be helpful to have the functionality to allow creating of parent directory and subdirectory from the check command, for instance:

--state-directory /var/test1/test2

Where the /var/test1 directory does not yet exist, this will currently error out.

This is intented to be a required argument/envvar without it being required default expression is "" and matches anything.. definitely not the intended default behavior.

User stories:

• Alert when the count (or total) of regex matches exceeds a threshold during a time period (e.g. if more than 10 aggregate log matches in 15 mins).

• Alert when the count (or total) of regex matches is below a threshold during a time period (e.g. if 10 aggregate log matches are expected in 30 mins and only 5 are observed).

Can we achieve this via a more robust state file (or state db via boltdb)?

Minimum requirements:

• On positive match, transmit event to agent events API
• Configurable state file support that informs the current byte offset
• -max-bytes flag
• Packaging

Inverse matching boolean logic is unuseful.

Inverse should map to logic that warns/critical "less than" instead of "greater than" count match

Whenever we execute the sensu-check-log, it throws an error. Here's the image for reference
error_screenshot

Additionally, Here are the few item you would require to replicate the issue

1. Log file : sample.log
2. Check_Command: sensu-check-log --log-file /var/tmp/sample.log --match-expr "java.lang.OutOfMemoryError" --state-directory /var/tmp/ --reset-state --critical-threshold 1 --critical-only --warning-threshold 0 --ignore-initial-run --verbose
3. Check_config: check.yml.txt

From customer report

Test setup

1. set up first log file with matching lines

echo "error" > /tmp/test-logs/first-test.log

2. setup second log file without matching lines

echo "success" > /tmp/test-logs/first-test.log

2. run dry run command test using -p and -e option to make sure both files are found

./sensu-check-log -d /tmp/test-state-directory -p /tmp/test-logs -e "log$" -m "error" --disable-event-generation --dry-run ; echo $?

3. return status should be non-zero but it is not

When deploying this check against existing log files, the first run may report any number of matches which could be quite old. This is resulting in user requests to suppress alerts for any matches found on the first run of the plugin.

Because this plugin uses a state file to keep track of the position reached in last execution, it seems to me that we can trivially add a flag which suppresses alerts when there's no pre-existing state file.

In internal ref https://secure.helpscout.net/conversation/1825973406/28629?folderId=5845954, it was mentioned that there is a feature in Tivoli's log check that allows users to filter out logs based on the modification time. The user provided the following help text to illustrate the features that the Tivoli log check provides.

FileComparisonMode
Specifies which log files are monitored when more than one matches a wildcard pattern. The
following values are available:
CompareByAllMatches
This value is the default behavior. All files that match the wildcard pattern that is
specified in LogSources are monitored.
CompareByLastUpdate
Of the files that match the wildcard pattern that is specified in LogSources, the file with
the most recent last update timestamp is monitored.
CompareBySize
Of the two or more files that match the file name pattern criteria, the bigger file is
selected for monitoring. Do not use CompareBySize with multiple matching files that are
being updated at the same time and increasing their file sizes. If the largest file is subject
to frequent change, monitoring might continually restart at the beginning of the newly
selected file. Instead, use CompareBySize when there is a set of matching files, but only
one is active and being updated at any specific time.
CompareByCreationTime
Of the files that match the wildcard

In internal ref https://secure.helpscout.net/conversation/2016338466/29933?folderId=4465878, the customer points out the lack of error handling when stating a non-existent state directory leading to a much more difficult time when troubleshooting. The error output is the generic help text/syntax error pasted below.

sensu-check-log [flags] sensu-check-log [command] Available Commands: help Help about any command version Print the version number of this plugin Flags: -a, --analyzer-procs int Number of parallel analyzer processes per file. (default 4) -t, --check-name-template string Check name to use in generated events (default "{{ .Check.Name }}-alert") -C, --critical-only Only issue critical status if matches are found -c, --critical-threshold int Minimum match count that results in an warning (default 5) -D, --disable-event-generation Disable event generation, send results to stdout instead. -n, --dry-run Suppress generation of events and report intended actions instead. (implies verbose) -u, --events-api-url string Agent Events API URL. (default http://127.0.0.1:4041/events) -h, --help help for sensu-check-log -I, --ignore-initial-run Suppresses alerts for any matches found

Add support for extracting actual metrics reported in a log file, or generating/calculating metrics based on some trend or "pattern" in a log file (e.g. "N errors in the last 10 seconds"). The latter capability more or less already exists via the -match flag, but it would need to produce event.metrics as a result.

As a result of internal ref https://secure.helpscout.net/conversation/1974013858/29641?folderId=5845954, we're investigating an issue where it appears that the check plugin is generating events via the agent API that never have their event.Check.Executed updated. This is resulting in confusion from the end-user perspective.

Feature request from customer needs design discussion.

Desired use case explanation

Plugin now supports processing multiple files using a filename regexp pattern, and a state directory to hold per file path information, and will generate a single event capturing the result information.

The request is to generate an unique event for each processed log file path.

The technical problem

each generated event needs to have an unique check name, but file paths as check names might not be suitable, as file paths can be arbitrarily long and have special characters that can not be used in the check names currently.

Any attempt to squash the file path into an uuid makes the check name potentially unreadable to a human who needs to parse the event. Any attempt to shorten the filepath into a reasonable length check name runs the risk of destroying check name uniqueness causes events to overwrite each other.

The situation

The good

matching is regexp based, having the matching string in the output can be very useful to tune the regexp when receive false positive

The bad

matching string degrades readability of the json output because it can be a long unbounded string.

The ugly

because this check plugin uses regexp to match both file names and log line strings, just providing a simple summary of error counts is operationally insufficient to be actionable. The output must at a minimum show which files matched.

Straw solution

1. remove matching string from output by default
2. add cmdline line argument to opt-in to matching string

This plugin currently raises an alert when the specified regexp is matched.

We understand from some customers that they have a use for the inverse case, meaning they want to be alerted when specified patterns are not present in the log.

It would be helpful to be able to specify a regex path/multiple files to look at, specifically for applications that write logs to dated folders. For example:

C:\Program Files\Application\logs\YYYY-MM-DD\log.txt

Being able to present the check configuration as the following would be nice.

sensu-check-log -log "C:\Program Files\Application\logs\*\*.txt" -match ERROR -state "C:\etc\sensu\log_state\Application"

This was previously accomplished with a modified version of the log plugin for Sensu Core, but given the current Ruby runtime does not support Windows that guy no longer functions.

The env var here: https://github.com/sensu/sensu-check-log/blob/main/main.go#L176

Should be CHECK_LOG_MISSING_OK not CHECK_LOG_MISING_OK.

dry-run output report is confusing for operators.
Need to rework the output to clear up confusion on expected event status ok,warn,critical

Presently symlinks don't work for --log-path args, but it would be 💯 if they did

Style guide is published here: https://github.com/sensu-plugins/community/blob/master/PLUGIN_STYLEGUIDE.md

Specifically, this readme would be more useful with an example check configuration. Here's one where I've captured sensu-backend logs into /tmp and I'm checking them for error:

type: CheckConfig
api_version: core/v2
metadata:
  name: tmp-backend-log
spec:
  check_hooks: null
  command: sensu-check-log -match error -state /tmp/backend.log.state -log /tmp/backend.log
  interval: 10
  publish: true
  runtime_assets:
  - sensu-check-log
  stdin: true
  subscriptions:
  - backend
  timeout: 10

It would be my preference to include an example which will be more immediately applicable to a wide cross section of users, but in most cases the standard log files on a Linux system are not readable by non-root users.