syslog input filtering not working [podman] about pastash HOT 11 CLOSED

I absolutely understand and as I mentioned earlier, I'm sure this isn't a bug.

Saw the issue closed and thought no to bother with examples. XD

I really appreciate you helping me in this and I will gather some more log message input examples ( before reaching PaStash ) and some example output log messages ( after PaStash) as soon as possible.

Please excuse my late replies due to being extremely busy.

from pastash.

Please shoot the examples anytime you want :) this is what we're here for, and welcome to our community!

from pastash.

Hello again @lmangani, apologies for reaching back after a long while, the reason was that your comment about my logs being "RFC 3164" gave me the idea to fix this format for logs before them reaching PaStash and so I did, and guess what everything worked out of the box afterward.

I just want to thank you for your help and this amazing tool.
Lots of love your way, regards, Sina.

from pastash.

Thanks for sharing an update and for the kind words, much appreciated! Let us know how the adventure proceeds in our Matrix channel and we'll gladly help you optimise things as they grow and scale.

from pastash.

Hello @Skipper0707 and thanks for opening this issue and happy to provide assistance.
Could you show me some of the input log data to understand how the filter might be operating?

from pastash.

Hello @lmangani and thanks for your quick response.

The following are some sample log messages before reaching PaStash ( they're all syslog messages ):

Mar 28 12:56:05 node21 stork-server[3100662]: isc.org/stork/server/apps/(*StatsPuller).getStatsFromApp
Mar 28 12:56:05 node21 stork-server[3100662]: #011/tmp/build/backend/server/apps/kea/statspuller.go:334
Mar 28 12:56:05 node21 stork-server[3100662]: isc.org/stork/server/apps/(*StatsPuller).pullStats
Mar 28 12:56:05 node21 stork-server[3100662]: #011/tmp/build/backend/server/apps/kea/statspuller.go
Mar 28 12:56:05 node21 stork-server[3100662]: isc.org/stork/util.(*PeriodicExecutor).executorLoop
Mar 28 12:56:05 node21 stork-server[3100662]: #011/tmp/build/backend/util/periodicexecutor.go
Mar 28 12:56:06 server17 sshd[1991444]: Failed password for root from <SOME IP> port <SOME PORT> ssh2
Mar 28 12:56:06 server17 sshd[1991442]: Failed password for root from <SOME IP> port <SOME PORT> ssh2
Mar 28 12:56:06 server17 sshd[2015779]: Failed password for root from <SOME IP> port <SOME PORT> ssh2
Mar 28 12:56:04 pod-node23 systemd[1115]: Started podman.scope.
Mar 28 12:56:04 pod-node23 systemd[1115]: podman-.scope: Succeeded.

As a matter of fact, I just realized when you select a label on Grafana Explorer (Log browser), it shows its value, for example when you select host it shows there are several values like node-1, node-2... but, when you select the value no logs are available or when you query {syslog_program="sshd"} a drop-down menu shows all the available programs but again when you fill in the value, nothing happens.

when you select the log messages on Grafana all of them have the same following labels to them:

Just that and nothing else. I suppose there is a special type of filtering that I should be using but I don't know what.

from pastash.

Your logs look like rfc3164 formatted or something. Try the following filter instead:

filter {
  regex {
    regex => "([A-Z][a-z][a-z]\s{1,2}\d{1,2}\s\d{2}[:]\d{2}[:]\d{2})\s([\w][\w\d\.@-]*)\s(.*)\[(.*)\]:\s(.*)$"
    fields => "timestamp,host,syslog_program,syslog_pid,message",
    numerical_fields => "syslog_pid",
    date_format => "MMM DD HH:mm:ss Z"
  }
}

and you should see something similar to the following:

Mar 28 12:56:05 node21 stork-server[3100662]: isc.org/stork/server/apps/(*StatsPuller).getStatsFromApp

[STDOUT] {
  "message": "isc.org/stork/server/apps/(*StatsPuller).getStatsFromApp",
  "host": "node21",
  "@timestamp": "2022-03-28T12:56:05.000+0200",
  "@version": "1",
  "syslog_program": "stork-server",
  "syslog_pid": 3100662
}

from pastash.

This seems to work so I'm closing the issue.
Feel free to continue the discussion, or join our new #cloki matrix channel: https://matrix.to/#/#cloki:matrix.org

from pastash.

I regret to say it actually didn't work, I tried adding and playing around with the solution you gave me and that's why it took a while for me to respond, sorry for that.

Although as if I've understood correctly, whatever the issue is I have to find the right filtering plugin or in case of regex, the right regex to make it work and that is already a very great guidance, however, I'd be delighted if you reopened the issue and kept guiding me in this.

Regards.

from pastash.

I've closed it since its not a bug, but I'm more than happy to help further. You probably have more data in your line such as syslog_priority which was not included in the examples?

EDIT: always attach an example because "it didn't work" is technically impossible to work with ;)

from pastash.

I sure will, thanks again.

from pastash.