add more info to RepeatedMsgReduction about rsyslog-doc HOT 3 OPEN

Real life problem example: http://www.fail2ban.org/wiki/index.php/Fail2ban:Community_Portal#Count_.22Last_message_repeated_N_times.22_correctly

from rsyslog-doc.

The solution to that is to avoid the use of RepeatedMsgReduction. What you are
seeing there is an attempt to work around the problems that it causes.

Unfortunantly, when you start getting logs delivered from multiple machines (so
that they are interleved), or out of order (very possible with batch processing
of logs) determining what the 'last message' was in order to treat 'message
repeated X times' as X of those messages, becomes hard or impossible.

This is why this now defaults to off in rsyslog, you just avoid this entire
problem

In addition, if you do turn it on, rsyslog appends the line to the 'message
repeated X times' message so that you actually can tell what the (beginning of
the) message was.

But the easiest thing to do is pretend that this monstrosity of a standard never
existed :-)

David Lang

On Mon, 5 May 2014, Thomas D. wrote:

Date: Mon, 05 May 2014 17:10:42 -0700
From: Thomas D. [email protected]
Reply-To: rsyslog/rsyslog-doc
<reply+i-27875432-f09b95c90f0673a0ff0de94208a53b30abd0825e-2208335@reply.g
ithub.com>
To: rsyslog/rsyslog-doc [email protected]
Cc: davidelang [email protected]
Subject: Re: [rsyslog-doc] add more info to RepeatedMsgReduction (#19)

Real life problem example: http://www.fail2ban.org/wiki/index.php/Fail2ban:Community_Portal#Count_.22Last_message_repeated_N_times.22_correctly

---

Reply to this email directly or view it on GitHub:
#19 (comment)

from rsyslog-doc.

I have updated the doc at least a bit, including some more of the fine
details. I also took the liberty to borrow some of your text, David ;)

https://github.com/rsyslog/rsyslog-doc/blob/b8df087851a40bfc177f55726ff9a31f45b113d5/source/configuration/global/options/rsconf1_repeatedmsgreduction.rst

Not perfect, but probably better. Hopefully someone will help to go through
all this legacy doc. We need to also merge this properly with the
new-style. I'll carry on on all this as good as I can :-)

Rainer

On Wed, Feb 19, 2014 at 2:44 PM, davidelang [email protected]:

http://www.rsyslog.com/doc/v8-devel/configuration/global/options/rsconf1_repeatedmsgreduction.html

I think this defaults to off, but the page doesn't make that clear.

This is really something that breaks a lot of log processing and so should
be discouraged

first, it would help to have a sample showing several logs, then the same
thing with this turned on. One thing is that I believe the behaviour
changed over versions. In very old versions the message logged was just
"last message repeated N times", but I believe around version 4 or so it
changed to include the message after this so you could tell what message
was repeated.

then there should be a blurb saying that while turning this on can save
some space in logs, most log analysis tools need to see the repeated
messages, they can't handle the "last message repeated" format. This is a
feature that worked decades ago when logs were small and reviewed by a
human, it fails badly on high volume logs processed by tools.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/19
.

from rsyslog-doc.