Please add Watchdog handling to icinga2.service file about deb-icinga2 HOT 12 CLOSED

Relevant PR where the support was added:
Icinga/icinga2#5996

from deb-icinga2.

And now I wonder why Mr. @dnsmichi deleted the docs about this topic:
Icinga/icinga2@f2102a1?diff=split

@Crunsher added the docs here and @dnsmichi overwrote them? Why?

from deb-icinga2.

Scroll down before making false accusations :) The content was just moved to the troubleshooting docs where this fits way better than "getting started" confusing the user.

from deb-icinga2.

Sorry I found the other file later. Too much hectic truble in this place today.

from deb-icinga2.

IMHO settings like Watchdog= and Restart= should be set by the user.

He was an easy possibility to do so. Docs exist.

Is this enabled for any other daemon by default?

from deb-icinga2.

Well, we already added this in an override for our installs, so here comes my own IMHO point:

Icinga is a monitoring solution, and if it stops working without anyone noticing, then bad things can happen.

The age-old question: who watches the watcher?

Systemd offers to supervise the process in a way, that ensures that is is working and operational.

I see no reason to not make use of that.

Aside: We are using the last status update in the database to check if icinga_programstatus.status_update_time was too long ago, and then we send an SMS. And this happened more than a couple times in the past 2 years.

But, we had to write this and implement it. Now systemd offers this for free, why not make use of it?

from deb-icinga2.

You could at least add this in a commented out block in the service file, so people can decide to uncomment it. Maybe add a link to the docs. Because this troubleshooting file is maybe not easy to find.

from deb-icinga2.

Opinions? @dgoetz @dnsmichi @lippserd

from deb-icinga2.

Adding it to the docs would be better then making it a default in my opinion. While the restarting is nice it typically hides problems which should better be fixed.

The WatchdogSec= we should set because the value has to be compatible with Icinga 2's own logic for sending heartbeats with sd_notify, so the recommended value is twice the interval Icinga 2 is sending heartbeats. For Restart=, StartLimitInterval=, StartLimitBurst=, StartLimitAction= I would only add to the docs some recommended values and why you should set them (or not). Especially the StartLimit values are very difficult to set per default as the Interval may depend on size of the environment and the Action depends on availability requirements and reboot time of the system.

Also I would not provide an block to uncomment in the service file as user should better use systemd's override mechanism.

from deb-icinga2.

Quote from here: http://0pointer.de/blog/projects/watchdog.html

First of all, to make software watchdog-supervisable it needs to be patched to send out "I am alive" signals in regular intervals in its event loop. Patching this is relatively easy. First, a daemon needs to read the WATCHDOG_USEC= environment variable. If it is set, it will contain the watchdog interval in usec formatted as ASCII text string, as it is configured for the service. The daemon should then issue sd_notify("WATCHDOG=1") calls every half of that interval. A daemon patched this way should transparently support watchdog functionality by checking whether the environment variable is set and honouring the value it is set to.

So Icinga should follow the setting from the service file and adjust it s interval accordingly.

from deb-icinga2.

@lazyfrosch There's docs already for such a watchdog, I would just add it there. https://icinga.com/docs/icinga2/latest/doc/15-troubleshooting/#systemd-watchdog

from deb-icinga2.

I guess most followed me in saying that this is an optional feature. Similar like crash dumps are not enabled by default.

Docs can be improved, if you have a suggestion, please open a PR or issue for Icinga 2.

Thanks for your involvement in Icinga, we really appreciate feedback.

from deb-icinga2.