More than one reporter? about nerve HOT 5 CLOSED

Just marking my issue as a question

from nerve.

Short answer: just use both nerve and monit side by side.

Long answer:

There are two types of things here:

• health checks
• acting on the results of health checks

We have two separate components. Nerve only does health checks, and the only action it takes is publishing the results of those health checks. Synapse acts on the results of the health check to configure haproxy.

We have additional alerting capabilities at Airbnb which also consume the results of the health checks to generate alerts, like monit would do. However, we don't have any component that actively tries to restore health. This is because we're worried that actively trying to restore health would only cause more problems.

Ideally, if your code encounters bugs, it would fail fast. We use runit to run all of our services, so they would get automatically restarted. This is how we run nerve and synapse in prod as well.

However, if you are failing health checks because of failing upstream dependencies, restarting the service would not help and might cause harm as a starting service hammers your dependencies. This would argue against the use of monit for actively intervening in failing health checks.

from nerve.

Ok, forget about monit restarting my services. Point is: both monit and nerve will do same health checks. Monit will start my alarm routines and nerve will notify zk one node is out. Does it make sense to put both alarm reporting and service discovery reporting on nerve?

from nerve.

like i said, i think it is best to have a separate tool to do the alerting, which is linked with all of the rest of your systems alerting; we're planning on open-sourcing such a tool soon.

from nerve.

I think there are lots of options here, either what igor has mentioned or you can do what we do at Yelp and monitor the other end of the equation in Synapse (check that enough instances are actually in HAProxy).

Since this hasn't had any action for a few years I'm going to close this. Feel free to re-open if these answers are insufficient :-)

from nerve.