Monitoring execution time of script about healthchecks HOT 15 CLOSED

Hello @audiolion, I'm happy to review and merge PR(s) implementing this feature.

I like your suggestion about doing this in stages: 1) add API support 2) document the API support 3) expose the measured times in UI 4) add configurable alerting based on measured times.

from healthchecks.

I've now added a /start endpoint. For now I'm treating it as an experimental feature and so I haven't updated the documentation yet.

Here's how it works for now. Let's say your ping URL is

https://hc-ping.com/bddfa063-d3ca-4fc6-8220-8ff6c712d4f7

Append "/start" at the end and that's the /start endpoint:

https://hc-ping.com/bddfa063-d3ca-4fc6-8220-8ff6c712d4f7/start

When the /start endpoint receives a request, a few things happen:

• Healthchecks logs a [Started] event. You will see it in the log, along with [OK] and [Failure] events
• In the web UI, check's status changes to "Started" ("▶" icon)
• It kicks off a timer: the check now must receive a ping within the grace time, or it will go down

In other words, if a job signals a "start", we expect it to then signal "success" within its grace time. If it does not, we send an alert.

There are still edge cases that I'm still thinking about how to handle best. I also have UI ideas on how to present the new functionality better. But this seems like a logical point to deploy the current implementation and give a status update.

If you're interested in this functionality and get a chance, you're welcome to try it out. If you notice any bugs, inconsistencies, or have other ideas on how to improve it, I'm all ears.

from healthchecks.

I think it's easier to send two requests (start and end).

I want to get an alert if execution time is longer than X

from healthchecks.

Released in v1.4.0

from healthchecks.

why not just time the execution of the command yourself?

from healthchecks.

For reference, cronitor.io provides this functionality and is very useful. For example, not all scripts or services properly return exit codes, which means chaining commands with && will run the healthchecks.io request regardless of the previous command's exit code, which is misleading, because a service or script that fails should return non-zero, so chained commands fail to execute, but not all do. In instances like this, knowing the average time a service or script runs will allow sending alerts based on a process that finished very fast, but did not provide a non-zero exit code signifying that it failed.

from healthchecks.

I would like this feature as well, and am willing to create a PR for it (likely multiple as this touches database, services, frontend, additional rulesets/validations) if the maintainers would accept it.

This will require a refactor of the service as we need to identify which routes we are monitoring, as stated earlier the simplest way to link up which cron routes we want to monitor start/end times for, we use the same hchks.io address hash, but have /start /end appended to the URL to signify we want to track the timing.

After that refactor we can update the frontend templates to include the elapsed time of each checks start/end and provide an average for that endpoint.

Finally we could implement validation rules for alerts when healthchecks start/end time is above/below a certain time or standard deviation or whatever. E.g. if it took 3 standard deviations longer to run sends a notification, or if it runs 3 standard deviations faster it sends notification, or if it takes more than 60 seconds send a notification, or if it takes less than .1 seconds send a notification.

from healthchecks.

cronalarm.com has this feature too
Between "start" and "end" pings the check is displayed in "running" state in the UI.
This is a great way to monitor what is running at the moment ...

from healthchecks.

Interested in this as well.

from healthchecks.

This feature would make choosing this project over cronitor/cronhub/deadmans snitch/etc a no brainer. Not having it is the reason I didn't just sign up at https://healthchecks.io

from healthchecks.

This would be an useful feature – agreed!
I'm planning on implementing it. It touches core parts of the system, so will take time to do properly.

For anybody reading this: are you mainly interested in just logging the execution time, or getting alerted when a job takes longer than expected to execute (even though it manages to finish before going "down")?

from healthchecks.

Logging the execution time is a must-have. Displaying in UI a job as running (ie between the start and end events) is also a great (and simple) feature.

However triggering an alert when getting higher than a threshold seems redundant to me. Periodic check + grace time is sufficient. If my cron job is not finished in time (grace time exhausted), i just want ONE alert that tell me "didnt run" or "didnt finish in time". I don't need another setting with running time, which will mean TWO redundant alerts.

from healthchecks.

I'm interested in being alerted when the job takes longer than expected. With the interval/grace only, you might never notice a job growing slowly in execution time over time.

from healthchecks.

in test for 1 week now, it is working great
thank you

from healthchecks.

Just started testing this and have a quick suggestions:

1. Regarding UI: don't use green (or at least not the same green as for the all-OK check-mark:
   
   Perhaps orange would be appropriate for a started timer?

2. Regarding syntax (or whatever category this falls under). I understand that the intended usage is that you start the timer with an appended /start and stop it with an ordinary ping (without any endpoint). It would be nice, though, if it would also accept a ping with a /stop endpoint for stopping the timer.

Why this redundancy? Not a big deal, but in my mind it just makes sense that a start command is followed by a stop (I intuitively appended a /stop before properly reading the instructions). But intuition and aesthetics aside, I think it also increases the readability of scripts if I can see that this is where the timer starts and this is where it stops.

Notably, though, while stopping a running timer should work both with and without /stop, a /stop should not be accepted as an ordinary ping (i.e. it will do nothing if it wasn't preceded by a /start)

from healthchecks.