cenkalti / dalga Goto Github PK

It might be cool to have a GUI dashboard for Dalga.

If I get around to this personally, I'd be interested to try out some tech like alpinejs, htmx, tailwindcss, and other low-overhead stuff.

@cenkalti what do you think?

I realized that Dalga does not have a pause/resume functionality. Would you be open to adding this feature?

There are some specific semantics that I'm interested in for this. For example, if a job would run weekly and the next run is on the 2nd of the month, pausing on the 1st and resuming on the 3rd should skip that iteration, and the job would run again on the 9th.

This is different than a delay caused by downtime. For instance, if the next run is on the 2nd and the system went down on the 1st and came up on the 3rd, it would immediately process the job and reschedule for the 9th.

(From what I understand this is already how Dalga would handle downtime, I am just providing a counter-example to illustrate what I am looking for in a Pause/Resume functionality.)

Since I will be using Dalga from another Go service, I will end up writing a client package to communicate with Dalga over HTTP. Would you be interested in upstreaming it into this repo?

It must be set by CI on tagged commits.

Required for #26

It is not possible to import v2 code with current module setup.

I am interested in adding tests to Dalga to verify its behavior over an extended period of time: firing jobs at the right intervals, retry behavior, etc. I am also interested in testing the interaction of Dalga with another system, including a flow of scheduling jobs, receiving notifications, pausing, resuming, retrying, rescheduling, etc.

To facilitate this, I would like to add github.com/mixer/clock to the project, and use it as the basis for Dalga's awareness of the current time. This means moving some of the time logic out of the SQL and into Go code.

In normal circumstances, the Clock library would use the normal system time, but in unit tests the time can be advanced by a day or a week at a time to simulate the passage of time and ensure that Dalga fires events at the expected points.

This is something that I added to Kala, and tried to add to robfig/cron. I understand it's not a small change but I think it's important for peace of mind when scheduling mission-critical tasks.

Once the various open PRs land, the documentation should be updated.

@tooolbox I have made some changes after your PRs are merged. Do you have any objections about them? Or, do you want to make additional changes before releasing v3.0.0?

There are some tests that should be written regarding how Dalga behaves with multiple instances. From reading the code, I have no doubts that it works well in a multi-instance setup, but there's no such thing as too many tests 😄

• If an instance claims a job and then dies, another instance should pick it up and finish it.
• Given a pool of jobs scheduled around the same point and running for a roughly equivalent amount of time, the distribution across the instances should be fairly equal.
• Given a pool of one-off jobs scheduled around the same point, no job should run twice.

Hey there, was looking over Dalga and I like that it's straightforward, backed by MySQL, and supports a multi-instance deployment. However, it lacks two things for my use case:

1. Support for ISO8601 intervals rather than pure seconds.
2. Customizing retries such as the backoff/interval, max limit, etc.

As I said this is a very straightforward project and the source is quite readable. I more or less see what I would need to do to use an ISO8601 interval, and I can make a PR for it. I'm sure retry-customization is not much beyond that.

Before I make any PRs, I wanted to ask: are you open to contributions? Are you interested in having more features in this project?

As a comparison I did quite a bit of work over at Kala which is a similar project, but the way it is structured I don't see how to make it highly-available, which is why I am considering Dalga.

I typically run systems with MariaDB instead of MySQL. This usually works out fine because the majority of syntax works across both system types.

In running the Dalga tests, I did encounter a problem with the SKIP LOCKED clause in Front() and found that MariaDB does not yet support this syntax.

1. Are you open to normalizing the SQL in this project to work across both systems?
2. If the performance improvement of things such as SKIP LOCKED are vital to you, would you be open to some sort of config or command-line flag to enable MariaDB compatibility?

Otherwise, a job may seem stuck in running status forever if remote end does not behave well.

It would be good to have a way for Dalga POSTs to be idempotent, meaning they contain some sort of information that allows a receiver to know "this is the Nth iteration of this job" irrespective of retries.

This could be as simple as taking the value of next_sched as a unix timestamp and including it as header or query param in the POST request. If the receiver keeps records of jobs, and it receives a duplicate next_sched timestamp header, it can throw it away. The header doesn't have to be called next_sched, it could be iteration or something.