kiernanmcgowan / forerunner Goto Github PK
View Code? Open in Web Editor NEWForerunner is a distributed job queue framework.
Home Page: https://github.com/kiernanmcgowan/forerunner
License: MIT License
Forerunner is a distributed job queue framework.
Home Page: https://github.com/kiernanmcgowan/forerunner
License: MIT License
Some event hooks are being called before the store callbacks, which is in appropriate if the store is a single source of truth in an application
I think it should be forerunner's responsibility to track and log failures in a way that makes it trivial to restart the job at the point where it failed. In this vein it would also be good for forerunner to die or at least do something when a job of a certain type fails too many times in a row, or to have an option for it.
I think composed jobs could be improved if they were treated internally as separate jobs and handled the same way by forerunner. I'm not sure of the best way to handle optional passthroughs though. I was thinking maybe if the object passed to the callback included an extra field purge
or something with the names of the fields that shouldn't be passed. Then forerunner could delete those fields as well as the purge field before giving it to the next task. Do you have any ideas?
If the very first task in a composed job is a progress event the job will fail with the error:
TypeError: Object #<Object> has no method 'progress:assigned'
at Object.editor_upload (/averylongpath/node_modules/forerunner/lib/worker.js:113:48)
It would be really useful to have an API on the manager that could return information about all workers connected to that manager.
Misc information that would be useful:
There needs to be an option to not use a cumulative data object; that is for the input field on task 1 to get deleted before the data object gets passed to task2. Otherwise when the object gets parsed by socket.io it'll exceed its maxBuffer
size and the process will die with a websocket parser forced user kick: max buffer size reached
error. It'd also be nice when JSON.stringifying/doing anything that loads the string into memory, because that would make it easier to debug individual tasks.
it might also be advantageous for check for maxBuffer when sending job event messages (in particular I'm looking at lib/utils/connection.js
), and/or add resiliency so that when the websocket kills the connection or any other forerunner-based error it doesn't kill the entire job runner.
Need to create a event hook for an 'aggregate failure' which is called when a job is marked as failed, but AFTER the store calls back
When we leave our forerunner system on for days on end, the logs being recorded by the manager for heartbeats to each worker really add up.
It would be great to be able to silence them or to make them much less frequent (heartbeats should still happen, but successful heartbeats should be recorded less frequently). Maybe bundle all the successful heartbeats into summary logs every 30 seconds or every minute?
Ex.
"Last 30 heartbeats from X were successful."
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.