Git Product home page Git Product logo

blezer's Introduction

blezer

npm npm

Blezer is a simple background job/task processing queue for Node.js (>= 7.6) using cluster & separate Node processes, powered by Redis.

Features

  • each worker runs its tasks in a separate Node.js process
  • RESTful JSON API
  • integrated UI
  • logging per job/task
  • delay job/task execution
  • job/task expiry value for being in active state

Install

npm install -g blezer

Usage

Step 1: Create a task

Each job triggers an execution of a Task i.e. a recipe what to do for that job. It is defined as a class with perform method. Task corresponds to Worker from similar projects such as resque or sidekiq. It is named this way to avoid the clash with cluster workers.

const { Task } = require('blezer');  

class LoopTask extends Task {
  perform(args) {
    this.log(`Running inside: ${process.pid}`);
    this.log(`Args: ${args}`);

    let progress = 0;
    for (var i = 0; i < 1e10; i++) {
      if (i % 1e8 == 0) {
        this.log(i)
        this.progress(progress, 100);
        progress++;
      }
    }
  }
}

module.exports = LoopTask

Step 2: Run the server

Put your tasks in tasks/ directory and run

blezer start

By default, it checks available number of cores and it instantiates the number of Node processes accordingly. You can specify number of process by hand using -c option. Type blezer start --help to see all available options.

Step 3: Enqueue a job

You can enqueue a job to perform given task from a JavaScript application

const { enqueue } = require('blezer');

enqueue('LoopTask', '[1, 2, 3]');

By default, the enqueue function puts the new job on default queue; this can be changed with the name parameter from options.

enqueue('LoopTask', '[1, 2, 3]', { name: 'high' });

A job can be scheduled to run at a specific time using scheduledAt parameter.

enqueue('LoopTask', '[1, 2, 3]', { name: 'high', scheduledAt: Date.now() + Sugar.Number.days(4) });

It is also possible to enqueue a job through Blezer REST API

http POST :3000/api/enqueue task=LoopTask args='[1,2,3]'

(optional) Step 4: Check the progress via UI

Go to localhost:3000 to check the job proegress through Blezer UI.

Blezer UI

Blezer comes with a built-in web UI that allows to quickly see the status of all jobs. Here's a preview of what it looks like:

Blezer UI

Environments

You can distinguish visually the UI between staging and production environments by specifying BLEZER_ENV variable accordingly. You can set this variable when launching Blezer with blezer start e.g.

BLEZER_ENV=production blezer start

It will add a small color bar at the top to help you identify at a glance which UI instance you are currently using.


Concepts

Queues

Queue is a list of Job items, stored so as to be retrievable and handled in the order of insertion. You can create a Queue by giving it a name. This is a lower level API which usually shouldn't be used directly - it's advised to use enqueue helper.

const { Queue } = require('blezer');
const highQueue = new Queue('high');

Logging

You can log on per job/task basis by using this.log(message) method, where message is an object or a string.

this.log("This is my log message");
this.log({ a: 1, b: 2});

Create tasks from CLI

You can create a task in tasks using CLI

blezer create foo

This command will create FooTask.js task in tasks/ directory.

Roadmap

Blezer keeps track of the upcoming fixes and features on GitHub Projects: Blezer Roadmap

Bug reports

We use Github Issues for managing bug reports and feature requests. If you run into problems, please search the issues or submit a new one here: https://github.com/zaiste/blezer/issues

Detailed bug reports are always great; it's event better if you are able to include test cases.

Roadmap

Contributing

  • run the code through prettier

blezer's People

Contributors

2can avatar mzdunek93 avatar tomkuk avatar zaiste avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar

blezer's Issues

Rich logging for job/task

sprintf-like support:

.log('$%d sent to %s', amount, user.name);

Use util.inspect()

this.log({ key1: 'some data', key2: 11 });
this.log([1, 5, 4, 2, 1]);
this.log(3.14);

Problem with restarting process after killing a task

        | Error: ENOENT: no such file or directory, scandir '/PROJECT_PATH/node_modules/blezer/ui/tasks'
        |     at Object.fs.readdirSync (fs.js:914:18)
        |     at requireAll (/PROJECT_PATH/node_modules/require-all/index.js:32:18)
        |     at Object.<anonymous> (/home/zdun/apps/bloom/node_modules/blezer/lib/load.js:1:99)
        |     at Module._compile (module.js:571:32)
        |     at Object.Module._extensions..js (module.js:580:10)
        |     at Module.load (module.js:488:32)
        |     at tryModuleLoad (module.js:447:12)
        |     at Function.Module._load (module.js:439:3)
        |     at Module.require (module.js:498:17)
        |     at require (internal/module.js:20:19)
        |     at Object.<anonymous> (/home/zdun/apps/bloom/node_modules/blezer/lib/index.js:19:19)
        |     at Module._compile (module.js:571:32)
        |     at Object.Module._extensions..js (module.js:580:10)
        |     at Module.load (module.js:488:32)
        |     at tryModuleLoad (module.js:447:12)
        |     at Function.Module._load (module.js:439:3)

Allow functions beside classes

Long running jobs could be also defined as functions. It needs to be analyzed how to plug in .log and .progress helpers in that case.

function isClass(func) {
  return typeof func === 'function' 
    && /^class\s/.test(Function.prototype.toString.call(func));
}

Tasks loading fail silently

When loading the tasks directory (lib/load.js), any file with a syntax error will make blezer unable to instanciate any task.
The require-all should be wrapped in a try/catch and log any error to the console.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.