Git Product home page Git Product logo

adonis-kue's Introduction

Adonis Kue Provider

A Kue provider for the Adonis framework.

This library provides an easy way to get started with an asynchronous job queue for AdonisJS.

Install

npm install --save adonis-kue

Configure

Register the kue provider in start/app.js:

const providers = [
  ...
  'adonis-kue/providers/KueProvider'
]

Register the commands provider in start/app.js:

const aceProviders = [
  ...
  'adonis-kue/providers/CommandsProvider'
]

Register the jobs in start/app.js:

const jobs = [
  ...
  'App/Jobs/Example'
]

And then export the jobs array:

module.exports = { providers, aceProviders, aliases, commands, jobs }

Add a configuration file in config/kue.js. For example:

'use strict'

const Env = use('Env')

module.exports = {
  // redis connection
  connection: Env.get('KUE_CONNECTION', 'kue')
}

Usage

Command List

Command Description
adonis kue:listen Starting the listener
adonis make:job Make a new Job (Queue)

Starting the listener

Starting an instance of the kue listener is easy with the included ace command. Simply run ./ace kue:listen or adonis kue:listen.

Creating your first job

They expose the following properties:

Name Required Type Static Description
concurrency false number true The number of concurrent jobs the handler will accept
key true string true A unique key for this job
handle true function false A function that is called for this job.

Here's an example.

Dispatching jobs

Now that your job listener is running and ready to do some asynchronous work, you can start dispatching jobs.

const kue = use('Kue')
const Job = use('App/Jobs/Example')
const data = { test: 'data' } // Data to be passed to job handle
const priority = 'normal' // Priority of job, can be low, normal, medium, high or critical
const attempts = 1 // Number of times to attempt job if it fails
const remove = true // Should jobs be automatically removed on completion
const jobFn = job => { // Function to be run on the job before it is saved
  job.backoff()
}
const job = kue.dispatch(Job.key, data, { priority, attempts, remove, jobFn })

// If you want to wait on the result, you can do this
const result = await job.result

Thanks

Special thanks to the creator(s) of AdonisJS for creating such a great framework.

adonis-kue's People

Contributors

ahmadarif avatar enniel avatar nrempel avatar simontabor avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar

adonis-kue's Issues

How to fix emitter.setMaxListeners() ?

How to set this configuration?

MaxListenersExceededWarning: Possible EventEmitter memory leak detected. 11 job ttl exceeded listeners added. Use emitter.setMaxListeners() to increase limit

Kue not running

Hi,
I have done the setup as in docs, but when i run kue:listen i get info: kue worker listening for 0 job(s)
in terminal, and nothing happens.

Enqueue Job from Adonis Command

I'm trying to enqueue a Kue Job from a Command, but command doesn't terminate and it keeps running.

Do I have to do something inside the command to close the process from the job?

[Discussion] adonis-queue vs adonis-kue: Whats the difference?

@nrempel I have taken a look at both your implementations of adonis-queue and adonis-kue. Can I ask why you switched to bull over kue in adonis-queue? Do you you feel one library does a better job than the other?

I have just integrated adonis-kue into my application, and at the same time extended the API. I was going to create a PR to incorporate my additions to the library and then I came across adonis-queue.

I can see that the API for adonis-queue is very similar to that of its kue counterpart. Would it be wise for me to create a PR for both libraries, since my changes can extend both packages? Or are you only going to be maintaining one of these packages?

Start listening on main adonis start script

Thank you for building this and introducing me to Kue! This could be very useful for a project I'm working on.

Is it possible to start the kue listener as part of the main server.js script? I can edit server.js and spawn a child process in there but this doesn't feel the most elegant solution.

kue-dashboard

Is there a possibility to view and manage the jobs with kue-dashboard?

Jobs generator

Will the command be made to create the jobs template?

The done callback and the job object should be passed to the job handler

Currently, only the job payload is passed to the handle() method of a job class, I think it could be better to inject the job object and the done() callback too in order to have the same abilities as the default kue API (cf queue.process() method).
You probably want to provide an higher level API in the job classes, but sometime you need to use job.progress() or job.log() inside the job handler, or use the done() callback to return job specific errors, or complete the job programmatically.

Does adonis-kue support different queue names?

Hi,

Is it possible to create different queues? I want to have several workers processing different queues with different kind of jobs.

I think in kue it is possible passing an option object with name property to the createQueue function.

Thank you

How do I try again?

Even when it falls into catch, it does not retry. I'd like you to do it, how's it going?

'use strict'

const Mail = use('Mail')
const Env = use('Env')

class SendEmail {
  // If this getter isn't provided, it will default to 1.
  // Increase this number to increase processing concurrency.
  static get concurrency () {
    return 1
  }

  // This is required. This is a unique key used to identify this job.
  static get key () {
    return 'send-email'
  }

  // This is where the work is done.
  async handle (data) {
    try {
      const content = Object.assign(data, { appUrl: Env.get('APP_SITE') })
      const test = await Mail.send(data.template, content, (message) => {
        message.from('[email protected]')
        message.to(data.email)
        message.subject(data.subject)
      })
    } catch (e) {
     // He does not try when he falls here.
      console.log(e)
    }
  }
}

module.exports = SendEmail

Dispatch:

  try {
    // GET EMAIL, NAME OF USER
    const userInstance = await User.findOrFail(tokenInstance.user_id)

    const subject = tokenInstance => tokenInstance.type === 'activate' ? 'Ativar sua conta' : 'Alteração de senha'
    
    const data = {
      content: {
        token: tokenInstance,
        user: userInstance
      },
      template: 'emails.token',
      subject: subject(tokenInstance),
      email: userInstance.email
    }
    const priority = 'medium' // Priority of job, can be low, normal, medium, high or critical
    const attempts = 5 // Number of times to attempt job if it fails
    const remove = true // Should jobs be automatically removed on completion
    const job = kue.dispatch(Job.key, data, priority, attempts, remove)
  } catch (e) {
    console.log(e)
  }

RuntimeException: E_MISSING_CONFIG

I'm a bit confused to set up adonis-kue.
I followed step by step the readme documentation, and even then I have had problems getting it to work. When I go to dispatch a job, it returns the following error.

2018-02-22T20:19:31.792Z - warning: 
  WARNING: Adonis has detected an unhandled promise rejection, which may
  cause undesired behavior in production.
  To stop this warning, use catch() on promises and wrap await
  calls inside try/catch.

RuntimeException: E_MISSING_CONFIG: redis://localhost:6379 is not defined inside config/redis.js file
> More details: https://err.sh/adonisjs/errors/E_MISSING_CONFIG
    at Function.missingConfig (/home/daniel/dev/vacinadigital/api/monolith/node_modules/@adonisjs/redis/node_modules/@adonisjs/generic-exceptions/src/RuntimeException.js:38:12)
    at Proxy.connection (/home/daniel/dev/vacinadigital/api/monolith/node_modules/@adonisjs/redis/src/Redis/index.js:92:33)
    at createClientFactory (/home/daniel/dev/vacinadigital/api/monolith/node_modules/adonis-kue/src/Kue/index.js:30:39)
    at Object.exports.createClient (/home/daniel/dev/vacinadigital/api/monolith/node_modules/kue/lib/redis.js:62:31)
    at new Queue (/home/daniel/dev/vacinadigital/api/monolith/node_modules/kue/lib/kue.js:99:52)
    at Function.exports.createQueue (/home/daniel/dev/vacinadigital/api/monolith/node_modules/kue/lib/kue.js:70:23)
    at Kue.get instance [as instance] (/home/daniel/dev/vacinadigital/api/monolith/node_modules/adonis-kue/src/Kue/index.js:44:28)
    at Kue.dispatch (/home/daniel/dev/vacinadigital/api/monolith/node_modules/adonis-kue/src/Kue/index.js:63:22)
    at Object.TokenHook.addTokenQueue (/home/daniel/dev/vacinadigital/api/monolith/app/Models/Hooks/TokenHook.js:18:19)
    at Hooks.exec (/home/daniel/dev/vacinadigital/api/monolith/node_modules/@adonisjs/lucid/src/Lucid/Hooks/index.js:166:13)
    at Proxy._insert (/home/daniel/dev/vacinadigital/api/monolith/node_modules/@adonisjs/lucid/src/Lucid/Model/index.js:695:41)
    at <anonymous>

Now I will show my settings:

# kue
KUE_CONNECTION=redis://localhost:6379

# redis
REDIS_CONNECTION=local

config/redis.js

'use strict'

/*
|--------------------------------------------------------------------------
| Redis Configuaration
|--------------------------------------------------------------------------
|
| Here we define the configuration for redis server. A single application
| can make use of multiple redis connections using the redis provider.
|
*/

const Env = use('Env')

module.exports = {
  /*
  |--------------------------------------------------------------------------
  | connection
  |--------------------------------------------------------------------------
  |
  | Redis connection to be used by default.
  |
  */
  connection: Env.get('REDIS_CONNECTION', 'local'),

  /*
  |--------------------------------------------------------------------------
  | local connection config
  |--------------------------------------------------------------------------
  |
  | Configuration for a named connection.
  |
  */
  local: {
    host: '127.0.0.1',
    port: 6379,
    password: null,
    db: 0,
    keyPrefix: ''
  },

  /*
  |--------------------------------------------------------------------------
  | cluster config
  |--------------------------------------------------------------------------
  |
  | Below is the configuration for the redis cluster.
  |
  */
  cluster: {
    clusters: [{
      host: '127.0.0.1',
      port: 6379,
      password: null,
      db: 0
    },
    {
      host: '127.0.0.1',
      port: 6380,
      password: null,
      db: 0
    }]
  }
}

config/kue.js

'use strict'

const Env = use('Env')

module.exports = {
  // redis connection
  connection: Env.get('KUE_CONNECTION', 'kue')
}

Something that passed in my eyes?

How to access Job object

Hi,

In kue, you can do:

kue.Job.get(11, function(err, job){
    console.log(job.type)
})

But when you are using adonis-kue provider the Job object is not exported so you have to require kue directly in order to get it. I think it would be a good idea to export it too in the provider to avoid to require kue directly.

How to mark job as failure on promise call at dispatch handle?

  // This is where the work is done.
  async handle(data) {
    await Mail.connection('mailgun')
      .raw(
        `<a href="${data.latestChapterLink}" target="_blank">${
          data.latestChapterLink
        }</a>`,
        message => {
          message.subject(data.mailSubject)
          message.text(data.latestChapterLink)
          message.from(`NovelNotifier<[email protected]>`)
          message.to(data.userEmail)
        }
      )
      .catch(err => {
        // Mark job as fail and log err so I can view it at failed jobs in kue-dashboard
      })
  }

what is KUE_CONNECTION

I am not understanding the set up. New to queue. Could you please tell me KUE_CONNECTION in kue.js file? I created this kue.js file. what should I pass there.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.