Git Product home page Git Product logo

sqs-consumer's Introduction

sqs-consumer

NPM downloads Build Status Code Climate Test Coverage

Build SQS-based applications without the boilerplate. Just define an async function that handles the SQS message processing.

Installation

npm install sqs-consumer --save

Usage

const { Consumer } = require('sqs-consumer');

const app = Consumer.create({
  queueUrl: 'https://sqs.eu-west-1.amazonaws.com/account-id/queue-name',
  handleMessage: async (message) => {
    // do some work with `message`
  }
});

app.on('error', (err) => {
  console.error(err.message);
});

app.on('processing_error', (err) => {
  console.error(err.message);
});

app.start();
  • The queue is polled continuously for messages using long polling.
  • Messages are deleted from the queue once the handler function has completed successfully.
  • Throwing an error (or returning a rejected promise) from the handler function will cause the message to be left on the queue. An SQS redrive policy can be used to move messages that cannot be processed to a dead letter queue.
  • By default messages are processed one at a time โ€“ a new message won't be received until the first one has been processed. To process messages in parallel, use the batchSize option detailed below.
  • By default, the default Node.js HTTP/HTTPS SQS agent creates a new TCP connection for every new request (AWS SQS documentation). To avoid the cost of establishing a new connection, you can reuse an existing connection by passing a new SQS instance with keepAlive: true.
const { Consumer } = require('sqs-consumer');
const AWS = require('aws-sdk');

const app = Consumer.create({
  queueUrl: 'https://sqs.eu-west-1.amazonaws.com/account-id/queue-name',
  handleMessage: async (message) => {
    // do some work with `message`
  },
  sqs: new AWS.SQS({
    httpOptions: {
      agent: new https.Agent({
        keepAlive: true
      })
    }
  })
});

app.on('error', (err) => {
  console.error(err.message);
});

app.on('processing_error', (err) => {
  console.error(err.message);
});

app.start();

Credentials

By default the consumer will look for AWS credentials in the places specified by the AWS SDK. The simplest option is to export your credentials as environment variables:

export AWS_SECRET_ACCESS_KEY=...
export AWS_ACCESS_KEY_ID=...

If you need to specify your credentials manually, you can use a pre-configured instance of the AWS SQS client:

const { Consumer } = require('sqs-consumer');
const AWS = require('aws-sdk');

AWS.config.update({
  region: 'eu-west-1',
  accessKeyId: '...',
  secretAccessKey: '...'
});

const app = Consumer.create({
  queueUrl: 'https://sqs.eu-west-1.amazonaws.com/account-id/queue-name',
  handleMessage: async (message) => {
    // ...
  },
  sqs: new AWS.SQS()
});

app.on('error', (err) => {
  console.error(err.message);
});

app.on('processing_error', (err) => {
  console.error(err.message);
});

app.on('timeout_error', (err) => {
 console.error(err.message);
});

app.start();

API

Consumer.create(options)

Creates a new SQS consumer.

Options

  • queueUrl - String - The SQS queue URL
  • region - String - The AWS region (default eu-west-1)
  • handleMessage - Function - An async function (or function that returns a Promise) to be called whenever a message is received. Receives an SQS message object as it's first argument.
  • handleMessageBatch - Function - An async function (or function that returns a Promise) to be called whenever a batch of messages is received. Similar to handleMessage but will receive the list of messages, not each message individually. If both are set, handleMessageBatch overrides handleMessage.
  • handleMessageTimeout - Number - Time in ms to wait for handleMessage to process a message before timing out. Emits timeout_error on timeout. By default, if handleMessage times out, the unprocessed message returns to the end of the queue.
  • attributeNames - Array - List of queue attributes to retrieve (i.e. ['All', 'ApproximateFirstReceiveTimestamp', 'ApproximateReceiveCount']).
  • messageAttributeNames - Array - List of message attributes to retrieve (i.e. ['name', 'address']).
  • batchSize - Number - The number of messages to request from SQS when polling (default 1). This cannot be higher than the AWS limit of 10.
  • visibilityTimeout - Number - The duration (in seconds) that the received messages are hidden from subsequent retrieve requests after being retrieved by a ReceiveMessage request.
  • heartbeatInterval - Number - The interval (in seconds) between requests to extend the message visibility timeout. On each heartbeat the visibility is extended by adding visibilityTimeout to the number of seconds since the start of the handler function. This value must less than visibilityTimeout.
  • terminateVisibilityTimeout - Boolean - If true, sets the message visibility timeout to 0 after a processing_error (defaults to false).
  • waitTimeSeconds - Number - The duration (in seconds) for which the call will wait for a message to arrive in the queue before returning.
  • authenticationErrorTimeout - Number - The duration (in milliseconds) to wait before retrying after an authentication error (defaults to 10000).
  • pollingWaitTimeMs - Number - The duration (in milliseconds) to wait before repolling the queue (defaults to 0).
  • sqs - Object - An optional AWS SQS object to use if you need to configure the client manually

consumer.start()

Start polling the queue for messages.

consumer.stop()

Stop polling the queue for messages.

consumer.isRunning

Returns the current polling state of the consumer: true if it is actively polling, false if it is not.

Events

Each consumer is an EventEmitter and emits the following events:

Event Params Description
error err, [message] Fired when an error occurs interacting with the queue. If the error correlates to a message, that error is included in Params
processing_error err, message Fired when an error occurs processing the message.
timeout_error err, message Fired when handleMessageTimeout is supplied as an option and if handleMessage times out.
message_received message Fired when a message is received.
message_processed message Fired when a message is successfully processed and removed from the queue.
response_processed None Fired after one batch of items (up to batchSize) has been successfully processed.
stopped None Fired when the consumer finally stops its work.
empty None Fired when the queue is empty (All messages have been consumed).

AWS IAM Permissions

Consumer will receive and delete messages from the SQS queue. Ensure sqs:ReceiveMessage and sqs:DeleteMessage access is granted on the queue being consumed.

Contributing

See contributing guidelines

sqs-consumer's People

Contributors

aklinkert avatar amilajack avatar anho avatar astrosam avatar cadz7 avatar deerawan avatar dependabot[bot] avatar dmitry-livchak avatar forbeslindesay avatar frsechet avatar greenkeeperio-bot avatar gregory avatar j-luong avatar jdecoster avatar jeanrauwers avatar jessegranger avatar mglagola avatar mrwillihog avatar nico-ulbricht avatar niklasr avatar nspragg avatar pablovilas avatar refactorr avatar rkt2spc avatar robertosousa1 avatar robinjmurphy avatar robtpaton avatar roniger avatar thales-gaddini avatar yoavmoon avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar

Forkers

isabella232

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.