Git Product home page Git Product logo

activejob-google_cloud_pubsub's Introduction

ActiveJob::GoogleCloudPubsub

Build Status Gem Version

Google Cloud Pub/Sub adapter and worker for ActiveJob

Installation

gem 'activejob-google_cloud_pubsub'

Usage

First, change the ActiveJob backend.

Rails.application.config.active_job.queue_adapter = :google_cloud_pubsub

Write the Job class and code to use it.

class HelloJob < ApplicationJob
  def perform(name)
    puts "hello, #{name}!"
  end
end
class HelloController < ApplicationController
  def say
    HelloJob.perform_later params[:name]
  end
end

In order to test the worker in your local environment, it is a good idea to use the Pub/Sub emulator provided by gcloud command. Refer to this document for the installation procedure.

When the installation is completed, execute the following command to start up the worker.

$ gcloud beta emulators pubsub start

(Switch to another terminal)

$ eval `gcloud beta emulators pubsub env-init`
$ cd path/to/your-app
$ bundle exec activejob-google_cloud_pubsub-worker --project_id=dummy

If you hit the previous action, the job will be executed. (Both the emulator and the worker stop with Ctrl+C)

Configuration

Adapter

When passing options to the adapter, you need to create the object instead of a symbol.

Rails.application.config.active_job.queue_adapter = ActiveJob::GoogleCloudPubsub::Adapter.new(
  async:  false,
  logger: Rails.logger,

  pubsub: Google::Cloud::Pubsub.new(
    project_id:  'MY-PROJECT-ID',
    credentials: 'path/to/keyfile.json'
  )
)

async

Whether to publish messages asynchronously.

Default: true

logger

The logger that outputs a message publishing error. Specify nil to disable logging.

Default: Logger.new($stdout)

pubsub

The instance of Google::Cloud::Pubsub::Project. Please see Google::Cloud::Pubsub.new for details.

Default: Google::Cloud::Pubsub.new

Worker

The following command line flags can be specified. All flags are optional.

--require=PATH

The path of the file to load before the worker starts up.

Default: ./config/environment

--queue=NAME

The name of the queue the worker handles.

Note: One worker can handle only one queue. If you use multiple queues, you need to launch multiple worker processes.

Default: default

--min_threads=N

Minimum number of worker threads.

Default: 0

--max_threads=N

Maximum number of worker threads.

Default: number of logical cores

--project_id=PROJECT_ID, --credentials=KEYFILE_PATH

Credentials of Google Cloud Platform. Please see the document for details.

Development

$ bundle exec rake spec

Contributing

Bug reports and pull requests are welcome on GitHub at https://github.com/ursm/activejob-google_cloud_pubsub.

License

The gem is available as open source under the terms of the MIT License.

activejob-google_cloud_pubsub's People

Contributors

tricknotes avatar ursm avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar

activejob-google_cloud_pubsub's Issues

message.acknowledge! early

In the spirit of how Rails handles exceptions in ActiveJobs I wonder if it might better to acknowledge a message right before the actual processing starts. Example:

message.acknowledge!
ActiveJob::Base.execute JSON.parse(message.data)

https://github.com/ursm/activejob-google_cloud_pubsub/blob/master/lib/activejob-google_cloud_pubsub/worker.rb#L70-L105

That way it is up to the user what to do with exceptions using retry_on and discard_on hooks.

After all I believe acknowledge should mean delivered, not processed.

Incompatible with latest version of google-cloud-pubsub

Hi,
There is a small incompatibility that causes the activejob-google_cloud_pubsub-worker daemon to fail when using with google-cloud-pubsub 0.27.0. Line 10 in pubsub_extension.rb (

) initializes a topic with the autocreate: true argument. That argument was removed in the latest release:
https://github.com/GoogleCloudPlatform/google-cloud-ruby/blob/c4ada6f89efd7239bdd78305038b43d046f1f53b/google-cloud-pubsub/CHANGELOG.md

I'm happy to issue a pr if you would like.

Undefined method 'enqueue' for Active job

Hello,

Firstly, thank you for this gem. I'm hoping to get some good use out of it.

I am attempting to use this gem in my app as a drop-in replacement for redis/sidekiq which is running background mail tasks.

Following the readme, I've setup the gem, and deployed the app. The application functions normally until a request to use the gem is fired. Specifically, I'm using devise to handle delivery of mail with the deliver_later

Code:

  # Override Devise notification to use background messaging queue.
  def send_devise_notification(notification, *args)
    devise_mailer.send(notification, self, *args).deliver_later
  end

Running this line of code yields in the following:

NoMethodError (undefined method `enqueue' for ActiveJob::GoogleCloudPubsub::Adapter:Class):

Looking through the gem source, I can see that the method does indeed exist, however at runtime it appears it's not available. Are there any additional setup configs which are perhaps assumed or is there something I've missed which could cause this?

I'm running Ruby 2.5.3 and Rails 4.2.1

Thank you for any help. ๐Ÿ˜„

OAuth error when starting rails server in development

I get

/gems/signet-0.8.1/lib/signet/oauth_2/client.rb:988:in `fetch_access_token': Authorization failed.  Server message: (Signet::AuthorizationError)
{
 "error": "invalid_grant",
 "error_description": "Bad Request"
}

when trying to start rails server while running emulator in development. I also get the following:

$ bundle exec activejob-google_cloud_pubsub-worker --project_id=dummy
Note: Google::Cloud::Logging is disabled because it failed to authorize with the service. (Authorization failed.  Server message:
{
 "error": "invalid_grant",
 "error_description": "Bad Request"
}) Falling back to the default Rails logger.
Note: Google::Cloud::Debugger is disabled because it failed to authorize with the service. (Authorization failed.  Server message:
{
 "error": "invalid_grant",
 "error_description": "Bad Request"
})
Note: Google::Cloud::ErrorReporting is disabled because it failed to authorize with the service. (Authorization failed.  Server message:
{
 "error": "invalid_grant",
 "error_description": "Bad Request"
})
Note: Google::Cloud::Trace is disabled because it failed to authorize with the service. (Authorization failed.  Server message:
{
 "error": "invalid_grant",
 "error_description": "Bad Request"
})

I guess I missed a step related to auth, however when I read the official docs they say no credentials etc necessary when using app engine. I also assumed the emulator wouldn't need auth? What am I missing?

Too many thread pools

google-cloud-pubsub is starting a thread pool to process incoming messages:

            @stream_pool.map do |stream|
              Thread.new { stream.start }
            end

activejob-google_cloud_pubsub is doing the same:

pool = Concurrent::ThreadPoolExecutor.new(min_threads: @min_threads, max_threads: @max_threads, max_queue: -1)

... effectively duplicating the efforts of google-cloud-pubsub, if I'm not mistaken?

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.