jondot / sneakers Goto Github PK

A fast background processing framework for Ruby and RabbitMQ

Home Page: https://github.com/jondot/sneakers

License: MIT License

Ruby 99.11% Shell 0.15% Dockerfile 0.39% Slim 0.35%

sneakers's Introduction

Sneakers

      __
  ,--'  >
  `=====

A high-performance RabbitMQ background processing framework for Ruby.

Sneakers is being used in production for both I/O and CPU intensive workloads, and have achieved the goals of high-performance and 0-maintenance, as designed.

Installation

Add this line to your application's Gemfile:

gem 'sneakers'

And then execute:

$ bundle

Or install it yourself as:

$ gem install sneakers

Documentation

A quick start guide is available in the section below.

Visit the wiki for more detailed documentation and GitHub releases for release notes.

A change log is also available.

Quick start

Set up a Gemfile

source 'https://rubygems.org'
gem 'sneakers'
gem 'json'
gem 'redis'

How do we add a worker? Firstly create a file and name it as boot.rb then create a worker named as Processor.

touch boot.rb

require 'sneakers'
require 'redis'
require 'json'

$redis = Redis.new

class Processor
  include Sneakers::Worker
  from_queue :logs

  def work(msg)
    err = JSON.parse(msg)
    if err["type"] == "error"
      $redis.incr "processor:#{err["error"]}"
    end

    ack!
  end
end

Let's test it out quickly from the command line:

$ sneakers work Processor --require boot.rb

We just told Sneakers to spawn a worker named Processor, but first --require a file that we dedicate to setting up environment, including workers and what-not.

If you go to your RabbitMQ admin now, you'll see a new queue named logs was created. Push a couple messages like below:

{
   "type": "error",
   "message": "HALP!",
   "error": "CODE001"
}

Publish a message with the bunny gem RabbitMQ client:

require 'bunny'

conn = Bunny.new
conn.start

ch = conn.create_channel
ch.default_exchange.publish({ type: 'error', message: 'HALP!', error: 'CODE001' }.to_json, routing_key: 'logs')

conn.close

And this is the output you should see at your terminal.

2013-10-11T19:26:36Z p-4718 t-ovqgyb31o DEBUG: [worker-logs:1:213mmy][#<Thread:0x007fae6b05cc58>][logs][{:prefetch=>10, :durable=>true, :ack=>true, :heartbeat_interval=>2, :exchange=>"sneakers"}] Working off: log log
2013-10-11T19:26:36Z p-4718 t-ovqgyrxu4 INFO: log log
2013-10-11T19:26:40Z p-4719 t-ovqgyb364 DEBUG: [worker-logs:1:h23iit][#<Thread:0x007fae6b05cd98>][logs][{:prefetch=>10, :durable=>true, :ack=>true, :heartbeat_interval=>2, :exchange=>"sneakers"}] Working off: log log
2013-10-11T19:26:40Z p-4719 t-ovqgyrx8g INFO: log log

We'll count errors and error types with Redis.

$ redis-cli monitor
1381520329.888581 [0 127.0.0.1:49182] "incr" "processor:CODE001"

We're basically done with the ceremonies and all is left is to do some real work.

Looking at metrics

Let's use the logging_metrics provider just for the sake of fun of seeing the metrics as they happen.

# boot.rb
require 'sneakers'
require 'redis'
require 'json'
require 'sneakers/metrics/logging_metrics'
Sneakers.configure(metrics: Sneakers::Metrics::LoggingMetrics.new)

# ... rest of code

Now push a message again and you'll see:

2013-10-11T19:44:37Z p-9219 t-oxh8owywg INFO: INC: work.Processor.started
2013-10-11T19:44:37Z p-9219 t-oxh8owywg INFO: TIME: work.Processor.time 0.00242
2013-10-11T19:44:37Z p-9219 t-oxh8owywg INFO: INC: work.Processor.handled.ack

Which increments started and handled.ack, and times the work unit.

From here, you can continue over to the Wiki

Docker

If you use Docker, there's some benefits to be had and you can use both docker and docker-compose with this project, in order to run tests, integration tests or a sample worker without setting up RabbitMQ or the environment needed locally on your development box.

To build a container run docker build . -t sneakers_sneakers
To run non-integration tests within a docker container, run docker run --rm sneakers_sneakers:latest
To run full integration tests within a docker topology including RabbitMQ, Redis (for integration worker) run scripts/local_integration, which will use docker-compose to orchestrate the topology and the sneakers Docker image to run the tests
To run a sample worker within Docker, try the TitleScraper example by running script/local_worker. This will use docker-compose as well. It will also help you get a feeling for how to run Sneakers in a Docker based production environment
Use Dockerfile.slim instead of Dockerfile for production docker builds. It generates a more compact image, while the "regular" Dockerfile generates a fatter image - yet faster to iterate when developing

Compatibility

Sneakers main branch: Ruby 3.0+
Sneakers 2.7.x and later (using Bunny 2.9): Ruby 2.2.x
Sneakers 1.1.x and later (using Bunny 2.x): Ruby 2.x
Sneakers 1.x.x and earlier: Ruby 1.9.x, 2.x

Contributing

Fork, implement, add tests, pull request, get my everlasting thanks and a respectable place here :).

Thanks:

To all Sneakers Contributors - you make this happen, thanks!

Copyright

sneakers's People

Contributors

Stargazers

Watchers

Forkers

bivek arielze wishdev frankdenbow sergei-matheson libin anbiniyar jay3126 bwstearns dizhu gilangmahardhika lnsoso ged maxigs torkale michaelklishin mistshi timgluz ooleem alex179 arjunmenon josephholsten jpemberthy konnichiwaruby vmware-archive rodrigosaito abevoelker shinobidevs jkogara tychobrailleur kyle-sge jaresty pb-pravin yesware pvijay127 stevenkaras etothepiipower zhipingtv simonmorley polkaspots danielfarrell arion vitalim baltazore alipio sodapopcan replaygaming trungpham adaptiv-io manavt skyairmj tankerwng jcalvert squarelover ridecharge prathamesh-sonpatki sebastianedwards warolv mboeh doughsay ronyv89 vcita overcastnetwork kensodev stephenreid hipyard dsandstrom tsagadar rud drewnichols ngrennan-inflection jsomara abezzub sheerun alucardpj cyoungberg dbyron0 alexanderbez jaronkk sibustephen salimane sidravic otobrglez pierre-pretorius meltwaterarchive sadjow mehditup duduribeiro alihalabyah herenow firhq reevoo michaelbaudino ricardobrazao alexriedler afonsotsukamoto groe xiangzhuyuan numidiasoft jedibatman

sneakers's Issues

JRuby support

Hi there,

Like the look of this project, definitely a fan of AMQP backed processing queues (used to love Minion) - and this looks really exciting.

Currently, this isn't usable on JRuby, as even with workers: 1 as a config option it still attempts to fork, resulting in:

NotImplementedError: fork is not available on this platform
                  fork at org/jruby/RubyProcess.java:1054

Have you thought at all about allowing a pure threads-only model for running on the JVM and possibly under Rubinius?

Support correlating multiple tasks into a batch

We often run into situations where multiple jobs are executed and we need to do something upon completion of all tasks. These could be multiple of the same task or mixed. Is this something that is supported or on the horizon?

Ability to define a cluster of RabbitMQ servers

We're looking at adding a couple of rabbitmq boxes as a cluster, not a priority for sneakers to automatically failover but it would be nice. Having a quick look at the source code looks like it would just be a case of adding something similar to what Sweatshop has for Carrot (https://github.com/famoseagle/sweatshop/blob/master/lib/message_queue/rabbit.rb) in to https://github.com/jondot/sneakers/blob/master/lib/sneakers/queue.rb

Thoughts?

sneakers-treadmill

There's a need for a system that would run/replay messages on sneakers, and verify that results have not regressed in terms of performance and correctness.

This will serve as a meta-issue for the various tasks required.

Technical goals:

There will be either a separate tool called sneakers-treadmill (pun intended) or a test-time utility in /spec.

Tests are divided to workload types, then, per test;
Load messages onto RabbitMQ
Configure sneakers with treadmill metrics handler
Spawn sneakers 4-worker cluster
Identify when queue is drained (RabbitMQ carrot admin gem?)
Flush out metrics from the treadmill metrics handler, and compare them to expected thresholds.
Output a textual log of metrics per version, along with a tree sha. This can be used in CI tools (but what CI tool is crazy enough to allow us to do all that per build?? :).

separate connections for sending vs. receiving message

See: http://bigwig.io/docs/message_throughput/

Can we have a separate connection for sending job, and a separate connection for receiving job?
Right now, we only one config called RABBITMQ_URL.
Perhaps, break it up into RABBITMQ_RX_URL, and RABBITMQ_TX_URL.

Test Mode

I can't seem to find a way to turn on a "Test Mode" of some sort where messages are just handled inline instead of being pushed onto rabbitmq and relying on that for delivery. Thoughts?

ack! method usage is confusing

ack! is just an alias to :ack as per https://github.com/jondot/sneakers/blob/master/lib/sneakers/worker.rb#L33

However, by Ruby conventions, I would expect bang method to do something rather than return something. Currently ack! call has to be placed at the very end of work method so :ack is returned from work. This is quite implicit assumption which is present in all examples, but I could not find anything in the documentation about it.

This confused me, as I was trying to log something after ack! call and messages was not acknowledged.

This issue can be fixed in two different ways:
a) changing implementation of ack! (and reject!, requeue!) to setting a variable instead of returning value
b) improving the docs

I think it's worth discussing.

Cannot excute foreman start

When i follow the pos of " How to: running a stand alone worker "

And i excute ' foreman start' then got errors:

/Users/fancy/.rvm/gems/ruby-2.0.0-p598@rails4/gems/foreman-0.77.0/lib/foreman/process.rb:54:in `spawn': Permission denied - bin/myworker (Errno::EACCES)
    from /Users/fancy/.rvm/gems/ruby-2.0.0-p598@rails4/gems/foreman-0.77.0/lib/foreman/process.rb:54:in `block in run'
    from /Users/fancy/.rvm/gems/ruby-2.0.0-p598@rails4/gems/foreman-0.77.0/lib/foreman/process.rb:53:in `chdir'

So I excute ' sudo foreman start' then got errors:

Users/fancy/.rvm/rubies/ruby-2.0.0-p598/lib/ruby/site_ruby/2.0.0/rubygems/dependency.rb:315:in `to_specs': Could not find 'foreman' (>= 0) among 13 total gem(s) (Gem::LoadError)
Checked in 'GEM_PATH=/Users/fancy/.gem/ruby/2.0.0:/Users/fancy/.rvm/rubies/ruby-2.0.0-p598/lib/ruby/gems/2.0.0', execute `gem env` for more information
    from /Users/fancy/.rvm/rubies/ruby-2.0.0-p598/lib/ruby/site_ruby/2.0.0/rubygems/dependency.rb:324:in `to_spec'
    from /Users/fancy/.rvm/rubies/ruby-2.0.0-p598/lib/ruby/site_ruby/2.0.0/rubygems/core_ext/kernel_gem.rb:58:in `gem'
    from /Users/fancy/.rvm/gems/ruby-2.0.0-p598@rails4/bin/foreman:22:in `<main>'
    from /Users/fancy/.rvm/gems/ruby-2.0.0-p598@rails4/bin/ruby_executable_hooks:15:in `eval'
    from /Users/fancy/.rvm/gems/ruby-2.0.0-p598@rails4/bin/ruby_executable_hooks:15:in `<main>'

evaluate `bunny_mock` in tests

Evaluate if there's a need for https://github.com/primedia/bunny_mock in dirty tests.

queue naming, Sneakers::Support::QueueName

We're using sneakers with a java backend and SaaS RabbitMQ provider. The java software doesn't have RACK_ENV, so we're having queue name mismatches in our staging and production environments.

I'm working on code that will make Sneakers::Support::QueueName "pluggable" via the configuration object, but will default to the current class. Does that feel right to you?

Single-threaded option

I don't know if this is a use-case that you want to support or not, but I thought it's worth noting down in case it is.

Use case here is that I need concurrency, but need to avoid threaded concurrency, since there are definitely threading issues in the code and libraries this app uses. Using forking and throwing more memory at it is doing fine for now.

I was thinking just setting { threads: 1 } in the Sneakers configuration would do the trick, but due to the design of Sneakers I slowly discovered this is not the case. It's quite tricky to explain, so please bear with me :)

Each Sneakers worker has its own Bunny instance, and therefore channel.
Bunny manages a thread pool per channel for running subscribe callbacks. This is 1 by default, but due to Sneakers having one for each worker class, this means a new worker thread pool is spawned for each worker.
Let's say MyClass is used by WorkerOne and WorkerTwo workers and uses some kind of global configuration that isn't thread-safe. It's now possible for a job to come in for both WorkerOne and WorkerTwo and trigger race-condition bugs, since each job is dispatched in a separate thread by Bunny.

Another small issue with this is the number of connections for Sneakers workers (n_threads_or_processes * n_worker_classes). If you're running your own RabbitMQ, this probably isn't an issue, but if you're on something like CloudAMQP this can get expensive quickly.

I haven't got permission (yet) from the boss to release any code, but the solution at the moment was:

Use ServerEngine to manage forking.
Have one Bunny::Channel per process (and therefore one connection and thread pool).
Instantiate each worker type and subscribe using block: false
Call work_pool.join to block until the Bunny work pool exits cleanly.
Do necessary cleanup before quitting (make sure jobs don't get killed halfway through).

Keen to hear your opinion on the above!

Problems with class loading in Rails production (like) environments

Hi,

there was a little issue with class loading (when processes/workers/threads are restarted i guess) that sometimes the workers did not load the rails environment properly.

Some errors like "circular dependency" or that methods on classes/objects are not found that should obviously be there.
It only happens in a environment where classes are cached (rails production or when otherwise activated).

The fix is to make sure the eager-loading the environment is manually triggered (which is deactivated by default for rake tasks).

Just add this line:

::Rails.application.eager_load!

Before doing anything else in the sneakers:run rake task and it works fine (did so far here).

Message Retry Handler Feedback

I was hoping to generate some discussion on the 'nicest' way to handle retries and errors in a Sneakers Handler.

I was playing around and came up with something like the following but it would require some changes to the way the worker handles acks/rejects and I wanted some feedback on it before I started pushing or making merge requests.

As you can see it would require the props and msg passed into the reject/error/timeout calls. It utilises a dead letter policy on the queue when called without requeue to push it into an exchange/queue with a ttl and a dead-letter-exchange setting of the originals sneakers exchange.

I'm also not sure about the best way to publish to an error exchange. I would have preferred a new error exchange to keep the routing_key the same instead of modifying the message but this also works.

Keen to get some feedback on it and the future direction.

Thanks for writing Sneakers!

module Sneakers
  module Handlers
    class Maxretry

      ###
      # Maxretry uses dead letter policies on Rabbitmq to requeue and retry messages after a timeout
      # 
      ###

      def initialize(channel)
        @channel = channel
      end

      def acknowledge(tag)
        @channel.acknowledge(tag, false)
      end

      def reject(tag, props, msg, requeue=false)
        # Check how many times it has been requeued
        if props[:headers].nil? or props[:headers]['x-death'].nil? or props[:headers]['x-death'].count < 5
          @channel.reject(tag, requeue)
        else
        # Retried more than the max times
          @exchange.publish({:msg => msg, :routing_key => rops[:headers]['routing-keys'][0]}.to_json, :routing_key => 'error')
          @channel.acknowledge(tag, false)
        end
      end

      def error(tag, props, msg, err)
        reject(tag, props, msg)
      end

      def timeout(tag, props, msg)
        reject(tag, props, msg)
      end

      def noop(tag)

      end
    end
  end
end

sneakers:run fails on configuration

On version 0.0.7 rake sneakers:run fails at line 41 in runner.rb file. Sneakers.logger is nil in this case. Probably adding Sneakers.configure into first line of #run method in runner.rb will help, but please let me know if it's good solution.

Don't include amqp connection options in logs

Currently, Sneakers::Worker#worker_trace logs the queue options, which can contain sensitive authentication information (username/password).

The connection details should be scrubbed (only display scheme://host:port/path) prior to being displayed, or omitted entirely (not ideal, as it does provide useful information)

Serverengine is 1.9.3+ only

I think 'serverengine' is the only dependency making Sneakers not run on 1.8.7, which would make Sneakers a top choice for projects trying to move from 1.8.7 to newer rubies such as the one I'm working on.

We're slicing out components out of our main 1.8.7 codebase to run as workers on newer ruby versions and for this it is a requirement that we can queue jobs from 1.8.7, but none of the queue packages like Resque, Sidekiq or Sneakers run on 1.8.7. There are Beanstalk based gems that would work just fine, but I'd like to use something more mainstream (rabbitmq preferably).

Is there something simple that could be done about 1.8.7 compatibility? I understand you may not have any interest in using time to add support for antique versions.

how to re-establish database connection?

how do we reestablish ActiveRecord database connection after forking the worker process?

Testing Your Worker page is gone?

The link to https://github.com/jondot/sneakers/wiki/Testing-Your-Worker at the end of https://github.com/jondot/sneakers/wiki/How-to:-running-a-stand-alone-worker takes me to a "Create new page" page. Sure would love to see that testing info if it's still around somewhere.

0.1.1/0.1.0.pre production stability testing

Full integration tests for 0.1.1, 0.1.0.pre, for worker scenario (non-publisher).

Stackdriver metrics handler

See https://github.com/jondot/sneakers/wiki/Contributing

Until: 6-feb-2014
Difficulty: 30min

A Stackdriver metrics handler was requested, and makes a nice low hanging fruit.
Can use https://github.com/grant-olson/stackdriver-ruby to push data and implementation
can be very similar to existing NewRelic handler.

Mysql2::Error: This connection is still waiting for a result, try again once you have the result

Hello, my worker has reported lots of these errors. After having google the errors, i think it might because two threads shared the same connection.
But i have set only 1 worker with 1 thread. So why could it happen?
Would you please give me a hand?

Be able to access headers in work()

For some cases, that i would like to use sneakers for, i need access to the AMQP headers in the work() method.

Should i just have do_work() hand it through to work()?
This would make it necessary to have work() accept more than one parameter in the future, but i think its the easiest way without changing much else.

unexpected stack trace in worker log

I'm right at the beginning of using sneakers version 1.0.2. I've got what I think is a minimal rspec test:

a wrapper script similar to https://github.com/jondot/sneakers/wiki/How-to:-running-a-stand-alone-worker
no elements in a durable queue
an rspec test that runs the wrapper script, sleeps for a second, sends the ServerEngine::Daemon::Signals::GRACEFUL_STOP signal, and verifies the exit status of the process is 0. I see similar behavior using IMMEDIATE_STOP.

In other words, I'm not putting anything on a queue, nor taking anything off the queue...just starting and stopping workers. 2 workers in this case.

When things are good, I see this in the worker log.

2015-03-10T21:21:30Z p-88652 t-owhtbwxr0 WARN: Loading runner configuration...
2015-03-10T21:21:30Z p-88652 t-owhtbwxr0 INFO: New configuration: #<Sneakers::Configuration:0x007fca2431e598 @hash={:runner_config_file=>nil, :metrics=>nil, :daemonize=>false, :start_worker_delay=>0.2, :workers=>2, :log=>"log/portal_worker.log", :pid_path=>"sneakers.pid", :timeout_job_after=>5, :prefetch=>10, :threads=>10, :durable=>true, :ack=>true, :heartbeat=>2, :exchange=>"sneakers", :exchange_type=>:direct, :hooks=>{}, :amqp=>nil, :vhost=>"/", :handler=>Sneakers::Handlers::Oneshot, :worker_type=>"process", :worker_classes=>[PortalEventWorker]}>
2015-03-10T21:21:30Z p-88652 t-owhtbwxr0 WARN: Loading runner configuration...
2015-03-10T21:21:30Z p-88652 t-owhtbwxr0 INFO: New configuration: #<Sneakers::Configuration:0x007fca24325c30 @hash={:runner_config_file=>nil, :metrics=>nil, :daemonize=>false, :start_worker_delay=>0.2, :workers=>2, :log=>"log/portal_worker.log", :pid_path=>"sneakers.pid", :timeout_job_after=>5, :prefetch=>10, :threads=>10, :durable=>true, :ack=>true, :heartbeat=>2, :exchange=>"sneakers", :exchange_type=>:direct, :hooks=>{}, :amqp=>nil, :vhost=>"/", :handler=>Sneakers::Handlers::Oneshot, :worker_type=>"process", :worker_classes=>[PortalEventWorker]}>
2015-03-10T21:21:30Z p-88652 t-owhtbwxr0 WARN: Loading runner configuration...
2015-03-10T21:21:30Z p-88652 t-owhtbwxr0 INFO: New configuration: #<Sneakers::Configuration:0x007fca2431eb38 @hash={:runner_config_file=>nil, :metrics=>nil, :daemonize=>false, :start_worker_delay=>0.2, :workers=>2, :log=>"log/portal_worker.log", :pid_path=>"sneakers.pid", :timeout_job_after=>5, :prefetch=>10, :threads=>10, :durable=>true, :ack=>true, :heartbeat=>2, :exchange=>"sneakers", :exchange_type=>:direct, :hooks=>{}, :amqp=>nil, :vhost=>"/", :handler=>Sneakers::Handlers::Oneshot, :worker_type=>"process", :worker_classes=>[PortalEventWorker]}>
2015-03-10T21:21:30Z p-88657 t-owhtbwxr0 INFO: Heartbeat interval used (in seconds): 2
2015-03-10T21:21:30Z p-88660 t-owhtbwxr0 INFO: Heartbeat interval used (in seconds): 2
I, [2015-03-10T14:21:30.620961 #88652]  INFO -- : Received graceful stop
2015-03-10T21:21:30Z p-88657 t-owhtd49w0 INFO: Shutting down workers
2015-03-10T21:21:30Z p-88660 t-owhtd49j8 INFO: Shutting down workers
I, [2015-03-10T14:21:30.749632 #88652]  INFO -- : Worker 0 finished with status 0
I, [2015-03-10T14:21:30.749701 #88652]  INFO -- : Worker 1 finished with status 0

When things don't work, I see some variation of:

  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/process_manager.rb:99:in `fork'
I, [2015-03-10T14:27:56.814948 #24799]  INFO -- : Worker 1 finished unexpectedly with status 1
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/process_manager.rb:99:in `fork'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/multi_process_server.rb:69:in `start_worker'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/multi_worker_server.rb:132:in `delayed_start_worker'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/multi_worker_server.rb:105:in `block in keepalive_workers'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/multi_worker_server.rb:97:in `each'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/multi_worker_server.rb:97:in `each_with_index'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/multi_worker_server.rb:97:in `keepalive_workers'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/multi_worker_server.rb:54:in `run'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/multi_process_server.rb:40:in `run'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/server.rb:88:in `main'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/daemon.rb:101:in `main'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/daemon.rb:90:in `run'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/sneakers-1.0.2/lib/sneakers/runner.rb:12:in `run'
  /Users/davidbyron/src/portal_worker/lib/portal_runner.rb:14:in `run'
  bin/event_worker:8:in `<main>'
Unexpected error undefined method `new' for [PortalEventWorker]:Array
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/sneakers-1.0.2/lib/sneakers/workergroup.rb:22:in `block in run'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/sneakers-1.0.2/lib/sneakers/workergroup.rb:22:in `map'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/sneakers-1.0.2/lib/sneakers/workergroup.rb:22:in `run'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/worker.rb:74:in `main'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/multi_process_server.rb:79:in `block in start_worker'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/process_manager.rb:107:in `call'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/process_manager.rb:107:in `block in fork'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/process_manager.rb:99:in `fork'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/process_manager.rb:99:in `fork'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/multi_process_server.rb:69:in `start_worker'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/multi_worker_server.rb:132:in `delayed_start_worker'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/multi_worker_server.rb:105:in `block in keepalive_workers'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/multi_worker_server.rb:97:in `each'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/multi_worker_server.rb:97:in `each_with_index'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/multi_worker_server.rb:97:in `keepalive_workers'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/multi_worker_server.rb:54:in `run'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/multi_process_server.rb:40:in `run'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/server.rb:88:in `main'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/daemon.rb:101:in `main'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/daemon.rb:90:in `run'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/sneakers-1.0.2/lib/sneakers/runner.rb:12:in `run'
  /Users/davidbyron/src/portal_worker/lib/portal_runner.rb:14:in `run'
  bin/event_worker:8:in `<main>'
I, [2015-03-10T14:27:57.487151 #24799]  INFO -- : Worker 2 finished unexpectedly with status 1
Unexpected error undefined method `new' for [PortalEventWorker]:Array
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/sneakers-1.0.2/lib/sneakers/workergroup.rb:22:in `block in run'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/sneakers-1.0.2/lib/sneakers/workergroup.rb:22:in `map'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/sneakers-1.0.2/lib/sneakers/workergroup.rb:22:in `run'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/worker.rb:74:in `main'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/multi_process_server.rb:79:in `block in start_worker'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/process_manager.rb:107:in `call'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/process_manager.rb:107:in `block in fork'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/process_manager.rb:99:in `fork'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/process_manager.rb:99:in `fork'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/multi_process_server.rb:69:in `start_worker'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/multi_worker_server.rb:132:in `delayed_start_worker'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/multi_worker_server.rb:105:in `block in keepalive_workers'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/multi_worker_server.rb:97:in `each'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/multi_worker_server.rb:97:in `each_with_index'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/multi_worker_server.rb:97:in `keepalive_workers'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/multi_worker_server.rb:54:in `run'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/multi_process_server.rb:40:in `run'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/server.rb:88:in `main'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/daemon.rb:101:in `main'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/daemon.rb:90:in `run'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/sneakers-1.0.2/lib/sneakers/runner.rb:12:in `run'
  /Users/davidbyron/src/portal_worker/lib/portal_runner.rb:14:in `run'
  bin/event_worker:8:in `<main>'
I, [2015-03-10T14:27:57.672796 #24799]  INFO -- : Worker 3 finished unexpectedly with status 1
Unexpected error undefined method `new' for [PortalEventWorker]:Array
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/sneakers-1.0.2/lib/sneakers/workergroup.rb:22:in `block in run'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/sneakers-1.0.2/lib/sneakers/workergroup.rb:22:in `map'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/sneakers-1.0.2/lib/sneakers/workergroup.rb:22:in `run'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/worker.rb:74:in `main'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/multi_process_server.rb:79:in `block in start_worker'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/process_manager.rb:107:in `call'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/process_manager.rb:107:in `block in fork'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/process_manager.rb:99:in `fork'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/process_manager.rb:99:in `fork'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/multi_process_server.rb:69:in `start_worker'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/multi_worker_server.rb:132:in `delayed_start_worker'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/multi_worker_server.rb:105:in `block in keepalive_workers'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/multi_worker_server.rb:97:in `each'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/multi_worker_server.rb:97:in `each_with_index'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/multi_worker_server.rb:97:in `keepalive_workers'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/multi_worker_server.rb:54:in `run'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/multi_process_server.rb:40:in `run'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/server.rb:88:in `main'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/daemon.rb:101:in `main'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/daemon.rb:90:in `run'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/sneakers-1.0.2/lib/sneakers/runner.rb:12:in `run'
  /Users/davidbyron/src/portal_worker/lib/portal_runner.rb:14:in `run'
  bin/event_worker:8:in `<main>'
I, [2015-03-10T14:27:57.704474 #24799]  INFO -- : Worker 0 finished unexpectedly with status 1
Unexpected error undefined method `new' for [PortalEventWorker]:Array
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/sneakers-1.0.2/lib/sneakers/workergroup.rb:22:in `block in run'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/sneakers-1.0.2/lib/sneakers/workergroup.rb:22:in `map'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/sneakers-1.0.2/lib/sneakers/workergroup.rb:22:in `run'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/worker.rb:74:in `main'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/multi_process_server.rb:79:in `block in start_worker'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/process_manager.rb:107:in `call'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/process_manager.rb:107:in `block in fork'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/process_manager.rb:99:in `fork'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/process_manager.rb:99:in `fork'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/multi_process_server.rb:69:in `start_worker'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/multi_worker_server.rb:132:in `delayed_start_worker'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/multi_worker_server.rb:105:in `block in keepalive_workers'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/multi_worker_server.rb:97:in `each'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/multi_worker_server.rb:97:in `each_with_index'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/multi_worker_server.rb:97:in `keepalive_workers'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/multi_worker_server.rb:54:in `run'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/multi_process_server.rb:40:in `run'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/server.rb:88:in `main'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/daemon.rb:101:in `main'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/daemon.rb:90:in `run'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/sneakers-1.0.2/lib/sneakers/runner.rb:12:in `run'
  /Users/davidbyron/src/portal_worker/lib/portal_runner.rb:14:in `run'
  bin/event_worker:8:in `<main>'
2015-03-10T21:27:57Z p-92879 t-oxwzwik0g WARN: Loading runner configuration...
2015-03-10T21:27:57Z p-92879 t-oxwzwik0g INFO: New configuration: #<Sneakers::Configuration:0x007ffe04152d90 @hash={:runner_config_file=>nil, :metrics=>nil, :daemonize=>false, :start_worker_delay=>0.2, :workers=>2, :log=>"log/portal_worker.log", :pid_path=>"sneakers.pid", :timeout_job_after=>5, :prefetch=>10, :threads=>10, :durable=>true, :ack=>true, :heartbeat=>2, :exchange=>"sneakers", :exchange_type=>:direct, :hooks=>{}, :amqp=>nil, :vhost=>"/", :handler=>Sneakers::Handlers::Oneshot, :worker_type=>"process", :worker_classes=>[PortalEventWorker]}>
2015-03-10T21:27:57Z p-92879 t-oxwzwik0g WARN: Loading runner configuration...
2015-03-10T21:27:57Z p-92879 t-oxwzwik0g INFO: New configuration: #<Sneakers::Configuration:0x007ffe043a8310 @hash={:runner_config_file=>nil, :metrics=>nil, :daemonize=>false, :start_worker_delay=>0.2, :workers=>2, :log=>"log/portal_worker.log", :pid_path=>"sneakers.pid", :timeout_job_after=>5, :prefetch=>10, :threads=>10, :durable=>true, :ack=>true, :heartbeat=>2, :exchange=>"sneakers", :exchange_type=>:direct, :hooks=>{}, :amqp=>nil, :vhost=>"/", :handler=>Sneakers::Handlers::Oneshot, :worker_type=>"process", :worker_classes=>[PortalEventWorker]}>
2015-03-10T21:27:57Z p-92879 t-oxwzwik0g WARN: Loading runner configuration...
2015-03-10T21:27:57Z p-92879 t-oxwzwik0g INFO: New configuration: #<Sneakers::Configuration:0x007ffe043a1420 @hash={:runner_config_file=>nil, :metrics=>nil, :daemonize=>false, :start_worker_delay=>0.2, :workers=>2, :log=>"log/portal_worker.log", :pid_path=>"sneakers.pid", :timeout_job_after=>5, :prefetch=>10, :threads=>10, :durable=>true, :ack=>true, :heartbeat=>2, :exchange=>"sneakers", :exchange_type=>:direct, :hooks=>{}, :amqp=>nil, :vhost=>"/", :handler=>Sneakers::Handlers::Oneshot, :worker_type=>"process", :worker_classes=>[PortalEventWorker]}>
2015-03-10T21:27:57Z p-92883 t-oxwzwik0g INFO: Heartbeat interval used (in seconds): 2
I, [2015-03-10T14:27:57.869869 #24799]  INFO -- : Worker 1 finished unexpectedly with status 1
Unexpected error undefined method `new' for [PortalEventWorker]:Array
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/sneakers-1.0.2/lib/sneakers/workergroup.rb:22:in `block in run'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/sneakers-1.0.2/lib/sneakers/workergroup.rb:22:in `map'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/sneakers-1.0.2/lib/sneakers/workergroup.rb:22:in `run'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/worker.rb:74:in `main'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/multi_process_server.rb:79:in `block in start_worker'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/process_manager.rb:107:in `call'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/process_manager.rb:107:in `block in fork'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/process_manager.rb:99:in `fork'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/process_manager.rb:99:in `fork'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/multi_process_server.rb:69:in `start_worker'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/multi_worker_server.rb:132:in `delayed_start_worker'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/multi_worker_server.rb:105:in `block in keepalive_workers'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/multi_worker_server.rb:97:in `each'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/multi_worker_server.rb:97:in `each_with_index'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/multi_worker_server.rb:97:in `keepalive_workers'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/multi_worker_server.rb:54:in `run'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/multi_process_server.rb:40:in `run'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/server.rb:88:in `main'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/daemon.rb:101:in `main'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/daemon.rb:90:in `run'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/sneakers-1.0.2/lib/sneakers/runner.rb:12:in `run'
  /Users/davidbyron/src/portal_worker/lib/portal_runner.rb:14:in `run'
  bin/event_worker:8:in `<main>'
I, [2015-03-10T14:27:57.892052 #24799]  INFO -- : Worker 2 finished unexpectedly with status 1
Unexpected error undefined method `new' for [PortalEventWorker]:Array
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/sneakers-1.0.2/lib/sneakers/workergroup.rb:22:in `block in run'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/sneakers-1.0.2/lib/sneakers/workergroup.rb:22:in `map'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/sneakers-1.0.2/lib/sneakers/workergroup.rb:22:in `run'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/worker.rb:74:in `main'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/multi_process_server.rb:79:in `block in start_worker'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/process_manager.rb:107:in `call'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/process_manager.rb:107:in `block in fork'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/process_manager.rb:99:in `fork'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/process_manager.rb:99:in `fork'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/multi_process_server.rb:69:in `start_worker'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/multi_worker_server.rb:132:in `delayed_start_worker'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/multi_worker_server.rb:105:in `block in keepalive_workers'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/multi_worker_server.rb:97:in `each'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/multi_worker_server.rb:97:in `each_with_index'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/multi_worker_server.rb:97:in `keepalive_workers'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/multi_worker_server.rb:54:in `run'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/multi_process_server.rb:40:in `run'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/server.rb:88:in `main'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/daemon.rb:101:in `main'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/daemon.rb:90:in `run'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/sneakers-1.0.2/lib/sneakers/runner.rb:12:in `run'
  /Users/davidbyron/src/portal_worker/lib/portal_runner.rb:14:in `run'
  bin/event_worker:8:in `<main>'
2015-03-10T21:27:58Z p-92886 t-oxwzwik0g INFO: Heartbeat interval used (in seconds): 2
I, [2015-03-10T14:27:58.083744 #24799]  INFO -- : Worker 3 finished unexpectedly with status 1
Unexpected error undefined method `new' for [PortalEventWorker]:Array
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/sneakers-1.0.2/lib/sneakers/workergroup.rb:22:in `block in run'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/sneakers-1.0.2/lib/sneakers/workergroup.rb:22:in `map'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/sneakers-1.0.2/lib/sneakers/workergroup.rb:22:in `run'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/worker.rb:74:in `main'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/multi_process_server.rb:79:in `block in start_worker'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/process_manager.rb:107:in `call'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/process_manager.rb:107:in `block in fork'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/process_manager.rb:99:in `fork'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/process_manager.rb:99:in `fork'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/multi_process_server.rb:69:in `start_worker'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/multi_worker_server.rb:132:in `delayed_start_worker'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/multi_worker_server.rb:105:in `block in keepalive_workers'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/multi_worker_server.rb:97:in `each'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/multi_worker_server.rb:97:in `each_with_index'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/multi_worker_server.rb:97:in `keepalive_workers'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/multi_worker_server.rb:54:in `run'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/multi_process_server.rb:40:in `run'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/server.rb:88:in `main'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/daemon.rb:101:in `main'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/daemon.rb:90:in `run'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/sneakers-1.0.2/lib/sneakers/runner.rb:12:in `run'
  /Users/davidbyron/src/portal_worker/lib/portal_runner.rb:14:in `run'
  bin/event_worker:8:in `<main>'
I, [2015-03-10T14:27:58.098047 #24799]  INFO -- : Worker 0 finished unexpectedly with status 1
Unexpected error undefined method `new' for [PortalEventWorker]:Array
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/sneakers-1.0.2/lib/sneakers/workergroup.rb:22:in `block in run'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/sneakers-1.0.2/lib/sneakers/workergroup.rb:22:in `map'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/sneakers-1.0.2/lib/sneakers/workergroup.rb:22:in `run'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/worker.rb:74:in `main'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/multi_process_server.rb:79:in `block in start_worker'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/process_manager.rb:107:in `call'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/process_manager.rb:107:in `block in fork'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/process_manager.rb:99:in `fork'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/process_manager.rb:99:in `fork'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/multi_process_server.rb:69:in `start_worker'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/multi_worker_server.rb:132:in `delayed_start_worker'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/multi_worker_server.rb:105:in `block in keepalive_workers'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/multi_worker_server.rb:97:in `each'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/multi_worker_server.rb:97:in `each_with_index'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/multi_worker_server.rb:97:in `keepalive_workers'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/multi_worker_server.rb:54:in `run'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/multi_process_server.rb:40:in `run'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/server.rb:88:in `main'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/daemon.rb:101:in `main'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/daemon.rb:90:in `run'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/sneakers-1.0.2/lib/sneakers/runner.rb:12:in `run'
  /Users/davidbyron/src/portal_worker/lib/portal_runner.rb:14:in `run'
  bin/event_worker:8:in `<main>'
I, [2015-03-10T14:27:58.287963 #24799]  INFO -- : Worker 1 finished unexpectedly with status 1
Unexpected error undefined method `new' for [PortalEventWorker]:Array
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/sneakers-1.0.2/lib/sneakers/workergroup.rb:22:in `block in run'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/sneakers-1.0.2/lib/sneakers/workergroup.rb:22:in `map'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/sneakers-1.0.2/lib/sneakers/workergroup.rb:22:in `run'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/worker.rb:74:in `main'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/multi_process_server.rb:79:in `block in start_worker'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/process_manager.rb:107:in `call'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/process_manager.rb:107:in `block in fork'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/process_manager.rb:99:in `fork'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/process_manager.rb:99:in `fork'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/multi_process_server.rb:69:in `start_worker'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/multi_worker_server.rb:132:in `delayed_start_worker'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/multi_worker_server.rb:105:in `block in keepalive_workers'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/multi_worker_server.rb:97:in `each'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/multi_worker_server.rb:97:in `each_with_index'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/multi_worker_server.rb:97:in `keepalive_workers'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/multi_worker_server.rb:54:in `run'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/multi_process_server.rb:40:in `run'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/server.rb:88:in `main'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/daemon.rb:101:in `main'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/daemon.rb:90:in `run'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/sneakers-1.0.2/lib/sneakers/runner.rb:12:in `run'
  /Users/davidbyron/src/portal_worker/lib/portal_runner.rb:14:in `run'
  bin/event_worker:8:in `<main>'
I, [2015-03-10T14:27:58.310475 #24799]  INFO -- : Worker 2 finished unexpectedly with status 1
Unexpected error undefined method `new' for [PortalEventWorker]:Array
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/sneakers-1.0.2/lib/sneakers/workergroup.rb:22:in `block in run'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/sneakers-1.0.2/lib/sneakers/workergroup.rb:22:in `map'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/sneakers-1.0.2/lib/sneakers/workergroup.rb:22:in `run'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/worker.rb:74:in `main'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/multi_process_server.rb:79:in `block in start_worker'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/process_manager.rb:107:in `call'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/process_manager.rb:107:in `block in fork'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/process_manager.rb:99:in `fork'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/process_manager.rb:99:in `fork'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/multi_process_server.rb:69:in `start_worker'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/multi_worker_server.rb:132:in `delayed_start_worker'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/multi_worker_server.rb:105:in `block in keepalive_workers'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/multi_worker_server.rb:97:in `each'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/multi_worker_server.rb:97:in `each_with_index'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/multi_worker_server.rb:97:in `keepalive_workers'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/multi_worker_server.rb:54:in `run'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/multi_process_server.rb:40:in `run'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/server.rb:88:in `main'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/daemon.rb:101:in `main'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/daemon.rb:90:in `run'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/sneakers-1.0.2/lib/sneakers/runner.rb:12:in `run'
  /Users/davidbyron/src/portal_worker/lib/portal_runner.rb:14:in `run'
  bin/event_worker:8:in `<main>'
I, [2015-03-10T14:27:58.410933 #92879]  INFO -- : Received graceful stop
I, [2015-03-10T14:27:58.478447 #24799]  INFO -- : Worker 3 finished unexpectedly with status 1
Unexpected error undefined method `new' for [PortalEventWorker]:Array
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/sneakers-1.0.2/lib/sneakers/workergroup.rb:22:in `block in run'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/sneakers-1.0.2/lib/sneakers/workergroup.rb:22:in `map'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/sneakers-1.0.2/lib/sneakers/workergroup.rb:22:in `run'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/worker.rb:74:in `main'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/multi_process_server.rb:79:in `block in start_worker'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/process_manager.rb:107:in `call'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/process_manager.rb:107:in `block in fork'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/process_manager.rb:99:in `fork'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/process_manager.rb:99:in `fork'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/multi_process_server.rb:69:in `start_worker'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/multi_worker_server.rb:132:in `delayed_start_worker'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/multi_worker_server.rb:105:in `block in keepalive_workers'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/multi_worker_server.rb:97:in `each'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/multi_worker_server.rb:97:in `each_with_index'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/multi_worker_server.rb:97:in `keepalive_workers'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/multi_worker_server.rb:54:in `run'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/multi_process_server.rb:40:in `run'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/server.rb:88:in `main'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/daemon.rb:101:in `main'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/daemon.rb:90:in `run'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/sneakers-1.0.2/lib/sneakers/runner.rb:12:in `run'
  /Users/davidbyron/src/portal_worker/lib/portal_runner.rb:14:in `run'
  bin/event_worker:8:in `<main>'
I, [2015-03-10T14:27:58.502501 #24799]  INFO -- : Worker 0 finished unexpectedly with status 1
Unexpected error undefined method `new' for [PortalEventWorker]:Array
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/sneakers-1.0.2/lib/sneakers/workergroup.rb:22:in `block in run'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/sneakers-1.0.2/lib/sneakers/workergroup.rb:22:in `map'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/sneakers-1.0.2/lib/sneakers/workergroup.rb:22:in `run'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/worker.rb:74:in `main'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/multi_process_server.rb:79:in `block in start_worker'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/process_manager.rb:107:in `call'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/process_manager.rb:107:in `block in fork'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/process_manager.rb:99:in `fork'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/process_manager.rb:99:in `fork'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/multi_process_server.rb:69:in `start_worker'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/multi_worker_server.rb:132:in `delayed_start_worker'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/multi_worker_server.rb:105:in `block in keepalive_workers'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/multi_worker_server.rb:97:in `each'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/multi_worker_server.rb:97:in `each_with_index'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/multi_worker_server.rb:97:in `keepalive_workers'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/multi_worker_server.rb:54:in `run'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/multi_process_server.rb:40:in `run'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/server.rb:88:in `main'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/daemon.rb:101:in `main'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/daemon.rb:90:in `run'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/sneakers-1.0.2/lib/sneakers/runner.rb:12:in `run'
  /Users/davidbyron/src/portal_worker/lib/portal_runner.rb:14:in `run'
  bin/event_worker:8:in `<main>'
2015-03-10T21:27:58Z p-92883 t-oxx0758go INFO: Shutting down workers
2015-03-10T21:27:58Z p-92886 t-oxx07583c INFO: Shutting down workers
I, [2015-03-10T14:27:58.536297 #92879]  INFO -- : Worker 0 finished with status 0
I, [2015-03-10T14:27:58.536418 #92879]  INFO -- : Worker 1 finished with status 0
I, [2015-03-10T14:27:58.679900 #24799]  INFO -- : Worker 1 finished unexpectedly with status 1
Unexpected error undefined method `new' for [PortalEventWorker]:Array
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/sneakers-1.0.2/lib/sneakers/workergroup.rb:22:in `block in run'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/sneakers-1.0.2/lib/sneakers/workergroup.rb:22:in `map'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/sneakers-1.0.2/lib/sneakers/workergroup.rb:22:in `run'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/worker.rb:74:in `main'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/multi_process_server.rb:79:in `block in start_worker'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/process_manager.rb:107:in `call'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/process_manager.rb:107:in `block in fork'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/process_manager.rb:99:in `fork'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/process_manager.rb:99:in `fork'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/multi_process_server.rb:69:in `start_worker'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/multi_worker_server.rb:132:in `delayed_start_worker'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/multi_worker_server.rb:105:in `block in keepalive_workers'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/multi_worker_server.rb:97:in `each'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/multi_worker_server.rb:97:in `each_with_index'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/multi_worker_server.rb:97:in `keepalive_workers'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/multi_worker_server.rb:54:in `run'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/multi_process_server.rb:40:in `run'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/server.rb:88:in `main'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/daemon.rb:101:in `main'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/daemon.rb:90:in `run'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/sneakers-1.0.2/lib/sneakers/runner.rb:12:in `run'
  /Users/davidbyron/src/portal_worker/lib/portal_runner.rb:14:in `run'
  bin/event_worker:8:in `<main>'
I, [2015-03-10T14:27:58.700909 #24799]  INFO -- : Worker 2 finished unexpectedly with status 1
Unexpected error undefined method `new' for [PortalEventWorker]:Array
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/sneakers-1.0.2/lib/sneakers/workergroup.rb:22:in `block in run'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/sneakers-1.0.2/lib/sneakers/workergroup.rb:22:in `map'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/sneakers-1.0.2/lib/sneakers/workergroup.rb:22:in `run'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/worker.rb:74:in `main'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/multi_process_server.rb:79:in `block in start_worker'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/process_manager.rb:107:in `call'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/process_manager.rb:107:in `block in fork'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/process_manager.rb:99:in `fork'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/process_manager.rb:99:in `fork'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/multi_process_server.rb:69:in `start_worker'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/multi_worker_server.rb:132:in `delayed_start_worker'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/multi_worker_server.rb:105:in `block in keepalive_workers'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/multi_worker_server.rb:97:in `each'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/multi_worker_server.rb:97:in `each_with_index'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/multi_worker_server.rb:97:in `keepalive_workers'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/multi_worker_server.rb:54:in `run'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/multi_process_server.rb:40:in `run'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/server.rb:88:in `main'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/daemon.rb:101:in `main'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/daemon.rb:90:in `run'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/sneakers-1.0.2/lib/sneakers/runner.rb:12:in `run'
  /Users/davidbyron/src/portal_worker/lib/portal_runner.rb:14:in `run'
  bin/event_worker:8:in `<main>'
I, [2015-03-10T14:27:58.880941 #24799]  INFO -- : Worker 3 finished unexpectedly with status 1
Unexpected error undefined method `new' for [PortalEventWorker]:Array
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/sneakers-1.0.2/lib/sneakers/workergroup.rb:22:in `block in run'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/sneakers-1.0.2/lib/sneakers/workergroup.rb:22:in `map'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/sneakers-1.0.2/lib/sneakers/workergroup.rb:22:in `run'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/worker.rb:74:in `main'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/multi_process_server.rb:79:in `block in start_worker'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/process_manager.rb:107:in `call'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/process_manager.rb:107:in `block in fork'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/process_manager.rb:99:in `fork'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/process_manager.rb:99:in `fork'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/multi_process_server.rb:69:in `start_worker'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/multi_worker_server.rb:132:in `delayed_start_worker'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/multi_worker_server.rb:105:in `block in keepalive_workers'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/multi_worker_server.rb:97:in `each'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/multi_worker_server.rb:97:in `each_with_index'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/multi_worker_server.rb:97:in `keepalive_workers'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/multi_worker_server.rb:54:in `run'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/multi_process_server.rb:40:in `run'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/server.rb:88:in `main'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/daemon.rb:101:in `main'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/daemon.rb:90:in `run'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/sneakers-1.0.2/lib/sneakers/runner.rb:12:in `run'
  /Users/davidbyron/src/portal_worker/lib/portal_runner.rb:14:in `run'
  bin/event_worker:8:in `<main>'
I, [2015-03-10T14:27:58.906311 #24799]  INFO -- : Worker 0 finished unexpectedly with status 1
Unexpected error undefined method `new' for [PortalEventWorker]:Array
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/sneakers-1.0.2/lib/sneakers/workergroup.rb:22:in `block in run'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/sneakers-1.0.2/lib/sneakers/workergroup.rb:22:in `map'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/sneakers-1.0.2/lib/sneakers/workergroup.rb:22:in `run'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/worker.rb:74:in `main'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/multi_process_server.rb:79:in `block in start_worker'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/process_manager.rb:107:in `call'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/process_manager.rb:107:in `block in fork'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/process_manager.rb:99:in `fork'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/process_manager.rb:99:in `fork'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/multi_process_server.rb:69:in `start_worker'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/multi_worker_server.rb:132:in `delayed_start_worker'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/multi_worker_server.rb:105:in `block in keepalive_workers'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/multi_worker_server.rb:97:in `each'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/multi_worker_server.rb:97:in `each_with_index'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/multi_worker_server.rb:97:in `keepalive_workers'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/multi_worker_server.rb:54:in `run'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/multi_process_server.rb:40:in `run'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/server.rb:88:in `main'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/daemon.rb:101:in `main'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/daemon.rb:90:in `run'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/sneakers-1.0.2/lib/sneakers/runner.rb:12:in `run'
  /Users/davidbyron/src/portal_worker/lib/portal_runner.rb:14:in `run'
  bin/event_worker:8:in `<main>'
I, [2015-03-10T14:27:59.107306 #24799]  INFO -- : Worker 1 finished unexpectedly with status 1
I, [2015-03-10T14:27:59.108368 #24799]  INFO -- : Worker 2 finished unexpectedly with status 1
Unexpected error undefined method `new' for [PortalEventWorker]:Array
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/sneakers-1.0.2/lib/sneakers/workergroup.rb:22:in `block in run'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/sneakers-1.0.2/lib/sneakers/workergroup.rb:22:in `map'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/sneakers-1.0.2/lib/sneakers/workergroup.rb:22:in `run'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/worker.rb:74:in `main'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/multi_process_server.rb:79:in `block in start_worker'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/process_manager.rb:107:in `call'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/process_manager.rb:107:in `block in fork'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/process_manager.rb:99:in `fork'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/process_manager.rb:99:in `fork'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/multi_process_server.rb:69:in `start_worker'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/multi_worker_server.rb:132:in `delayed_start_worker'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/multi_worker_server.rb:105:in `block in keepalive_workers'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/multi_worker_server.rb:97:in `each'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/multi_worker_server.rb:97:in `each_with_index'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/multi_worker_server.rb:97:in `keepalive_workers'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/multi_worker_server.rb:54:in `run'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/multi_process_server.rb:40:in `run'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/server.rb:88:in `main'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/daemon.rb:101:in `main'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/daemon.rb:90:in `run'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/sneakers-1.0.2/lib/sneakers/runner.rb:12:in `run'
  /Users/davidbyron/src/portal_worker/lib/portal_runner.rb:14:in `run'
  bin/event_worker:8:in `<main>'
Unexpected error undefined method `new' for [PortalEventWorker]:Array
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/sneakers-1.0.2/lib/sneakers/workergroup.rb:22:in `block in run'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/sneakers-1.0.2/lib/sneakers/workergroup.rb:22:in `map'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/sneakers-1.0.2/lib/sneakers/workergroup.rb:22:in `run'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/worker.rb:74:in `main'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/multi_process_server.rb:79:in `block in start_worker'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/process_manager.rb:107:in `call'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/process_manager.rb:107:in `block in fork'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/process_manager.rb:99:in `fork'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/process_manager.rb:99:in `fork'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/multi_process_server.rb:69:in `start_worker'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/multi_worker_server.rb:132:in `delayed_start_worker'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/multi_worker_server.rb:105:in `block in keepalive_workers'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/multi_worker_server.rb:97:in `each'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/multi_worker_server.rb:97:in `each_with_index'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/multi_worker_server.rb:97:in `keepalive_workers'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/multi_worker_server.rb:54:in `run'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/multi_process_server.rb:40:in `run'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/server.rb:88:in `main'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/daemon.rb:101:in `main'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/daemon.rb:90:in `run'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/sneakers-1.0.2/lib/sneakers/runner.rb:12:in `run'
  /Users/davidbyron/src/portal_worker/lib/portal_runner.rb:14:in `run'
  bin/event_worker:8:in `<main>'
I, [2015-03-10T14:27:59.319662 #24799]  INFO -- : Worker 3 finished unexpectedly with status 1
I, [2015-03-10T14:27:59.320924 #24799]  INFO -- : Worker 0 finished unexpectedly with status 1
Unexpected error undefined method `new' for [PortalEventWorker]:Array
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/sneakers-1.0.2/lib/sneakers/workergroup.rb:22:in `block in run'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/sneakers-1.0.2/lib/sneakers/workergroup.rb:22:in `map'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/sneakers-1.0.2/lib/sneakers/workergroup.rb:22:in `run'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/worker.rb:74:in `main'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/multi_process_server.rb:79:in `block in start_worker'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/process_manager.rb:107:in `call'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/process_manager.rb:107:in `block in fork'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/process_manager.rb:99:in `fork'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/process_manager.rb:99:in `fork'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/multi_process_server.rb:69:in `start_worker'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/multi_worker_server.rb:132:in `delayed_start_worker'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/multi_worker_server.rb:105:in `block in keepalive_workers'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/multi_worker_server.rb:97:in `each'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/multi_worker_server.rb:97:in `each_with_index'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/multi_worker_server.rb:97:in `keepalive_workers'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/multi_worker_server.rb:54:in `run'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/multi_process_server.rb:40:in `run'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/server.rb:88:in `main'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/daemon.rb:101:in `main'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/daemon.rb:90:in `run'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/sneakers-1.0.2/lib/sneakers/runner.rb:12:in `run'
  /Users/davidbyron/src/portal_worker/lib/portal_runner.rb:14:in `run'
  bin/event_worker:8:in `<main>'
Unexpected error undefined method `new' for [PortalEventWorker]:Array
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/sneakers-1.0.2/lib/sneakers/workergroup.rb:22:in `block in run'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/sneakers-1.0.2/lib/sneakers/workergroup.rb:22:in `map'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/sneakers-1.0.2/lib/sneakers/workergroup.rb:22:in `run'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/worker.rb:74:in `main'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/multi_process_server.rb:79:in `block in start_worker'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/process_manager.rb:107:in `call'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/process_manager.rb:107:in `block in fork'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/process_manager.rb:99:in `fork'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/process_manager.rb:99:in `fork'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/multi_process_server.rb:69:in `start_worker'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/multi_worker_server.rb:132:in `delayed_start_worker'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/multi_worker_server.rb:105:in `block in keepalive_workers'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/multi_worker_server.rb:97:in `each'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/multi_worker_server.rb:97:in `each_with_index'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/multi_worker_server.rb:97:in `keepalive_workers'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/multi_worker_server.rb:54:in `run'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/multi_process_server.rb:40:in `run'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/server.rb:88:in `main'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/daemon.rb:101:in `main'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/daemon.rb:90:in `run'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/sneakers-1.0.2/lib/sneakers/runner.rb:12:in `run'
  /Users/davidbyron/src/portal_worker/lib/portal_runner.rb:14:in `run'
  bin/event_worker:8:in `<main>'
I, [2015-03-10T14:27:59.512837 #24799]  INFO -- : Worker 1 finished unexpectedly with status 1
I, [2015-03-10T14:27:59.513886 #24799]  INFO -- : Worker 2 finished unexpectedly with status 1
Unexpected error undefined method `new' for [PortalEventWorker]:Array
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/sneakers-1.0.2/lib/sneakers/workergroup.rb:22:in `block in run'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/sneakers-1.0.2/lib/sneakers/workergroup.rb:22:in `map'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/sneakers-1.0.2/lib/sneakers/workergroup.rb:22:in `run'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/worker.rb:74:in `main'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/multi_process_server.rb:79:in `block in start_worker'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/process_manager.rb:107:in `call'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/process_manager.rb:107:in `block in fork'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/process_manager.rb:99:in `fork'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/process_manager.rb:99:in `fork'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/multi_process_server.rb:69:in `start_worker'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/multi_worker_server.rb:132:in `delayed_start_worker'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/multi_worker_server.rb:105:in `block in keepalive_workers'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/multi_worker_server.rb:97:in `each'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/multi_worker_server.rb:97:in `each_with_index'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/multi_worker_server.rb:97:in `keepalive_workers'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/multi_worker_server.rb:54:in `run'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/multi_process_server.rb:40:in `run'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/server.rb:88:in `main'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/daemon.rb:101:in `main'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/daemon.rb:90:in `run'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/sneakers-1.0.2/lib/sneakers/runner.rb:12:in `run'
  /Users/davidbyron/src/portal_worker/lib/portal_runner.rb:14:in `run'
  bin/event_worker:8:in `<main>'
Unexpected error undefined method `new' for [PortalEventWorker]:Array
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/sneakers-1.0.2/lib/sneakers/workergroup.rb:22:in `block in run'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/sneakers-1.0.2/lib/sneakers/workergroup.rb:22:in `map'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/sneakers-1.0.2/lib/sneakers/workergroup.rb:22:in `run'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/worker.rb:74:in `main'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/multi_process_server.rb:79:in `block in start_worker'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/process_manager.rb:107:in `call'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/process_manager.rb:107:in `block in fork'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/process_manager.rb:99:in `fork'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/process_manager.rb:99:in `fork'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/multi_process_server.rb:69:in `start_worker'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/multi_worker_server.rb:132:in `delayed_start_worker'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/multi_worker_server.rb:105:in `block in keepalive_workers'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/multi_worker_server.rb:97:in `each'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/multi_worker_server.rb:97:in `each_with_index'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/multi_worker_server.rb:97:in `keepalive_workers'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/multi_worker_server.rb:54:in `run'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/multi_process_server.rb:40:in `run'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/server.rb:88:in `main'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/daemon.rb:101:in `main'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/daemon.rb:90:in `run'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/sneakers-1.0.2/lib/sneakers/runner.rb:12:in `run'
  /Users/davidbyron/src/portal_worker/lib/portal_runner.rb:14:in `run'
  bin/event_worker:8:in `<main>'
I, [2015-03-10T14:27:59.708275 #24799]  INFO -- : Worker 3 finished unexpectedly with status 1
Unexpected error undefined method `new' for [PortalEventWorker]:Array
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/sneakers-1.0.2/lib/sneakers/workergroup.rb:22:in `block in run'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/sneakers-1.0.2/lib/sneakers/workergroup.rb:22:in `map'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/sneakers-1.0.2/lib/sneakers/workergroup.rb:22:in `run'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/worker.rb:74:in `main'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/multi_process_server.rb:79:in `block in start_worker'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/process_manager.rb:107:in `call'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/process_manager.rb:107:in `block in fork'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/process_manager.rb:99:in `fork'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/process_manager.rb:99:in `fork'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/multi_process_server.rb:69:in `start_worker'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/multi_worker_server.rb:132:in `delayed_start_worker'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/multi_worker_server.rb:105:in `block in keepalive_workers'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/multi_worker_server.rb:97:in `each'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/multi_worker_server.rb:97:in `each_with_index'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/multi_worker_server.rb:97:in `keepalive_workers'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/multi_worker_server.rb:54:in `run'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/multi_process_server.rb:40:in `run'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/server.rb:88:in `main'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/daemon.rb:101:in `main'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/daemon.rb:90:in `run'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/sneakers-1.0.2/lib/sneakers/runner.rb:12:in `run'
  /Users/davidbyron/src/portal_worker/lib/portal_runner.rb:14:in `run'
  bin/event_worker:8:in `<main>'
I, [2015-03-10T14:27:59.733075 #24799]  INFO -- : Worker 0 finished unexpectedly with status 1
Unexpected error undefined method `new' for [PortalEventWorker]:Array
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/sneakers-1.0.2/lib/sneakers/workergroup.rb:22:in `block in run'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/sneakers-1.0.2/lib/sneakers/workergroup.rb:22:in `map'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/sneakers-1.0.2/lib/sneakers/workergroup.rb:22:in `run'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/worker.rb:74:in `main'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/multi_process_server.rb:79:in `block in start_worker'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/process_manager.rb:107:in `call'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/process_manager.rb:107:in `block in fork'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/process_manager.rb:99:in `fork'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/process_manager.rb:99:in `fork'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/multi_process_server.rb:69:in `start_worker'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/multi_worker_server.rb:132:in `delayed_start_worker'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/multi_worker_server.rb:105:in `block in keepalive_workers'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/multi_worker_server.rb:97:in `each'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/multi_worker_server.rb:97:in `each_with_index'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/multi_worker_server.rb:97:in `keepalive_workers'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/multi_worker_server.rb:54:in `run'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/multi_process_server.rb:40:in `run'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/server.rb:88:in `main'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/daemon.rb:101:in `main'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/daemon.rb:90:in `run'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/sneakers-1.0.2/lib/sneakers/runner.rb:12:in `run'
  /Users/davidbyron/src/portal_worker/lib/portal_runner.rb:14:in `run'
  bin/event_worker:8:in `<main>'
I, [2015-03-10T14:27:59.913021 #24799]  INFO -- : Worker 1 finished unexpectedly with status 1
Unexpected error undefined method `new' for [PortalEventWorker]:Array
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/sneakers-1.0.2/lib/sneakers/workergroup.rb:22:in `block in run'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/sneakers-1.0.2/lib/sneakers/workergroup.rb:22:in `map'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/sneakers-1.0.2/lib/sneakers/workergroup.rb:22:in `run'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/worker.rb:74:in `main'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/multi_process_server.rb:79:in `block in start_worker'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/process_manager.rb:107:in `call'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/process_manager.rb:107:in `block in fork'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/process_manager.rb:99:in `fork'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/process_manager.rb:99:in `fork'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/multi_process_server.rb:69:in `start_worker'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/multi_worker_server.rb:132:in `delayed_start_worker'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/multi_worker_server.rb:105:in `block in keepalive_workers'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/multi_worker_server.rb:97:in `each'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/multi_worker_server.rb:97:in `each_with_index'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/multi_worker_server.rb:97:in `keepalive_workers'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/multi_worker_server.rb:54:in `run'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/multi_process_server.rb:40:in `run'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/server.rb:88:in `main'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/daemon.rb:101:in `main'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/daemon.rb:90:in `run'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/sneakers-1.0.2/lib/sneakers/runner.rb:12:in `run'
  /Users/davidbyron/src/portal_worker/lib/portal_runner.rb:14:in `run'
  bin/event_worker:8:in `<main>'
I, [2015-03-10T14:27:59.934361 #24799]  INFO -- : Worker 2 finished unexpectedly with status 1
Unexpected error undefined method `new' for [PortalEventWorker]:Array
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/sneakers-1.0.2/lib/sneakers/workergroup.rb:22:in `block in run'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/sneakers-1.0.2/lib/sneakers/workergroup.rb:22:in `map'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/sneakers-1.0.2/lib/sneakers/workergroup.rb:22:in `run'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/worker.rb:74:in `main'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/multi_process_server.rb:79:in `block in start_worker'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/process_manager.rb:107:in `call'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/process_manager.rb:107:in `block in fork'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/process_manager.rb:99:in `fork'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/process_manager.rb:99:in `fork'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/multi_process_server.rb:69:in `start_worker'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/multi_worker_server.rb:132:in `delayed_start_worker'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/multi_worker_server.rb:105:in `block in keepalive_workers'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/multi_worker_server.rb:97:in `each'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/multi_worker_server.rb:97:in `each_with_index'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/multi_worker_server.rb:97:in `keepalive_workers'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/multi_worker_server.rb:54:in `run'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/multi_process_server.rb:40:in `run'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/server.rb:88:in `main'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/daemon.rb:101:in `main'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/serverengine-1.5.10/lib/serverengine/daemon.rb:90:in `run'
  /Users/davidbyron/src/portal_worker/vendor/bundle/gems/sneakers-1.0.2/lib/sneakers/runner.rb:12:in `run'
  /Users/davidbyron/src/portal_worker/lib/portal_runner.rb:14:in `run'
  bin/event_worker:8:in `<main>'
I, [2015-03-10T14:28:00.116158 #24799]  INFO -- : Worker 3 finished unexpectedly with status 1

It sort of looks like something is stomping on the config data structure here: https://github.com/jondot/sneakers/blob/master/lib/sneakers/workergroup.rb#L22, but I haven't gotten much farther than that. Adding more log output changes the timing enough that I no longer see the stack traces.

I haven't figured out yet whether this is a sneakers issue or a serverengine issue. I'd love some help.

Apologies for the mixed log message formatting. With any luck it doesn't totally get in the way.

Thanks much.

-DB

all messages seems to receive an ack

i'm using sneakers (0.1.1.pre) but it happens with sneakers (0.0.7)
all message receive an ack, even i specify reject! in the work method of the worker OR set

from_queue(
    'queue_name',
    :ack => false
  )

ideas about?

pass logger to bunny connection

make bunny logger configurable, otherwise it will use its default logger.

"Could not obtain a database connection" and other concurrency errors

I am having a lot of database timeout errors, which tend to occur in groups like this:

2014-04-14T00:42:40Z p-29936 t-owd74tqb4 ERROR: could not obtain a database connection within 5.000 seconds (waited 5.000 seconds)
2014-04-14T00:42:40Z p-29936 t-owd74tk38 ERROR: could not obtain a database connection within 5.000 seconds (waited 5.000 seconds)
2014-04-14T00:42:40Z p-29936 t-owd74tqms ERROR: could not obtain a database connection within 5.000 seconds (waited 5.000 seconds)
2014-04-14T00:42:40Z p-29936 t-owd74tk74 ERROR: could not obtain a database connection within 5.000 seconds (waited 5.000 seconds)
2014-04-14T00:42:40Z p-29936 t-owd74tqiw ERROR: could not obtain a database connection within 5.000 seconds (waited 5.000 seconds)
2014-04-14T00:42:40Z p-29936 t-owd74u3pc ERROR: could not obtain a database connection within 5.000 seconds (waited 5.000 seconds)
2014-04-14T00:42:40Z p-29967 t-owd74tk5g ERROR: could not obtain a database connection within 5.000 seconds (waited 5.000 seconds)
2014-04-14T00:42:40Z p-29967 t-owd74tra4 ERROR: could not obtain a database connection within 5.000 seconds (waited 5.000 seconds)
2014-04-14T00:42:40Z p-29967 t-owd74tqyg ERROR: could not obtain a database connection within 5.000 seconds (waited 5.000 seconds)

This is after setting pool: 15 in database.yml and setting both :prefetch => 15 and :threads => 15 in my sneakers initializer (config/initializers/sneakers.rb). I have my Postgres database configured to accept 300 connections and the sneakers rake task is the only thing running against it. This is running on an Intel i5 CPU which presents 4 cores to the OS, so between that and the default :workers value in lib/sneakers.rb I expect it's starting 4 processes. When I change these thread variables to 25, the errors seem less frequent, but I still see them. I read the Auto-scaling section in the wiki and I'm not sure why this should happen if the pool size == the sneakers thread size.

Here are some other errors I'm seeing that seem concurrency-related (sorry for the weird linebreaks, copied+pasted from my console):

2014-04-14T01:23:50Z p-21926 t-orivh32jo ERROR: deadlock; recursive locking
2014-04-14T01:23:50Z p-21926 t-orivh32jo ERROR: deadlock; recursive locking
2014-04-14T01:23:50Z p-21926 t-orivh32jo ERROR: deadlock; recursive locking
2014-04-14T01:23:50Z p-21926 t-orivh0fmg ERROR: Attempt to unlock a mutex which is locked by another thread
2014-04-14T01:23:50Z p-21926 t-orivh1b88 ERROR: Attempt to unlock a mutex which is locked by another thread
2014-04-14T01:23:50Z p-21926 t-orivh1bvk ERROR: Attempt to unlock a mutex which is locked by another thread
2014-04-14T01:23:50Z p-21926 t-orivh26sc ERROR: Attempt to unlock a mutex which is locked by another thread
2014-04-14T01:23:50Z p-21926 t-orivh0sh8 ERROR: Attempt to unlock a mutex which is locked by another thread
2014-04-14T01:23:50Z p-21926 t-orivh32jo ERROR: deadlock; recursive locking
2014-04-14T01:23:50Z p-21926 t-orivh26w8 ERROR: Attempt to unlock a mutex which is locked by another thread
2014-04-14T01:23:50Z p-21926 t-orivh0s1o ERROR: Attempt to unlock a mutex which is locked by another thread
2014-04-14T01:23:50Z p-21926 t-orivh1bg0 ERROR: Attempt to unlock a mutex which is locked by another thread
2014-04-14T01:23:50Z p-21926 t-orivh32vc ERROR: Attempt to unlock a mutex which is locked by another thread
2014-04-14T01:23:50Z p-21926 t-orivh0sl4 ERROR: Attempt to unlock a mutex which is locked by another thread
2014-04-14T01:23:50Z p-21926 t-orivh0lzw ERROR: Attempt to unlock a mutex which is locked by another thread
2014-04-14T01:23:50Z p-21926 t-orivh26kk ERROR: Attempt to unlock a mutex which is locked by another thread
2014-04-14T01:23:50Z p-21926 t-orivh0ssw ERROR: Attempt to unlock a mutex which is locked by another thread
2014-04-14T01:23:50Z p-21926 t-orivh32jo ERROR: deadlock; recursive locking
2014-04-14T01:23:50Z p-21926 t-orivh32jo ERROR: deadlock; recursive locking
2014-04-14T01:23:50Z p-21926 t-orivh32jo ERROR: deadlock; recursive locking

2014-04-14T01:23:36Z p-21967 t-oriv4a1l4 ERROR: PG::DuplicatePstatement: ERROR:  prepared statement "a9" already exists

2014-04-14T01:23:41Z p-21967 t-orivh3ybk ERROR: timeout
E, [2014-04-14T01:23:41.352062 #21957] ERROR -- #<Bunny::Session:69863549381840 [email protected]:5672, vhost=/>: Exc
eption in the reader loop: IOError: closed stream
E, [2014-04-14T01:23:41.393205 #21957] ERROR -- #<Bunny::Session:69863549381840 [email protected]:5672, vhost=/>: Bac
ktrace: 
E, [2014-04-14T01:23:41.393251 #21957] ERROR -- #<Bunny::Session:69863549381840 [email protected]:5672, vhost=/>:   /
home/vagrant/.rbenv/versions/2.0.0-p451/lib/ruby/gems/2.0.0/gems/bunny-1.1.9/lib/bunny/cruby/socket.rb:48:in `select'
E, [2014-04-14T01:23:41.393272 #21957] ERROR -- #<Bunny::Session:69863549381840 [email protected]:5672, vhost=/>:   /
home/vagrant/.rbenv/versions/2.0.0-p451/lib/ruby/gems/2.0.0/gems/bunny-1.1.9/lib/bunny/cruby/socket.rb:48:in `rescue in
 read_fully'
E, [2014-04-14T01:23:41.393305 #21957] ERROR -- #<Bunny::Session:69863549381840 [email protected]:5672, vhost=/>:   /
home/vagrant/.rbenv/versions/2.0.0-p451/lib/ruby/gems/2.0.0/gems/bunny-1.1.9/lib/bunny/cruby/socket.rb:39:in `read_full
y'
E, [2014-04-14T01:23:41.393323 #21957] ERROR -- #<Bunny::Session:69863549381840 [email protected]:5672, vhost=/>:   /
home/vagrant/.rbenv/versions/2.0.0-p451/lib/ruby/gems/2.0.0/gems/bunny-1.1.9/lib/bunny/transport.rb:200:in `read_next_f
rame'
E, [2014-04-14T01:23:41.393340 #21957] ERROR -- #<Bunny::Session:69863549381840 [email protected]:5672, vhost=/>:   /
home/vagrant/.rbenv/versions/2.0.0-p451/lib/ruby/gems/2.0.0/gems/bunny-1.1.9/lib/bunny/reader_loop.rb:63:in `run_once'
E, [2014-04-14T01:23:41.393356 #21957] ERROR -- #<Bunny::Session:69863549381840 [email protected]:5672, vhost=/>:   /
home/vagrant/.rbenv/versions/2.0.0-p451/lib/ruby/gems/2.0.0/gems/bunny-1.1.9/lib/bunny/reader_loop.rb:33:in `block in r
un_loop'
E, [2014-04-14T01:23:41.393372 #21957] ERROR -- #<Bunny::Session:69863549381840 [email protected]:5672, vhost=/>:   /
home/vagrant/.rbenv/versions/2.0.0-p451/lib/ruby/gems/2.0.0/gems/bunny-1.1.9/lib/bunny/reader_loop.rb:30:in `loop'
E, [2014-04-14T01:23:41.393390 #21957] ERROR -- #<Bunny::Session:69863549381840 [email protected]:5672, vhost=/>:   /
home/vagrant/.rbenv/versions/2.0.0-p451/lib/ruby/gems/2.0.0/gems/bunny-1.1.9/lib/bunny/reader_loop.rb:30:in `run_loop'
2014-04-14T01:23:41Z p-21967 t-orivh1bvk ERROR: timeout
E, [2014-04-14T01:23:41.537716 #21957] ERROR -- #<Bunny::Session:69863550593340 [email protected]:5672, vhost=/>: Exc
eption in the reader loop: IOError: stream closed
2014-04-14T01:23:41Z p-21967 t-orivh0s1o ERROR: timeout
2014-04-14T01:23:41Z p-21957 t-orivh0m3s ERROR: timeout
E, [2014-04-14T01:23:41.340814
 #21957] ERROR -- #<Bunny::Session:69863555601680 [email protected]:5672, vhost=/>: Exception in the reader loop: Err
no::ECONNRESET: Connection reset by peer
E, [2014-04-14T01:23:42.028937 #21957] ERROR -- #<Bunny::Session:69863555601680 [email protected]:5672, vhost=/>: Bac
ktrace: 
E, [2014-04-14T01:23:42.028967 #21957] ERROR -- #<Bunny::Session:69863555601680 [email protected]:5672, vhost=/>:   /
home/vagrant/.rbenv/versions/2.0.0-p451/lib/ruby/gems/2.0.0/gems/bunny-1.1.9/lib/bunny/cruby/socket.rb:41:in `read_nonb
lock'
E, [2014-04-14T01:23:42.028991 #21957] ERROR -- #<Bunny::Session:69863555601680 [email protected]:5672, vhost=/>:   /
home/vagrant/.rbenv/versions/2.0.0-p451/lib/ruby/gems/2.0.0/gems/bunny-1.1.9/lib/bunny/cruby/socket.rb:41:in `block in 
read_fully'
E, [2014-04-14T01:23:42.029014 #21957] ERROR -- #<Bunny::Session:69863555601680 [email protected]:5672, vhost=/>:   /
home/vagrant/.rbenv/versions/2.0.0-p451/lib/ruby/gems/2.0.0/gems/bunny-1.1.9/lib/bunny/cruby/socket.rb:40:in `loop'
E, [2014-04-14T01:23:42.029035 #21957] ERROR -- #<Bunny::Session:69863555601680 [email protected]:5672, vhost=/>:   /
home/vagrant/.rbenv/versions/2.0.0-p451/lib/ruby/gems/2.0.0/gems/bunny-1.1.9/lib/bunny/cruby/socket.rb:40:in `read_full
y'
E, [2014-04-14T01:23:42.029052 #21957] ERROR -- #<Bunny::Session:69863555601680 [email protected]:5672, vhost=/>:   /
home/vagrant/.rbenv/versions/2.0.0-p451/lib/ruby/gems/2.0.0/gems/bunny-1.1.9/lib/bunny/transport.rb:200:in `read_next_f
rame'
E, [2014-04-14T01:23:42.029077 #21957] ERROR -- #<Bunny::Session:69863555601680 [email protected]:5672, vhost=/>:   /
home/vagrant/.rbenv/versions/2.0.0-p451/lib/ruby/gems/2.0.0/gems/bunny-1.1.9/lib/bunny/reader_loop.rb:63:in `run_once'
E, [2014-04-14T01:23:42.029095 #21957] ERROR -- #<Bunny::Session:69863555601680 [email protected]:5672, vhost=/>:   /
home/vagrant/.rbenv/versions/2.0.0-p451/lib/ruby/gems/2.0.0/gems/bunny-1.1.9/lib/bunny/reader_loop.rb:33:in `block in r
un_loop'
E, [2014-04-14T01:23:42.029115 #21957] ERROR -- #<Bunny::Session:69863555601680 [email protected]:5672, vhost=/>:   /
home/vagrant/.rbenv/versions/2.0.0-p451/lib/ruby/gems/2.0.0/gems/bunny-1.1.9/lib/bunny/reader_loop.rb:30:in `loop'
E, [2014-04-14T01:23:42.029133 #21957] ERROR -- #<Bunny::Session:69863555601680 [email protected]:5672, vhost=/>:   /
home/vagrant/.rbenv/versions/2.0.0-p451/lib/ruby/gems/2.0.0/gems/bunny-1.1.9/lib/bunny/reader_loop.rb:30:in `run_loop'
2014-04-14T01:23:41Z p-21967 t-orivh26u0 ERROR: timeout

Unexpected error Connection-level error: CHANNEL_ERROR - expected 'channel.open'

My methodology for testing here is to queue up a thousand or more messages in RabbitMQ, then start up sneakers and see how it handles a cold start.

edit: Here's my full sneakers initializer if that's helpful:

require 'sneakers'
Sneakers.configure(:amqp      => ENV['AMQP_URL'],
                   :daemonize => false,
                   :prefetch  => 15,
                   :threads   => 15)

self-daemonization is depricated

It's news to me that self-demonization is now considered a bad practice.
I think it should be left in for people that want to use it.
Can author elaborate why it is depricated and marked for removal?

Does it have ssl support?

Asynchronously publish message

Is it possible to blindly enqueue the job without waiting for rabbitmq confirmation. It will definitely speed up the web response time since it doesn't need to wait on rabbitmq. Especially if rabbitmq is under heavy load.

And if there was an error, just simply log it, or provide a hook to asynchronously handle it.

Version number confusion

Hey there, I'm a little confused about the version of sneakers. The current master branch has the version at 0.1.1.pre and yet rubygems lists the latest version as 1.0.0.

Also, 1.0.0 won't work for me because it depends on bunny 1.1.3 when I have other dependencies using bunny 1.6.3. I see the recent pull request #77 fixing this, when will this be published?

Also, also, when I try and require the latest sneakers (1.0.0 as installed by rubygems) I get this: NameError: uninitialized constant Sneakers::Configuration::Forwardable.

Workflow suggestion: tag when releasing new gem?

There are now 11 releases at https://rubygems.org/gems/sneakers, but only 2 tags in the repo. Would it be possible to tag the git-repo when a new gem-version is created? That way it becomes easy to review changes, github supports it very well with urls like this: 0.1.0.pre...0.1.1.pre

Just a suggestion, thanks!

Support for `Sneakers::Worker#nack!`

The wiki refers to a Sneakers::Worker#nack!, but no such thing seems to exist in the worker or the specs.

Sneakers seems not to respect the vhost parameter from the amqp_url

Hi,

I had an issue where the vhost parameter from the amqp_url was not respected.

As example for the amqp_url "amqp://user:password@rabbitmq-host:5672/%2fvhost"
Sneakers would set the vhost to the default value of "/" instead of "/vhost".

Only setting the vhost option explicitly to "/vhost" solved the issue.

Thanks,
Nik

Sneakers workers only log error messages, not backtraces

Sneakers currently only logs error messages, not backtraces. This makes debugging difficult. I'm not sure if this is by design; if not I'm happy to fix it and send a pull request.

Sneakers::Publisher does not preserve vhost

I configured sneakers using the following parameters

Sneakers.configure :heartbeat => 2,
:amqp => 'amqp://guest:guest@hostname:5672',
:vhost => 'search',
:exchange => 'search.topic',
:exchange_type => :topic

And tried to publish to a queue in the search vhost

p = Publisher.new
p.publish('blah', to_queue: 'search.something')

but nothing showed up in my queue.

So I dug into the code: https://github.com/jondot/sneakers/blob/master/lib/sneakers/publisher.rb

And I discovered this snippet:

def ensure_connection!
@bunny = Bunny.new(@opts[:amqp], heartbeat: @opts[:heartbeat])
@bunny.start
@channel = @bunny.create_channel
@eXchange = @channel.exchange(@opts[:exchange], type: @opts[:exchange_type], durable: @opts[:durable])
end

Which suggests that the vhost needs to be passed through when Bunny.new is called: http://rubybunny.info/articles/connecting.html

So I propose that we change the call to read:

@bunny = Bunny.new(@opts[:amqp], hearbeat: @opts[:heartbeat], vhost: @opts[:vhost])

rake sneakers:run sometimes ignores SIGINT

Sometimes when running the rake task, the workers do not respond to SIGINT. I send a Ctrl+c and it says it's shutting down workers, but it doesn't. In fact I can hold Ctrl+c and it just floods the screen with "Shutting down workers" messages, but doesn't shut everything down. If I send the rake task to background with Ctrl+z and try to kill the PID, it likewise doesn't respond. I end up having to kill -9 it.

In my initializer I have :daemonize => false, in case that's relevant.

Here is the console output from a run I just had where this occurred (sorry for the weird linebreaks, copied+pasted from my console):

^C2014-04-13T22:59:48Z p-23076 t-os3pt2j64 INFO: Heartbeat: running threads [54]
2014-04-13T22:59:48Z p-23076 t-os3q5m354 INFO: Shutting down workers
2014-04-13T22:59:48Z p-23076 t-os3q5m354 DEBUG: [worker-store1:1:hhkhcb][#<ServerEngine::SignalThread:0x007f
29e3d16cd0>][store1][{:runner_config_file=>nil, :metrics=>nil, :daemonize=>false, :start_worker_delay=>0.2, 
:workers=>4, :log=>#<IO:<STDOUT>>, :pid_path=>"sneakers.pid", :timeout_job_after=>30, :prefetch=>15, :threads=>15, :dur
able=>true, :ack=>true, :heartbeat=>2, :amqp=>"amqp://foo:[email protected]:5672", :vhost=>"/", :exchange=>"sneakers", :exchange_type=>:dir
ect, :hooks=>{}, :handler=>Sneakers::Handlers::Oneshot}] Stopping worker: unsubscribing.
2014-04-13T22:59:48Z p-23076 t-os3q5m354 DEBUG: [worker-store1:1:hhkhcb][#<ServerEngine::SignalThread:0x007f
29e3d16cd0>][store1][{:runner_config_file=>nil, :metrics=>nil, :daemonize=>false, :start_worker_delay=>0.2, 
:workers=>4, :log=>#<IO:<STDOUT>>, :pid_path=>"sneakers.pid", :timeout_job_after=>30, :prefetch=>15, :threads=>15, :dur
able=>true, :ack=>true, :heartbeat=>2, :amqp=>"amqp://foo:[email protected]:5672", :vhost=>"/", :exchange=>"sneakers", :exchange_type=>:dir
ect, :hooks=>{}, :handler=>Sneakers::Handlers::Oneshot}] Stopping worker: I'm gone.
2014-04-13T22:59:48Z p-23076 t-os3q5m354 DEBUG: [worker-store2:1:fm59b2][#<ServerEngine::SignalThread:0x007f29e3
d16cd0>][store2][{:runner_config_file=>nil, :metrics=>nil, :daemonize=>false, :start_worker_delay=>0.2, :workers
=>4, :log=>#<IO:<STDOUT>>, :pid_path=>"sneakers.pid", :timeout_job_after=>30, :prefetch=>15, :threads=>15, :durable=>tr
ue, :ack=>true, :heartbeat=>2, :amqp=>"amqp://foo:[email protected]:5672", :vhost=>"/", :exchange=>"sneakers", :exchange_type=>:direct, :ho
oks=>{}, :handler=>Sneakers::Handlers::Oneshot}] Stopping worker: unsubscribing.
Unexpected error Trying to send frame through a closed connection. Frame is #<AMQ::Protocol::MethodFrame:0x007f29ac0694
10 @payload="\x00<\x00\x1E bunny-1397429831000-559756359092\x00", @channel=2>, method class is AMQ::Protocol::Basic::Ca
ncel
  /home/vagrant/.rbenv/versions/2.0.0-p451/lib/ruby/gems/2.0.0/gems/bunny-1.1.9/lib/bunny/session.rb:846:in `send_frame
'
  /home/vagrant/.rbenv/versions/2.0.0-p451/lib/ruby/gems/2.0.0/gems/bunny-1.1.9/lib/bunny/channel.rb:917:in `basic_canc
el'
  /home/vagrant/.rbenv/versions/2.0.0-p451/lib/ruby/gems/2.0.0/gems/bunny-1.1.9/lib/bunny/consumer.rb:84:in `cancel'
  /home/vagrant/.rbenv/versions/2.0.0-p451/lib/ruby/gems/2.0.0/gems/sneakers-0.1.1.pre/lib/sneakers/queue.rb:48:in `uns
ubscribe'
  /home/vagrant/.rbenv/versions/2.0.0-p451/lib/ruby/gems/2.0.0/gems/sneakers-0.1.1.pre/lib/sneakers/worker.rb:93:in `st
op'
  /home/vagrant/.rbenv/versions/2.0.0-p451/lib/ruby/gems/2.0.0/gems/sneakers-0.1.1.pre/lib/sneakers/workergroup.rb:40:i
n `block in stop'
  /home/vagrant/.rbenv/versions/2.0.0-p451/lib/ruby/gems/2.0.0/gems/sneakers-0.1.1.pre/lib/sneakers/workergroup.rb:39:i
n `each'
  /home/vagrant/.rbenv/versions/2.0.0-p451/lib/ruby/gems/2.0.0/gems/sneakers-0.1.1.pre/lib/sneakers/workergroup.rb:39:i
n `stop'
  /home/vagrant/.rbenv/versions/2.0.0-p451/lib/ruby/gems/2.0.0/gems/serverengine-1.5.7/lib/serverengine/worker.rb:60:in
 `block (2 levels) in install_signal_handlers'
  /home/vagrant/.rbenv/versions/2.0.0-p451/lib/ruby/gems/2.0.0/gems/serverengine-1.5.7/lib/serverengine/signal_thread.r
b:98:in `call'
  /home/vagrant/.rbenv/versions/2.0.0-p451/lib/ruby/gems/2.0.0/gems/serverengine-1.5.7/lib/serverengine/signal_thread.r
b:98:in `main'
^C2014-04-13T22:59:48Z p-23076 t-os3pt2j64 INFO: Heartbeat: running threads [54]
2014-04-13T22:59:48Z p-23076 t-os3q5m354 INFO: Shutting down workers
2014-04-13T22:59:48Z p-23076 t-os3q5m354 DEBUG: [worker-store1:1:hhkhcb][#<ServerEngine::SignalThread:0x007f
29e3d16cd0>][store1][{:runner_config_file=>nil, :metrics=>nil, :daemonize=>false, :start_worker_delay=>0.2, :workers=>4, :log=>#<IO:<STDOUT>>, :pid_path=>"sneakers.pid", :timeout_job_after=>30, :prefetch=>15, :threads=>15, :dur
able=>true, :ack=>true, :heartbeat=>2, :amqp=>"amqp://foo:[email protected]:5672", :vhost=>"/", :exchange=>"sneakers", :exchange_type=>:dir
ect, :hooks=>{}, :handler=>Sneakers::Handlers::Oneshot}] Stopping worker: unsubscribing.
2014-04-13T22:59:48Z p-23076 t-os3q5m354 DEBUG: [worker-store1:1:hhkhcb][#<ServerEngine::SignalThread:0x007f
29e3d16cd0>][store1][{:runner_config_file=>nil, :metrics=>nil, :daemonize=>false, :start_worker_delay=>0.2, 
:workers=>4, :log=>#<IO:<STDOUT>>, :pid_path=>"sneakers.pid", :timeout_job_after=>30, :prefetch=>15, :threads=>15, :dur
able=>true, :ack=>true, :heartbeat=>2, :amqp=>"amqp://foo:[email protected]:5672", :vhost=>"/", :exchange=>"sneakers", :exchange_type=>:dir
ect, :hooks=>{}, :handler=>Sneakers::Handlers::Oneshot}] Stopping worker: I'm gone.
2014-04-13T22:59:48Z p-23076 t-os3q5m354 DEBUG: [worker-store2:1:fm59b2][#<ServerEngine::SignalThread:0x007f29e3
d16cd0>][store2][{:runner_config_file=>nil, :metrics=>nil, :daemonize=>false, :start_worker_delay=>0.2, :workers
=>4, :log=>#<IO:<STDOUT>>, :pid_path=>"sneakers.pid", :timeout_job_after=>30, :prefetch=>15, :threads=>15, :durable=>tr
ue, :ack=>true, :heartbeat=>2, :amqp=>"amqp://foo:[email protected]:5672", :vhost=>"/", :exchange=>"sneakers", :exchange_type=>:direct, :ho
oks=>{}, :handler=>Sneakers::Handlers::Oneshot}] Stopping worker: unsubscribing.
Unexpected error Trying to send frame through a closed connection. Frame is #<AMQ::Protocol::MethodFrame:0x007f29ac0731
18 @payload="\x00<\x00\x1E bunny-1397429831000-559756359092\x00", @channel=2>, method class is AMQ::Protocol::Basic::Ca
ncel
  /home/vagrant/.rbenv/versions/2.0.0-p451/lib/ruby/gems/2.0.0/gems/bunny-1.1.9/lib/bunny/session.rb:846:in `send_frame
'
  /home/vagrant/.rbenv/versions/2.0.0-p451/lib/ruby/gems/2.0.0/gems/bunny-1.1.9/lib/bunny/channel.rb:917:in `basic_canc
el'
  /home/vagrant/.rbenv/versions/2.0.0-p451/lib/ruby/gems/2.0.0/gems/bunny-1.1.9/lib/bunny/consumer.rb:84:in `cancel'
  /home/vagrant/.rbenv/versions/2.0.0-p451/lib/ruby/gems/2.0.0/gems/sneakers-0.1.1.pre/lib/sneakers/queue.rb:48:in `uns
ubscribe'
  /home/vagrant/.rbenv/versions/2.0.0-p451/lib/ruby/gems/2.0.0/gems/sneakers-0.1.1.pre/lib/sneakers/worker.rb:93:in `st
op'
  /home/vagrant/.rbenv/versions/2.0.0-p451/lib/ruby/gems/2.0.0/gems/sneakers-0.1.1.pre/lib/sneakers/workergroup.rb:40:i
n `block in stop'
  /home/vagrant/.rbenv/versions/2.0.0-p451/lib/ruby/gems/2.0.0/gems/sneakers-0.1.1.pre/lib/sneakers/workergroup.rb:39:i
n `each'
  /home/vagrant/.rbenv/versions/2.0.0-p451/lib/ruby/gems/2.0.0/gems/sneakers-0.1.1.pre/lib/sneakers/workergroup.rb:39:i
n `stop'
  /home/vagrant/.rbenv/versions/2.0.0-p451/lib/ruby/gems/2.0.0/gems/serverengine-1.5.7/lib/serverengine/worker.rb:60:in
 `block (2 levels) in install_signal_handlers'
  /home/vagrant/.rbenv/versions/2.0.0-p451/lib/ruby/gems/2.0.0/gems/serverengine-1.5.7/lib/serverengine/signal_thread.r
b:98:in `call'
  /home/vagrant/.rbenv/versions/2.0.0-p451/lib/ruby/gems/2.0.0/gems/serverengine-1.5.7/lib/serverengine/signal_thread.r
b:98:in `main'
^C2014-04-13T22:59:48Z p-23076 t-os3pt2j64 INFO: Heartbeat: running threads [54]
2014-04-13T22:59:48Z p-23076 t-os3q5m354 INFO: Shutting down workers
2014-04-13T22:59:48Z p-23076 t-os3q5m354 DEBUG: [worker-store1:1:hhkhcb][#<ServerEngine::SignalThread:0x007f
29e3d16cd0>][store1][{:runner_config_file=>nil, :metrics=>nil, :daemonize=>false, :start_worker_delay=>0.2, 
:workers=>4, :log=>#<IO:<STDOUT>>, :pid_path=>"sneakers.pid", :timeout_job_after=>30, :prefetch=>15, :threads=>15, :dur
able=>true, :ack=>true, :heartbeat=>2, :amqp=>"amqp://foo:[email protected]:5672", :vhost=>"/", :exchange=>"sneakers", :exchange_type=>:dir
ect, :hooks=>{}, :handler=>Sneakers::Handlers::Oneshot}] Stopping worker: unsubscribing.
2014-04-13T22:59:48Z p-23076 t-os3q5m354 DEBUG: [worker-store1:1:hhkhcb][#<ServerEngine::SignalThread:0x007f
29e3d16cd0>][store1][{:runner_config_file=>nil, :metrics=>nil, :daemonize=>false, :start_worker_delay=>0.2, 
:workers=>4, :log=>#<IO:<STDOUT>>, :pid_path=>"sneakers.pid", :timeout_job_after=>30, :prefetch=>15, :threads=>15, :dur
able=>true, :ack=>true, :heartbeat=>2, :amqp=>"amqp://foo:[email protected]:5672", :vhost=>"/", :exchange=>"sneakers", :exchange_type=>:dir
ect, :hooks=>{}, :handler=>Sneakers::Handlers::Oneshot}] Stopping worker: I'm gone.
2014-04-13T22:59:48Z p-23076 t-os3q5m354 DEBUG: [worker-store2:1:fm59b2][#<ServerEngine::SignalThread:0x007f29e3
d16cd0>][store2][{:runner_config_file=>nil, :metrics=>nil, :daemonize=>false, :start_worker_delay=>0.2, :workers
=>4, :log=>#<IO:<STDOUT>>, :pid_path=>"sneakers.pid", :timeout_job_after=>30, :prefetch=>15, :threads=>15, :durable=>tr
ue, :ack=>true, :heartbeat=>2, :amqp=>"amqp://foo:[email protected]:5672", :vhost=>"/", :exchange=>"sneakers", :exchange_type=>:direct, :ho
oks=>{}, :handler=>Sneakers::Handlers::Oneshot}] Stopping worker: unsubscribing.
Unexpected error Trying to send frame through a closed connection. Frame is #<AMQ::Protocol::MethodFrame:0x007f29ac0796
08 @payload="\x00<\x00\x1E bunny-1397429831000-559756359092\x00", @channel=2>, method class is AMQ::Protocol::Basic::Ca
ncel
  /home/vagrant/.rbenv/versions/2.0.0-p451/lib/ruby/gems/2.0.0/gems/bunny-1.1.9/lib/bunny/session.rb:846:in `send_frame
'
  /home/vagrant/.rbenv/versions/2.0.0-p451/lib/ruby/gems/2.0.0/gems/bunny-1.1.9/lib/bunny/channel.rb:917:in `basic_canc
el'
  /home/vagrant/.rbenv/versions/2.0.0-p451/lib/ruby/gems/2.0.0/gems/bunny-1.1.9/lib/bunny/consumer.rb:84:in `cancel'
  /home/vagrant/.rbenv/versions/2.0.0-p451/lib/ruby/gems/2.0.0/gems/sneakers-0.1.1.pre/lib/sneakers/queue.rb:48:in `uns
ubscribe'
  /home/vagrant/.rbenv/versions/2.0.0-p451/lib/ruby/gems/2.0.0/gems/sneakers-0.1.1.pre/lib/sneakers/worker.rb:93:in `st
op'
  /home/vagrant/.rbenv/versions/2.0.0-p451/lib/ruby/gems/2.0.0/gems/sneakers-0.1.1.pre/lib/sneakers/workergroup.rb:40:i
n `block in stop'
  /home/vagrant/.rbenv/versions/2.0.0-p451/lib/ruby/gems/2.0.0/gems/sneakers-0.1.1.pre/lib/sneakers/workergroup.rb:39:i
n `each'
  /home/vagrant/.rbenv/versions/2.0.0-p451/lib/ruby/gems/2.0.0/gems/sneakers-0.1.1.pre/lib/sneakers/workergroup.rb:39:i
n `stop'
  /home/vagrant/.rbenv/versions/2.0.0-p451/lib/ruby/gems/2.0.0/gems/serverengine-1.5.7/lib/serverengine/worker.rb:60:in
 `block (2 levels) in install_signal_handlers'
  /home/vagrant/.rbenv/versions/2.0.0-p451/lib/ruby/gems/2.0.0/gems/serverengine-1.5.7/lib/serverengine/signal_thread.r
b:98:in `call'
  /home/vagrant/.rbenv/versions/2.0.0-p451/lib/ruby/gems/2.0.0/gems/serverengine-1.5.7/lib/serverengine/signal_thread.r
b:98:in `main'
^Z
[1]+  Stopped                 rake sneakers:run
vagrant@precise-base:/vagrant$ ps aux | grep rake
vagrant  22926  2.6  0.4 1224600 32580 pts/1   Tl   22:57   0:04 ruby /home/vagrant/.rbenv/versions/2.0.0-p451/bin/rake
 sneakers:run
vagrant  23076  5.8  2.1 3080344 176568 pts/1  Tl   22:57   0:09 ruby /home/vagrant/.rbenv/versions/2.0.0-p451/bin/rake
 sneakers:run
vagrant  26499  0.0  0.0   8112   932 pts/1    S+   22:59   0:00 grep rake
vagrant@precise-base:/vagrant$ kill 22926
vagrant@precise-base:/vagrant$ kill 23076
vagrant@precise-base:/vagrant$ ps aux | grep rake
vagrant  22926  2.3  0.4 1224600 32584 pts/1   Tl   22:57   0:04 ruby /home/vagrant/.rbenv/versions/2.0.0-p451/bin/rake
 sneakers:run
vagrant  23076  5.1  2.1 3080344 176572 pts/1  Tl   22:57   0:09 ruby /home/vagrant/.rbenv/versions/2.0.0-p451/bin/rake
 sneakers:run
vagrant@precise-base:/vagrant$ kill -9 22926
vagrant@precise-base:/vagrant$ kill -9 23076
-bash: kill: (23076) - No such process
[1]+  Killed                  rake sneakers:run

git tag releases

Please tag your releases as you push them to ruby gems so consumers of your gem can use github to diff between releases.

Lost message with multiprocess？

Run this [WORKERS=TradeWorker rake sneakers:run] command first.
the work coding :

class TradeWorker
    QUEUENAME = "TradeWorker"
    include Sneakers::Worker
    from_queue QUEUENAME,
        # :workers => 1, # 进程数，默认为4
        :durable => true,
        :ack => true,
        :threads => 50, # default is 10
        :prefetch => 50, # A good practice is to set up a prefetch policy against RabbitMQ of at least the amount of threads involved
        :start_worker_delay => 0 # 默认0.2
        # :exchange => QUEUENAME
        # :timeout_job_after => 10

    def work(msg)
        t = Time.now.to_f
        a,b = msg.split(":")
        out = {
            :id => a,
            :push => b,
            :pull => t,
            :cost_time => t - b.to_f
        }
        Rails.logger.info out

        # Trade::Trade.tb_trade_create(id) unless id.blank?
        ack! #  表示 处理结束
    end 
end

then publish 10000 messages into the queue, used :

10000.times do |n|
     Sneakers.publish("#{n}:#{Time.now.to_f}", :to_queue => TradeWorker::QUEUENAME)
end

the lost message problem happend!

there is no problem when publish 10000 messages into the queue,used

conn = Bunny.new#("amqp://guest:guest@localhost:5672")
conn.start
channel  = conn.create_channel
queue  = channel.queue(TradeWorker::QUEUENAME, :durable => true)
10000.times do |n|
     # Sneakers.publish("#{n}:#{Time.now.to_f}", :to_queue => TradeWorker::QUEUENAME)
    queue.publish("#{n}:#{Time.now.to_f}", :routing_key => TradeWorker::QUEUENAME, persistent: true)
end
channel.close

there is also no problem when set the Sneakers.configure workers to 1.

changed the open source [~/.rvm/gems/ruby-2.0.0-p451/gems/sneakers-0.0.7/lib/sneakers/queue.rb]
into:

def subscribe(worker)
    @bunny = Bunny.new(@opts[:amqp], :vhost => @opts[:vhost], :heartbeat => @opts[:heartbeat])
    @bunny.start

    @channel = @bunny.create_channel
    @channel.prefetch(@opts[:prefetch])

    @exchange = @channel.exchange(@opts[:exchange],
                                  :type => @opts[:exchange_type],
                                  :durable => @opts[:durable])

    handler = @handler_klass.new(@channel)

    queue = @channel.queue(@name, :durable => @opts[:durable])
    queue.bind(@exchange, :routing_key => @name)

    @consumer = queue.subscribe(:block => false, :ack => @opts[:ack]) do | hdr, props, msg |
      handler.acknowledge(hdr.delivery_tag) ######## just receive msg,don't do worker ############ 
      # worker.do_work(hdr, props, msg, handler)
    end
    nil
end

the lost message problem alse happend!

Intermittent spec failures

When running the Sneakers spec suite, I get random spec failures every 5 or 6 times I it. Here's a sample:

Finished in 1.378827s, 36.2627 runs/s, 34.0869 assertions/s.

  1) Failure:
Sneakers::Worker::#initialize::builds an internal queue#test_0001_should build a queue with correct configuration given defaults [/Users/norman/work/flexminder/sneakers/spec/sneakers/worker_spec.rb:170]:
--- expected
+++ actual
@@ -1 +1 @@
-{:runner_config_file=>nil, :metrics=>nil, :daemonize=>true, :start_worker_delay=>0.2, :workers=>4, :log=>"sneakers.log", :pid_path=>"sneakers.pid", :timeout_job_after=>5, :prefetch=>10, :threads=>10, :durable=>true, :ack=>true, :amqp=>"amqp://guest:guest@localhost:5672", :vhost=>"/", :exchange=>"sneakers", :exchange_type=>:direct, :hooks=>{}, :handler=>Sneakers::Handlers::Oneshot, :heartbeat=>2}
+{:runner_config_file=>nil, :metrics=>nil, :daemonize=>true, :start_worker_delay=>0.2, :workers=>4, :log=>"sneakers.log", :pid_path=>"sneakers.pid", :timeout_job_after=>5, :prefetch=>10, :threads=>10, :durable=>false, :ack=>true, :heartbeat=>1, :amqp=>"amqp://someuser:somepassword@somehost:5672", :vhost=>"/", :exchange=>"another_exchange", :exchange_type=>:topic, :hooks=>{}, :handler=>Sneakers::Handlers::Oneshot}



  2) Failure:
Sneakers::Worker::#initialize::builds an internal queue#test_0002_should build a queue with given configuration [/Users/norman/work/flexminder/sneakers/spec/sneakers/worker_spec.rb:177]:
--- expected
+++ actual
@@ -1 +1 @@
-{:runner_config_file=>nil, :metrics=>nil, :daemonize=>true, :start_worker_delay=>0.2, :workers=>4, :log=>"sneakers.log", :pid_path=>"sneakers.pid", :timeout_job_after=>1, :prefetch=>40, :threads=>50, :durable=>false, :ack=>false, :amqp=>"amqp://guest:guest@localhost:5672", :vhost=>"/", :exchange=>"dummy", :exchange_type=>:direct, :hooks=>{}, :handler=>Sneakers::Handlers::Oneshot, :heartbeat=>5}
+{:runner_config_file=>nil, :metrics=>nil, :daemonize=>true, :start_worker_delay=>0.2, :workers=>4, :log=>"sneakers.log", :pid_path=>"sneakers.pid", :timeout_job_after=>1, :prefetch=>40, :threads=>50, :durable=>false, :ack=>false, :heartbeat=>5, :amqp=>"amqp://someuser:somepassword@somehost:5672", :vhost=>"/", :exchange=>"dummy", :exchange_type=>:topic, :hooks=>{}, :handler=>Sneakers::Handlers::Oneshot}


50 runs, 47 assertions, 2 failures, 0 errors, 0 skips
Coverage report generated for RSpec to /Users/norman/work/flexminder/sneakers/coverage. 711 / 780 LOC (91.15%) covered.
rake aborted!
Command failed with status (1): [ruby -I"lib:spec" -I"/Users/norman/.rbenv/versions/2.1.2/lib/ruby/gems/2.1.0/gems/rake-10.3.2/lib" "/Users/norman/.rbenv/versions/2.1.2/lib/ruby/gems/2.1.0/gems/rake-10.3.2/lib/rake/rake_test_loader.rb" "spec/sneakers/cli_spec.rb" "spec/sneakers/concerns/logging_spec.rb" "spec/sneakers/concerns/metrics_spec.rb" "spec/sneakers/publisher_spec.rb" "spec/sneakers/queue_spec.rb" "spec/sneakers/sneakers_spec.rb" "spec/sneakers/support/utils_spec.rb" "spec/sneakers/worker_spec.rb" ]

I'll look into it further on my end but I wanted to log the issue to track it in case anybody else is seeing it.

Sneakers 1.0

I think it's safe to say: Sneakers is production grade.

Goals

At all of my production deployments, Sneakers outlived and far exceeded it's original goal. On of them, it's been almost 2 years, 4 BILLION jobs processed, supporting 5 completely different use cases.

Contributions

I'm humbled that a lot of smart people contributed to this project and major companies adopted it. All I hope for, is that it has saved you time, and made your job easier and more fun to do.

That it solved a problem for you.

With that in mind - I'd like to share with you the news, and what's coming.

Where we're going

Rebranding [WIP] - hired a design studio, work is in progress.
New front page, new look.
Blog [WIP] - The main purpose is community updates. Tutorials, and use cases will be uploaded and you'll be able to see how everyone is using Sneakers out there.
Guest appearances - any of you guys

All the additional work I've taken upon myself to do is not a step for commercialization like other open-source libraries may have done in the past. It's a way for me to give back a big Thank You ❤️ for your contributions and adoption of Sneakers.

I believe open-source should stay the same open-source it was back in 99', when I contributed my first project: free, and free of hidden interests.

I hope that Sneakers will always be free, friendly, accessible, and fast :).

Towards 1.0!

I'll be keeping this issue updated along the upcoming couple of weeks.

I'm the meanwhile, here's a thanks to each of you (by no special order) that contributed, provided their feedback and thoughts over pull requests and issues:

@justin-yesware @trungpham @Baltazore @arion @norman @arielze @michaelklishin @imothee @timgluz @tychobrailleur @SebastianEdwards @jaresty @abevoelker @rodrigosaito @jpemberthy @lkang @adamniedzielski @carlhoerberg @sicarrots @magnum @anhuiliujun @simonmorley @nathanpalmer @stevenkaras @bobbytables @doughsay @LukeWinikates @ged @MaxGIS @MadBomber @glenjamin @kmayer @wangli3274 @kke @msaspence @graf-abolmasov @yaauie @jgwmaxwell

Be able to provide full connection url instead of separate :amqp and :vhost

I've faced quite odd behaviour of Sneakers when I've deployed application on Heroku.

Heroku's RabbitMQ Addon provides full amqp url like amqp://login:pass@host:port/vhost, and pure Bunny do understand this string easily. However, due to https://github.com/jondot/sneakers/blob/master/lib/sneakers/queue.rb#L19 I have to parse this line, extract vhost and provide it for Sneakers manually.

Here my suggestion.
Bunny.new method supports either connection_string or hash with options, so it would be nice to leverage this feature and pass Sneakers::Config as it is. So, everyone who likes to pass :ampq and :vhost separately would be able to do that, on the other hand it would be easy to deploy on Heroku platform.

`:ack` is deprecated. Please use `:manual_ack` instead

https://github.com/ruby-amqp/bunny/blob/master/lib/bunny/queue.rb#L169

Using 0.10.0 w/ Bunny 1.6.0:
[DEPRECATION] :ack is deprecated. Please use :manual_ack instead.

Publishing tasks?

I'm not sure if I'm just being dense here, but I can't seem to see any docs or examples of publishing messages into Rabbit, only worker docs.

Are publishers outside the scope of sneakers?

pid missing when spawning by worker group

I'm trying to use the new spawner feature to handling different workloads.

I have two workers which are split into two groups:

node_workers:
  classes: NodeActionsWorker
  workers: 2
job_workers:
  classes: JobActionsWorker
  workers: 1

I start the ruby process to spawn workers

bundle exec ruby -e "require 'sneakers/spawner';Sneakers::Spawner.spawn"

The processes list below

ps -ef | grep sneakers
  501 24223     1   0  2:26PM ??         0:00.17 ruby /Users/twer/Workspace/ScaleWorks/vendor/bundle/ruby/1.9.1/bin/rake sneakers:run
  501 24225     1   0  2:26PM ??         0:00.20 ruby /Users/twer/Workspace/ScaleWorks/vendor/bundle/ruby/1.9.1/bin/rake sneakers:run
  501 24226 24223   0  2:26PM ??         0:00.19 ruby /Users/twer/Workspace/ScaleWorks/vendor/bundle/ruby/1.9.1/bin/rake sneakers:run
  501 24227 24225   0  2:26PM ??         0:00.19 ruby /Users/twer/Workspace/ScaleWorks/vendor/bundle/ruby/1.9.1/bin/rake sneakers:run
  501 24232 24225   0  2:26PM ??         0:00.14 ruby /Users/twer/Workspace/ScaleWorks/vendor/bundle/ruby/1.9.1/bin/rake sneakers:run

So far so good, but when I cat the pids, I found the one of the pids is missing:

cat tmp/pids/sneakers.pid
24225

Then I realize that the two work group share one global config:

# in config/initializers/sneaker.rb

Sneakers.configure  :pid_path => 'tmp/pids/sneakers.pid', 
                                  ##other config items

I try to config pid_path within each worker, but it does not work.

Stop daemonize worker

How can we stop a worker running in background ?

Feedback Required: Best way to do integration testing

Hi all,
I'd like to brainstorm the various ways we've been doing integration tests with Sneakers.
First let's define integration test to be one of these

A requirement to interact with some AMQP broker.
A requirement to interact with RabbitMQ specifically.
A requirement to go out of Sneaker's sneaker-bunny API surface area boundary.

My initial ideas:
For (3)

Use mock gems like https://github.com/rentpath/bunny_mock
Create and use a sneakers/testing library which mocks out all of the surface areas of sneakers<->bunny from the sneakers side.

For (2)

Use a RabbitMQ management gem to CRUD queues and exchanges
Need some sort of bullet-proof process control if we spawn a Sneakers worker

For (1)

All of the methods in (2) that were suggested
Build, or find an in-memory implementation of an AMQP broker that can be easily and programmatically controlled. Doesn't have to be implemented in Ruby.

I'm wondering what do you all think? perhaps @rud @michaelklishin @norman have direct feedback on this?

Thanks :)

Where are the tags that correspond to released RubyGems versions?

Most Ruby projects that release to RubyGems push tags that correspond to that version, e.g. a tag of v0.0.7 corresponds to the released gem version 0.0.7. It's the default workflow when you release gems with Bundler's rake release task.

Not saying you need to follow this workflow, but it is really handy to have tags that match the RubyGems version for reference.

For instance, with the patch I just released, it would be nice if I could rebase it on top of whatever commit corresponds to RubyGems version 0.0.7 in my private repo, so that I'm not riding master, which I always assume is a bit more unstable. But due to the lack of release tags I can't do that quickly, since I don't know which git commit corresponds to the RubyGems 0.0.7 version.

Using with God or Runit

I've push moved a big project over from Hutch to Sneakers. Happier days are ahead I can see :) This is a much lighter and more flexible project.

All is well, however, we are struggling to get things to pay nice with either runit or god.

God in particular seems unfriendly, I'm not sure why.

I've created a god config. file as so:

God.watch do |w|
  w.name = "sneakers"
  w.dir = "/var/www/html/cucumbertony/production/current/"
  w.start = "RAILS_ENV=production bundle exec WORKERS=MyWorkers,AnotherWorker rake sneakers:run"
  w.keepalive
end

It looks like it starts and I can see the connections in RabbitMQ. But on closer inspection, I can see it's exiting and a new pid is created.

I've tried to force a pid_file in the sneakers config AND the god config but this doesn't help either.

The god log shows the following over and over and over and over.

I [2014-07-29 23:19:39]  INFO: sneakers moved 'up' to 'up'
I [2014-07-29 23:19:44]  INFO: sneakers [trigger] process is not running (ProcessRunning)
I [2014-07-29 23:19:44]  INFO: sneakers move 'up' to 'start'

Can you suggest something I should / could do to facilitate running this with god?

Performance is misleading

Your Performance wiki page is misleading. Benchmarking noops isn't anything close to realistic or useful to potential users. Please provide a real benchmark where each message requires some real processing (hits a database, performs some calculation, etc) and I think you'll find that Sneakers doesn't process jobs any faster than Sidekiq.

Errno::ENOMEM: Cannot allocate memory - fork

During testing on a Debian machine, whilst running sneakers, I'm continuously presented with the error:

Errno::ENOMEM: Cannot allocate memory - fork

My CPU and memory usage are through the roof. I've dropped the threads and workers to 1 a piece.

However, over time the machines die a very slow and painful death.

Is this considered production ready?

Recommend Projects

React

A declarative, efficient, and flexible JavaScript library for building user interfaces.
Vue.js

🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript

TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
TensorFlow

An Open Source Machine Learning Framework for Everyone
Django

The Web framework for perfectionists with deadlines.
Laravel

A PHP framework for web artisans
D3

Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

javascript

JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
web

Some thing interesting about web. New door for the world.
server

A server is a program made to process requests and deliver data to clients.
Machine learning

Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Visualization

Some thing interesting about visualization, use data art
Game

Some thing interesting about game, make everyone happy.

Recommend Org

Facebook

We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft

Open source projects and samples from Microsoft.
Google

Google ❤️ Open Source for everyone.
Alibaba

Alibaba Open Source for everyone
D3

Data-Driven Documents codes.
Tencent

China tencent open source team.