Exception handling when Heroku API call fails? about autoscaler HOT 14 CLOSED

Good point. I may have to review Exceptional Ruby for library behavior recommendations - if it just ate exceptions, you'd have unprocessed jobs backing up for no apparent reason.

At the moment, the scale happens before the yield, so the job won't be enqueued

from autoscaler.

hm, yikes, that seems problematic. seems like some of my jobs got through and some didn't. can't really make sense of it.

anyway, i figure it's better to have the jobs in the queue and no workers to process them than no jobs and no workers, knowwhatimean?

from autoscaler.

How's this?

39d5bd8

from autoscaler.

👍

from autoscaler.

Great, Heroku API is down today! Deploying w/ the github version to see what happens... Who would've thought we'd get a chance to test so soon.

EDIT: Sadly, no dice, can't deploy either...

from autoscaler.

I thought that heroku's tools used the API as well, although I had some hope that deploy was significant separate service in it's own right :-(

from autoscaler.

Any input on how to implement a custom handler correctly? I'm doing the following, but maybe there's a simpler/better way:

  exception_handler = ->(exception) do
    Honeybadger.notify(
      error_class: exception.class,
      error_message: exception.message
    )
  end

  Sidekiq.configure_server do |config|
    config.redis = sidekiq_redis if sidekiq_redis
    config.server_middleware do |chain|
      scaler = Autoscaler::HerokuScaler.new
      scaler.exception_handler = exception_handler
      chain.add(Autoscaler::Sidekiq::Server, scaler, 60)
    end
  end

from autoscaler.

That's about what I intended.

from autoscaler.

I've never been so excited to see Heroku API errors! Exceptions the last 15 minutes have been getting logged but jobs are still queuing. Yay!

Thank you, again.

from autoscaler.

How do you feel about rescuing Exception like sidekiq does rather than limiting it to Excon? I imagine other stuff could go wrong as well that we aren't thinking of ;)

https://github.com/mperham/sidekiq/blob/master/lib/sidekiq/processor.rb#L54

from autoscaler.

Mostly blanket exception catching makes me nervous about masking errors.

Sidekiq has a slightly different domain - it has to wrap arbitrary user code, and then respond appropriately (such as retry) One reason not to catch all exceptions is to let Sidekiq continue it's usual error handling.

from autoscaler.

True, but we wouldn't catch those, right? I don't think they would bubble up to Autoscaler... I'm in the same camp as you, but in situations like this where queuing is more important than performing, it just seems to make sense.

from autoscaler.

Another day, another Heroku API failure. Here's the latest exception that's now bubbling up to my app and causing sidekiq not to enqueue: Heroku::API::Errors::ErrorWithResponse: Expected(200) <=> Actual(503 Service Unavailable)

from autoscaler.

New idea: is it possible to handle the scaling after sidekiq has successfully enqueued the job? That way no matter what happens with autoscaler, we know jobs won't be lost...

from autoscaler.