Git Product home page Git Product logo

Comments (6)

JustinLove avatar JustinLove commented on June 16, 2024 1

Do you suggest increasing worker_capacity? What would be a good way to estimate how to configure it?

The default was 25, I noticed checking up sidekiq that they changed the default threads per worker to 10 at some point. You could probably go larger than that depending on your needs - since you are using autoscaler I assume the incoming task rate is not too large. The default strategy was 1/0 because I expected a single worker to be able to eventually handle everything with intermittent task additions.

I'd guess estimation would come down to how long tasks take, and how long is "too long". So if the threads per worker is 10, tasks take 2 seconds, and you want them to resolve in no more than 10 seconds, capacity could be 50.

from autoscaler.

JustinLove avatar JustinLove commented on June 16, 2024

How often does this occur? Is it at any particular time? - you referenced a shutdown issue. Are any other errors reported?

Best case would be a fork of the sample project that demonstrates the error.

from autoscaler.

echan00 avatar echan00 commented on June 16, 2024

@JustinLove the error occurred 3 times within a span of 30 minutes. They were all under normal operations in my production environment.

Forking a sample project will be tough, figuring out how to replicate the error will be even tougher. But I'll look out for this error some more.

Below is the full traceback:

**ThreadError: can't alloc thread**
File /app/vendor/bundle/ruby/2.5.0/gems/autoscaler-1.0.0/lib/autoscaler/sidekiq/thread_server.rb line 52 in new
File /app/vendor/bundle/ruby/2.5.0/gems/autoscaler-1.0.0/lib/autoscaler/sidekiq/thread_server.rb line 52 in wait_for_downscale
File /app/vendor/bundle/ruby/2.5.0/gems/autoscaler-1.0.0/lib/autoscaler/sidekiq/thread_server.rb line 26 in ensure in call
File /app/vendor/bundle/ruby/2.5.0/gems/autoscaler-1.0.0/lib/autoscaler/sidekiq/thread_server.rb line 26 in call
File /app/vendor/bundle/ruby/2.5.0/gems/sidekiq-5.2.5/lib/sidekiq/middleware/chain.rb line 130 in block in invoke
File /app/vendor/bundle/ruby/2.5.0/gems/rollbar-2.18.2/lib/rollbar/plugins/sidekiq/plugin.rb line 11 in call
File /app/vendor/bundle/ruby/2.5.0/gems/sidekiq-5.2.5/lib/sidekiq/middleware/chain.rb line 130 in block in invoke
File /app/vendor/bundle/ruby/2.5.0/gems/scout_apm-2.4.21/lib/scout_apm/background_job_integrations/sidekiq.rb line 68 in call
File /app/vendor/bundle/ruby/2.5.0/gems/sidekiq-5.2.5/lib/sidekiq/middleware/chain.rb line 130 in block in invoke
File /app/vendor/bundle/ruby/2.5.0/gems/sidekiq-5.2.5/lib/sidekiq/middleware/chain.rb line 133 in invoke
File /app/vendor/bundle/ruby/2.5.0/gems/sidekiq-5.2.5/lib/sidekiq/processor.rb line 166 in block in process
File /app/vendor/bundle/ruby/2.5.0/gems/sidekiq-5.2.5/lib/sidekiq/processor.rb line 137 in block (6 levels) in dispatch
File /app/vendor/bundle/ruby/2.5.0/gems/sidekiq-5.2.5/lib/sidekiq/job_retry.rb line 108 in local
File /app/vendor/bundle/ruby/2.5.0/gems/sidekiq-5.2.5/lib/sidekiq/processor.rb line 136 in block (5 levels) in dispatch
File /app/vendor/bundle/ruby/2.5.0/gems/sidekiq-5.2.5/lib/sidekiq/rails.rb line 43 in block in call
File /app/vendor/bundle/ruby/2.5.0/gems/activesupport-5.1.6/lib/active_support/execution_wrapper.rb line 85 in wrap
File /app/vendor/bundle/ruby/2.5.0/gems/activesupport-5.1.6/lib/active_support/reloader.rb line 68 in block in wrap
File /app/vendor/bundle/ruby/2.5.0/gems/activesupport-5.1.6/lib/active_support/execution_wrapper.rb line 85 in wrap
File /app/vendor/bundle/ruby/2.5.0/gems/activesupport-5.1.6/lib/active_support/reloader.rb line 67 in wrap
File /app/vendor/bundle/ruby/2.5.0/gems/sidekiq-5.2.5/lib/sidekiq/rails.rb line 42 in call
File /app/vendor/bundle/ruby/2.5.0/gems/sidekiq-5.2.5/lib/sidekiq/processor.rb line 132 in block (4 levels) in dispatch
File /app/vendor/bundle/ruby/2.5.0/gems/sidekiq-5.2.5/lib/sidekiq/processor.rb line 243 in stats
File /app/vendor/bundle/ruby/2.5.0/gems/sidekiq-5.2.5/lib/sidekiq/processor.rb line 127 in block (3 levels) in dispatch
File /app/vendor/bundle/ruby/2.5.0/gems/sidekiq-5.2.5/lib/sidekiq/job_logger.rb line 8 in call
File /app/vendor/bundle/ruby/2.5.0/gems/sidekiq-5.2.5/lib/sidekiq/processor.rb line 126 in block (2 levels) in dispatch
File /app/vendor/bundle/ruby/2.5.0/gems/sidekiq-5.2.5/lib/sidekiq/job_retry.rb line 73 in global
File /app/vendor/bundle/ruby/2.5.0/gems/sidekiq-5.2.5/lib/sidekiq/processor.rb line 125 in block in dispatch
File /app/vendor/bundle/ruby/2.5.0/gems/sidekiq-5.2.5/lib/sidekiq/logging.rb line 48 in with_context
File /app/vendor/bundle/ruby/2.5.0/gems/sidekiq-5.2.5/lib/sidekiq/logging.rb line 42 in with_job_hash_context
File /app/vendor/bundle/ruby/2.5.0/gems/sidekiq-5.2.5/lib/sidekiq/processor.rb line 124 in dispatch
File /app/vendor/bundle/ruby/2.5.0/gems/sidekiq-5.2.5/lib/sidekiq/processor.rb line 165 in process
File /app/vendor/bundle/ruby/2.5.0/gems/sidekiq-5.2.5/lib/sidekiq/processor.rb line 83 in process_one
File /app/vendor/bundle/ruby/2.5.0/gems/sidekiq-5.2.5/lib/sidekiq/processor.rb line 71 in run
File /app/vendor/bundle/ruby/2.5.0/gems/sidekiq-5.2.5/lib/sidekiq/util.rb line 16 in watchdog
File /app/vendor/bundle/ruby/2.5.0/gems/sidekiq-5.2.5/lib/sidekiq/util.rb line 25 in block in safe_thread

from autoscaler.

JustinLove avatar JustinLove commented on June 16, 2024

My problem is I don't really have a sense of what is going on when the error occurs. Would you be able to post some cleaned logs with event sequences for jobs submitted/finished, and scale up/down? Failing that what kind of job patterns do you have?

One thing I noticed is that you were doing one worker per queued job, whereas default sidekiq is set up for multiple threads per worker. Even if you've got it configured for one thread per worker, it seems like that could have a lot of scaling churn.

from autoscaler.

echan00 avatar echan00 commented on June 16, 2024

I agree with you. It is tough to debug this issue without being able to replicate it. I'll monitor for this error, I imagine it will come up again.

One thing I noticed is that you were doing one worker per queued job, whereas default sidekiq is set up for multiple threads per worker. Even if you've got it configured for one thread per worker, it seems like that could have a lot of scaling churn.

Do you suggest increasing worker_capacity? What would be a good way to estimate how to configure it?

from autoscaler.

echan00 avatar echan00 commented on June 16, 2024

Sorry for the long wait, finally back looking at this.

I believe the following is related to the error:

02:08:12 app/worker.3: 4 TID-gn0jk9kdo DocTranslateWorker JID-fae072ecfaa5c5cde69b4d0f INFO: start
02:08:15 heroku/worker.3: State changed from up to down

Worker 3 is has been assigned to start on a job, but immediately after the heroku scaler decides to scale that worker down and causes the error. It is likely related to the fact that worker_capacity: 1 and like you mentioned there is a lot worker churn because the jobs come in short bursts requiring starting multiple workers at a time (but none needed when not needed).

Full log of worker 3:

02:07:52 heroku/worker.3: Starting process with command `bundle exec sidekiq -C config/sidekiq.yml`
02:07:53 heroku/worker.3: State changed from starting to up
02:08:09 app/worker.3: 4 TID-gn0imq158 INFO: Booting Sidekiq 5.2.5 with redis options {:url=>"redis://h:[email protected]:10199", :size=>3, :id=>"Sidekiq-server-PID-4"}
02:08:09 app/worker.3: "Setting up auto-scaledown"
02:08:12 app/worker.3: 4 TID-gn0imq158 INFO: Running in ruby 2.5.0p0 (2017-12-25 revision 61468) [x86_64-linux]
02:08:12 app/worker.3: 4 TID-gn0imq158 INFO: Starting processing, hit Ctrl-C to stop
02:08:12 app/worker.3: 4 TID-gn0jk9kdo DocTranslateWorker JID-fae072ecfaa5c5cde69b4d0f INFO: start
02:08:15 heroku/worker.3: State changed from up to down
02:08:17 heroku/worker.3: Stopping all processes with SIGTERM
02:08:17 app/worker.3: 4 TID-gn0imq158 INFO: Shutting down
02:08:17 app/worker.3: 4 TID-gn0imq158 INFO: Terminating quiet workers
02:08:17 app/worker.3: 4 TID-gn0jk9yiw INFO: Scheduler exiting...
02:08:17 app/worker.3: 4 TID-gn0imq158 INFO: Pausing to allow workers to finish...
02:08:24 app/worker.3: 4 TID-gn0imq158 WARN: Terminating 1 busy worker threads
02:08:24 app/worker.3: 4 TID-gn0imq158 WARN: Work still in progress [#<struct Sidekiq::BasicFetch::UnitOfWork queue="queue:default", job="{\"class\":\"DocTranslateWorker\",\"args\":[396,3],\"retry\":1,\"queue\":\"default\",\"backtrace\":20,\"jid\":\"fae072ecfaa5c5cde69b4d0f\",\"created_at\":1572948426.9953856,\"enqueued_at\":1572948426.9969578}">]
02:08:24 app/worker.3: 4 TID-gn0imq158 INFO: Pushed 1 jobs back to Redis
02:08:24 app/worker.3: 4 TID-gn0imq158 INFO: Bye!
02:08:24 app/worker.3: 4 TID-gn0jk9kdo DocTranslateWorker JID-fae072ecfaa5c5cde69b4d0f INFO: fail: 12.772 sec
02:08:24 app/worker.3: 4 TID-gn0jk9kdo WARN: {"context":"Job raised exception","job":{"class":"DocTranslateWorker","args":[396,3],"retry":1,"queue":"default","backtrace":20,"jid":"fae072ecfaa5c5cde69b4d0f","created_at":1572948426.9953856,"enqueued_at":1572948426.9969578},"jobstr":"{\"class\":\"DocTranslateWorker\",\"args\":[396,3],\"retry\":1,\"queue\":\"default\",\"backtrace\":20,\"jid\":\"fae072ecfaa5c5cde69b4d0f\",\"created_at\":1572948426.9953856,\"enqueued_at\":1572948426.9969578}"}
02:08:24 app/worker.3: 4 TID-gn0jk9kdo WARN: ThreadError: can't alloc thread
02:08:24 app/worker.3: 4 TID-gn0jk9kdo WARN: /app/vendor/bundle/ruby/2.5.0/gems/autoscaler-1.0.0/lib/autoscaler/sidekiq/thread_server.rb:52:in `new'
02:08:24 app/worker.3: /app/vendor/bundle/ruby/2.5.0/gems/autoscaler-1.0.0/lib/autoscaler/sidekiq/thread_server.rb:52:in `wait_for_downscale'
02:08:24 app/worker.3: /app/vendor/bundle/ruby/2.5.0/gems/autoscaler-1.0.0/lib/autoscaler/sidekiq/thread_server.rb:26:in `ensure in call'
02:08:24 app/worker.3: /app/vendor/bundle/ruby/2.5.0/gems/autoscaler-1.0.0/lib/autoscaler/sidekiq/thread_server.rb:26:in `call'
02:08:24 app/worker.3: /app/vendor/bundle/ruby/2.5.0/gems/sidekiq-5.2.5/lib/sidekiq/middleware/chain.rb:130:in `block in invoke'
02:08:24 app/worker.3: /app/vendor/bundle/ruby/2.5.0/gems/sidekiq-status-1.1.4/lib/sidekiq-status/server_middleware.rb:50:in `call'
02:08:24 app/worker.3: /app/vendor/bundle/ruby/2.5.0/gems/sidekiq-5.2.5/lib/sidekiq/middleware/chain.rb:130:in `block in invoke'
02:08:24 app/worker.3: /app/vendor/bundle/ruby/2.5.0/gems/rollbar-2.18.2/lib/rollbar/plugins/sidekiq/plugin.rb:11:in `call'
02:08:24 app/worker.3: /app/vendor/bundle/ruby/2.5.0/gems/sidekiq-5.2.5/lib/sidekiq/middleware/chain.rb:130:in `block in invoke'

from autoscaler.

Related Issues (20)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.