Comments (6)
JustinLove
commented on June 16, 2024
1
Do you suggest increasing worker_capacity? What would be a good way to estimate how to configure it?
The default was 25, I noticed checking up sidekiq that they changed the default threads per worker to 10 at some point. You could probably go larger than that depending on your needs - since you are using autoscaler I assume the incoming task rate is not too large. The default strategy was 1/0 because I expected a single worker to be able to eventually handle everything with intermittent task additions.
I'd guess estimation would come down to how long tasks take, and how long is "too long". So if the threads per worker is 10, tasks take 2 seconds, and you want them to resolve in no more than 10 seconds, capacity could be 50.
from autoscaler.
JustinLove
commented on June 16, 2024
How often does this occur? Is it at any particular time? - you referenced a shutdown issue. Are any other errors reported?
Best case would be a fork of the sample project that demonstrates the error.
from autoscaler.
echan00
commented on June 16, 2024
@JustinLove the error occurred 3 times within a span of 30 minutes. They were all under normal operations in my production environment.
Forking a sample project will be tough, figuring out how to replicate the error will be even tougher. But I'll look out for this error some more.
Below is the full traceback:
**ThreadError: can't alloc thread**
File /app/vendor/bundle/ruby/2.5.0/gems/autoscaler-1.0.0/lib/autoscaler/sidekiq/thread_server.rb line 52 in new
File /app/vendor/bundle/ruby/2.5.0/gems/autoscaler-1.0.0/lib/autoscaler/sidekiq/thread_server.rb line 52 in wait_for_downscale
File /app/vendor/bundle/ruby/2.5.0/gems/autoscaler-1.0.0/lib/autoscaler/sidekiq/thread_server.rb line 26 in ensure in call
File /app/vendor/bundle/ruby/2.5.0/gems/autoscaler-1.0.0/lib/autoscaler/sidekiq/thread_server.rb line 26 in call
File /app/vendor/bundle/ruby/2.5.0/gems/sidekiq-5.2.5/lib/sidekiq/middleware/chain.rb line 130 in block in invoke
File /app/vendor/bundle/ruby/2.5.0/gems/rollbar-2.18.2/lib/rollbar/plugins/sidekiq/plugin.rb line 11 in call
File /app/vendor/bundle/ruby/2.5.0/gems/sidekiq-5.2.5/lib/sidekiq/middleware/chain.rb line 130 in block in invoke
File /app/vendor/bundle/ruby/2.5.0/gems/scout_apm-2.4.21/lib/scout_apm/background_job_integrations/sidekiq.rb line 68 in call
File /app/vendor/bundle/ruby/2.5.0/gems/sidekiq-5.2.5/lib/sidekiq/middleware/chain.rb line 130 in block in invoke
File /app/vendor/bundle/ruby/2.5.0/gems/sidekiq-5.2.5/lib/sidekiq/middleware/chain.rb line 133 in invoke
File /app/vendor/bundle/ruby/2.5.0/gems/sidekiq-5.2.5/lib/sidekiq/processor.rb line 166 in block in process
File /app/vendor/bundle/ruby/2.5.0/gems/sidekiq-5.2.5/lib/sidekiq/processor.rb line 137 in block (6 levels) in dispatch
File /app/vendor/bundle/ruby/2.5.0/gems/sidekiq-5.2.5/lib/sidekiq/job_retry.rb line 108 in local
File /app/vendor/bundle/ruby/2.5.0/gems/sidekiq-5.2.5/lib/sidekiq/processor.rb line 136 in block (5 levels) in dispatch
File /app/vendor/bundle/ruby/2.5.0/gems/sidekiq-5.2.5/lib/sidekiq/rails.rb line 43 in block in call
File /app/vendor/bundle/ruby/2.5.0/gems/activesupport-5.1.6/lib/active_support/execution_wrapper.rb line 85 in wrap
File /app/vendor/bundle/ruby/2.5.0/gems/activesupport-5.1.6/lib/active_support/reloader.rb line 68 in block in wrap
File /app/vendor/bundle/ruby/2.5.0/gems/activesupport-5.1.6/lib/active_support/execution_wrapper.rb line 85 in wrap
File /app/vendor/bundle/ruby/2.5.0/gems/activesupport-5.1.6/lib/active_support/reloader.rb line 67 in wrap
File /app/vendor/bundle/ruby/2.5.0/gems/sidekiq-5.2.5/lib/sidekiq/rails.rb line 42 in call
File /app/vendor/bundle/ruby/2.5.0/gems/sidekiq-5.2.5/lib/sidekiq/processor.rb line 132 in block (4 levels) in dispatch
File /app/vendor/bundle/ruby/2.5.0/gems/sidekiq-5.2.5/lib/sidekiq/processor.rb line 243 in stats
File /app/vendor/bundle/ruby/2.5.0/gems/sidekiq-5.2.5/lib/sidekiq/processor.rb line 127 in block (3 levels) in dispatch
File /app/vendor/bundle/ruby/2.5.0/gems/sidekiq-5.2.5/lib/sidekiq/job_logger.rb line 8 in call
File /app/vendor/bundle/ruby/2.5.0/gems/sidekiq-5.2.5/lib/sidekiq/processor.rb line 126 in block (2 levels) in dispatch
File /app/vendor/bundle/ruby/2.5.0/gems/sidekiq-5.2.5/lib/sidekiq/job_retry.rb line 73 in global
File /app/vendor/bundle/ruby/2.5.0/gems/sidekiq-5.2.5/lib/sidekiq/processor.rb line 125 in block in dispatch
File /app/vendor/bundle/ruby/2.5.0/gems/sidekiq-5.2.5/lib/sidekiq/logging.rb line 48 in with_context
File /app/vendor/bundle/ruby/2.5.0/gems/sidekiq-5.2.5/lib/sidekiq/logging.rb line 42 in with_job_hash_context
File /app/vendor/bundle/ruby/2.5.0/gems/sidekiq-5.2.5/lib/sidekiq/processor.rb line 124 in dispatch
File /app/vendor/bundle/ruby/2.5.0/gems/sidekiq-5.2.5/lib/sidekiq/processor.rb line 165 in process
File /app/vendor/bundle/ruby/2.5.0/gems/sidekiq-5.2.5/lib/sidekiq/processor.rb line 83 in process_one
File /app/vendor/bundle/ruby/2.5.0/gems/sidekiq-5.2.5/lib/sidekiq/processor.rb line 71 in run
File /app/vendor/bundle/ruby/2.5.0/gems/sidekiq-5.2.5/lib/sidekiq/util.rb line 16 in watchdog
File /app/vendor/bundle/ruby/2.5.0/gems/sidekiq-5.2.5/lib/sidekiq/util.rb line 25 in block in safe_thread
from autoscaler.
JustinLove
commented on June 16, 2024
My problem is I don't really have a sense of what is going on when the error occurs. Would you be able to post some cleaned logs with event sequences for jobs submitted/finished, and scale up/down? Failing that what kind of job patterns do you have?
One thing I noticed is that you were doing one worker per queued job, whereas default sidekiq is set up for multiple threads per worker. Even if you've got it configured for one thread per worker, it seems like that could have a lot of scaling churn.
from autoscaler.
echan00
commented on June 16, 2024
I agree with you. It is tough to debug this issue without being able to replicate it. I'll monitor for this error, I imagine it will come up again.
One thing I noticed is that you were doing one worker per queued job, whereas default sidekiq is set up for multiple threads per worker. Even if you've got it configured for one thread per worker, it seems like that could have a lot of scaling churn.
Do you suggest increasing worker_capacity? What would be a good way to estimate how to configure it?
from autoscaler.
echan00
commented on June 16, 2024
Sorry for the long wait, finally back looking at this.
I believe the following is related to the error:
02:08:12 app/worker.3: 4 TID-gn0jk9kdo DocTranslateWorker JID-fae072ecfaa5c5cde69b4d0f INFO: start
02:08:15 heroku/worker.3: State changed from up to down
Worker 3 is has been assigned to start on a job, but immediately after the heroku scaler decides to scale that worker down and causes the error. It is likely related to the fact that worker_capacity: 1 and like you mentioned there is a lot worker churn because the jobs come in short bursts requiring starting multiple workers at a time (but none needed when not needed).
Full log of worker 3:
02:07:52 heroku/worker.3: Starting process with command `bundle exec sidekiq -C config/sidekiq.yml`
02:07:53 heroku/worker.3: State changed from starting to up
02:08:09 app/worker.3: 4 TID-gn0imq158 INFO: Booting Sidekiq 5.2.5 with redis options {:url=>"redis://h:[email protected]:10199", :size=>3, :id=>"Sidekiq-server-PID-4"}
02:08:09 app/worker.3: "Setting up auto-scaledown"
02:08:12 app/worker.3: 4 TID-gn0imq158 INFO: Running in ruby 2.5.0p0 (2017-12-25 revision 61468) [x86_64-linux]
02:08:12 app/worker.3: 4 TID-gn0imq158 INFO: Starting processing, hit Ctrl-C to stop
02:08:12 app/worker.3: 4 TID-gn0jk9kdo DocTranslateWorker JID-fae072ecfaa5c5cde69b4d0f INFO: start
02:08:15 heroku/worker.3: State changed from up to down
02:08:17 heroku/worker.3: Stopping all processes with SIGTERM
02:08:17 app/worker.3: 4 TID-gn0imq158 INFO: Shutting down
02:08:17 app/worker.3: 4 TID-gn0imq158 INFO: Terminating quiet workers
02:08:17 app/worker.3: 4 TID-gn0jk9yiw INFO: Scheduler exiting...
02:08:17 app/worker.3: 4 TID-gn0imq158 INFO: Pausing to allow workers to finish...
02:08:24 app/worker.3: 4 TID-gn0imq158 WARN: Terminating 1 busy worker threads
02:08:24 app/worker.3: 4 TID-gn0imq158 WARN: Work still in progress [#<struct Sidekiq::BasicFetch::UnitOfWork queue="queue:default", job="{\"class\":\"DocTranslateWorker\",\"args\":[396,3],\"retry\":1,\"queue\":\"default\",\"backtrace\":20,\"jid\":\"fae072ecfaa5c5cde69b4d0f\",\"created_at\":1572948426.9953856,\"enqueued_at\":1572948426.9969578}">]
02:08:24 app/worker.3: 4 TID-gn0imq158 INFO: Pushed 1 jobs back to Redis
02:08:24 app/worker.3: 4 TID-gn0imq158 INFO: Bye!
02:08:24 app/worker.3: 4 TID-gn0jk9kdo DocTranslateWorker JID-fae072ecfaa5c5cde69b4d0f INFO: fail: 12.772 sec
02:08:24 app/worker.3: 4 TID-gn0jk9kdo WARN: {"context":"Job raised exception","job":{"class":"DocTranslateWorker","args":[396,3],"retry":1,"queue":"default","backtrace":20,"jid":"fae072ecfaa5c5cde69b4d0f","created_at":1572948426.9953856,"enqueued_at":1572948426.9969578},"jobstr":"{\"class\":\"DocTranslateWorker\",\"args\":[396,3],\"retry\":1,\"queue\":\"default\",\"backtrace\":20,\"jid\":\"fae072ecfaa5c5cde69b4d0f\",\"created_at\":1572948426.9953856,\"enqueued_at\":1572948426.9969578}"}
02:08:24 app/worker.3: 4 TID-gn0jk9kdo WARN: ThreadError: can't alloc thread
02:08:24 app/worker.3: 4 TID-gn0jk9kdo WARN: /app/vendor/bundle/ruby/2.5.0/gems/autoscaler-1.0.0/lib/autoscaler/sidekiq/thread_server.rb:52:in `new'
02:08:24 app/worker.3: /app/vendor/bundle/ruby/2.5.0/gems/autoscaler-1.0.0/lib/autoscaler/sidekiq/thread_server.rb:52:in `wait_for_downscale'
02:08:24 app/worker.3: /app/vendor/bundle/ruby/2.5.0/gems/autoscaler-1.0.0/lib/autoscaler/sidekiq/thread_server.rb:26:in `ensure in call'
02:08:24 app/worker.3: /app/vendor/bundle/ruby/2.5.0/gems/autoscaler-1.0.0/lib/autoscaler/sidekiq/thread_server.rb:26:in `call'
02:08:24 app/worker.3: /app/vendor/bundle/ruby/2.5.0/gems/sidekiq-5.2.5/lib/sidekiq/middleware/chain.rb:130:in `block in invoke'
02:08:24 app/worker.3: /app/vendor/bundle/ruby/2.5.0/gems/sidekiq-status-1.1.4/lib/sidekiq-status/server_middleware.rb:50:in `call'
02:08:24 app/worker.3: /app/vendor/bundle/ruby/2.5.0/gems/sidekiq-5.2.5/lib/sidekiq/middleware/chain.rb:130:in `block in invoke'
02:08:24 app/worker.3: /app/vendor/bundle/ruby/2.5.0/gems/rollbar-2.18.2/lib/rollbar/plugins/sidekiq/plugin.rb:11:in `call'
02:08:24 app/worker.3: /app/vendor/bundle/ruby/2.5.0/gems/sidekiq-5.2.5/lib/sidekiq/middleware/chain.rb:130:in `block in invoke'
from autoscaler.
Related Issues (20)
- Handle heroku api downtimes HOT 1
- Project requires more active maintainer HOT 3
- 403 Forbidden when scaling worker HOT 1
- support for Sidekiq 3.0.0 HOT 4
- undefined method `async' for nil:NilClass HOT 2
- Autoscaling multiple workers HOT 1
- support for Sidekiq 3.1 and 3.2 HOT 2
- Upgrade to platform-api HOT 2
- time to release a new version? HOT 15
- Linear scaling strategy with long running jobs HOT 3
- Handling multiple queues with a single worker dyno HOT 4
- How to use LinearScalingStrategy? HOT 1
- Project maintainer should also be a user HOT 3
- Error when scaling down - Unable to create thread HOT 10
- don't want to use set_initial_workers HOT 5
- Sidekiq 5 support? HOT 2
- Heroku platform API error HOT 3
- Rename ENV vars HOT 4
- Autoscaler is passaing Incorrect Authentication "type" param to platform-api HOT 4
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from autoscaler.