Comments (2)
The way I have been running it in production at scale is to have a decoupled redis database living outside of the k8s cluster and then just let the autoscaler do its thing depending on workloads. Usually I work off CPU or Memory depending on the system, but there are also ways to autoscale based on number of jobs in a queue for example.
The way I structure the setup is like this:
- Redis living on its own server. If you use AWS, you can use Elasticache just fine.
- I use resque-pool to manage which workers startup for each queue.
- Depending on your unit of separation, each one starts up resque pool which then forks off each of the workers. If you're using k8s, it would be pods. If you use bare metal or some other kind of VPS instance, each one would start up with a finite amount of workers.
- Servers/pods/instances can die off but the state of the queue is tracked in redis externally so you can just add new boxes into rotation.
- Redis also tracks the state of the workers, so if a resque main process is killed off and it hasn't had a chance to report in, you may have an incorrect number of workers in your resque dashboard. To get around that, I run a script that periodically cleans them up. That script looks something like this:
Resque.workers.each do |w|
w.unregister_worker if w.processing['run_at'] && Time.now - w.processing['run_at'].to_time > 1000
end
Hope that answers your questions!
from resque.
Does the master process require it be running monolith style to control all the workers across various nodes? Is there a way to configure this (not kubernetes) or is it all just going to be ok, spinning this thing up over and over again, with various master processes that are only aware of their own nodes?
is any state contained within the resque master process or does it keep everything that other resque containers might need to know in redis just by default?
from resque.
Related Issues (20)
- Unsupported command argument type: TrueClass. Migration from redis gem v4 to v5 HOT 5
- Web Interface failing with Internal Server Error
- Blocking background worker process over SSH HOT 2
- Make the signal for child termination configurable HOT 1
- Workers not terminating as expected HOT 1
- resque-status is not compatible with Resque > 2 HOT 2
- Experiencing a lot of database overhead with forked jobs HOT 3
- deadlock during deallocation of threads and fork
- [feature request] filter sensitive args in resque server
- DirtyExit VS PruneDeadWorkerDirtyExit - what is the difference? HOT 1
- USR2 signal is happening but my resque job is going to dirty exit HOT 2
- start_heartbeat can die while worker is still running HOT 1
- Unpatched CVE-2022-44303 - Reflected XSS HOT 4
- Please publish security advisories for XSS vulnerabilities HOT 4
- Search Feature in failed jobs HOT 2
- Cannot use with Rack 3.0 because of Sinatra (resque-web); Blocked upgrade to Rails 7.1 HOT 6
- Ruby 3.3 and stuck workers. HOT 23
- Overview UI Broken Due to Nonce Whitelist HOT 2
- Code & comment for redis initializer method could be clearer
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from resque.