Comments (4)
#42 is just about recovering from a BrokenProcessPool
. It does nothing to prevent the exception in the first place. The exception is raised when the Python process hosting the job exits unexpectedly.
The most likely cause your job is hitting a memory limit and the process is killed.
from faktory_worker_python.
#42 is just about recovering from a
BrokenProcessPool
. It does nothing to prevent the exception in the first place. The exception is raised when the Python process hosting the job exits unexpectedly.The most likely cause your job is hitting a memory limit and the process is killed.
I see. The worker is set with a concurrency of 1, so that means that all the memory available for the k8s pod is also available to that single process (currently 600Mi). But if that limit is exceeded, the pod should be restarted instead of throwing this error. Is there some other memory configurations regarding the worker that I should pay attention? Increasing the number of processes can help with this? (I can always spawn more pods to compensate for the single process in the worker, but more processes per pod seems interesting)
Thanks for the reply! @cdrx
from faktory_worker_python.
that means that all the memory available for the k8s pod is also available to that single process (currently 600Mi).
There are two processes. The master process, which starts a second process - the child (worker) that actually runs your job. Linux will kill the child process if is using more memory than the k8s allows. K8s will only restart the pod if the master process dies.
If you have a concurrency of 1, you can try to use threads in the worker instead. This is done by passing use_threads=True
at Worker creation.
It will run only a single process, so k8s will notice if it is killed for ignoring a memory limit.
from faktory_worker_python.
Thanks! I tried this today and the error has not appeared, but I'll have to wait until some real users try the service and see if the error appears again. In the meantime, do you kwnow some form of measuring each process memory consumption? it will help in the future so I dont need 1 pod per worker and can up the concurrency a bit
from faktory_worker_python.
Related Issues (20)
- i/o timeout Closing connection
- worker thread got OSError: [WinError 6] The handle is invalid when stop worker from Web UI
- Release 1.0
- Add support for sending exceptions to Sentry HOT 5
- Worker crashes if there's an error when fetching new jobs HOT 1
- Getting INFO times out after starting a lot of workers HOT 2
- Example worker restarting itself on 3.9 HOT 6
- Cannot install version 0.5.0 via pip HOT 3
- ImportError: cannot import name 'BrokenThreadPool' for python version less than 3.7 HOT 1
- Flag version >0.5 to only be python 3.7+ compatible HOT 2
- Batch Support HOT 3
- How to use the `at` argument in queue? HOT 1
- Worker heartbeat crashes with "type 'NoneType' is not iterable" HOT 1
- Remove references to `priority` as a job argument
- FaktoryConnectionResetError in Connection.reply
- AttributeError: Can't pickle local object 'ETL_task.<locals>.task' HOT 6
- Explaination on the thread execution model HOT 4
- Make a new release (v1.1) HOT 1
- Access to Job status in Python client for faktory
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from faktory_worker_python.