Comments (6)
To address the issue of worker instances continuously restarting after upgrading to Superset version 4.0.0, consider the following steps:
-
Compatibility Check: Ensure all components, including Python 3.11 and Node 16, are compatible with Superset 4.0.0. While Python 3.11 is compatible, Node.js version 18.20.1 is recommended for the best development experience with Superset's frontend.
-
Review Custom Bootstrap Script: Inspect your custom bootstrap script for any conflicts or deprecated packages that might not be compatible with Superset 4.0.0.
-
Worker Configuration: Verify the
command
used to start worker instances in your helm chart is correct for Superset 4.0.0, focusing on the path tosuperset_bootstrap.sh
and the Celery command. -
Liveness and Readiness Probes: Ensure the configuration for these probes in the
supersetWorker
section is correctly set up for Superset 4.0.0, as incorrect configurations can lead to continuous restarts. -
Environment Variables and Configurations: Check that all necessary environment variables and configurations are correctly set up, especially those derived from secrets or config maps.
-
Log Analysis: Review the logs of the restarting worker instances for errors or warnings that could indicate the cause of the restarts.
-
Dependencies and Package Versions: Make sure all dependencies specified in your bootstrap script or helm chart are compatible with Superset 4.0.0.
-
Resource Limits: Review and adjust the resource limits and requests for the worker instances if they are being terminated due to resource constraints.
-
Review Helm Chart Annotations and Checksums: Update checksums for configurations if changes are made, ensuring Kubernetes recognizes and applies these changes.
-
Security Concerns with
runAsUser
: Running containers as root (UID 0) is discouraged due to security risks. Consider running containers with a non-root user.
These steps should help identify and resolve the issue with worker instances restarting. If the problem persists, further investigation into Superset's change logs for version 4.0.0 may be necessary to identify any breaking changes or additional migration steps required.
Details
Sources
- supersetWorker Liveness Probes always fail causing restarts
- helm/superset/values.yaml
- helm/superset/templates/deployment-worker.yaml
- helm/superset/templates/deployment.yaml
- CHANGELOG/4.0.0.md
- pyproject.toml
- superset-frontend/.nvmrc
About Dosu
This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.
Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.
To continue the conversation, mention @dosu.
from superset.
It's an issue with Liveness probe :
Liveness probe failed: ['/usr/local/bin', '/app/pythonpath', '/usr/local/lib/python310.zip', '/usr/local/lib/python3.10', '/usr/local/lib/python3.10/lib-dynload', '/usr/local/lib/python3.10/site-packages', '/app', '/app/configs/'] Loaded your LOCAL configuration at [/app/pythonpath/superset_config.py] logging was configured successfully 2024-05-13 02:49:48,445:INFO:superset.utils.logging_configurator:logging was configured successfully 2024-05-13 02:49:48,447:INFO:root:Configured event logger of type <class 'superset.utils.log.DBEventLogger'> Error: No nodes replied within time constraint
from superset.
After running 'celery -A superset.tasks.celery_app:app inspect ping -d celery@$HOSTNAME' manually in worker pod, response was Error:
root@superset-worker-54fcfd4b-bzpzd:/app# celery -A superset.tasks.celery_app:app inspect ping -d celery@$HOSTNAME
['/usr/local/bin', '/app/pythonpath', '/usr/local/lib/python310.zip', '/usr/local/lib/python3.10', '/usr/local/lib/python3.10/lib-dynload', '/usr/local/lib/python3.10/site-packages', '/app', '/app/configs/']
Loaded your LOCAL configuration at [/app/pythonpath/superset_config.py]
logging was configured successfully
2024-05-13 02:58:06,359:INFO:superset.utils.logging_configurator:logging was configured successfully
2024-05-13 02:58:06,361:INFO:root:Configured event logger of type <class 'superset.utils.log.DBEventLogger'>
Error: No nodes replied within time constraint
from superset.
After done some digging, it appears to be related to the Celery version issue with Redis. #28471
from superset.
After upgrade to celery 5.4, flower can see workers listed, but the ping for liveness within worker still fails with no nodes replied.
from superset.
LivenessProbe still fails with 'Error: No nodes replied within time constraint' as result from "celery -A superset.tasks.celery_app:app inspect ping -d celery@$HOSTNAME"
from superset.
Related Issues (20)
- [SIP-133] Update "Time Range" filter in dashboard HOT 6
- HTTP Headers Configuration on config.py not overriding. HOT 3
- "embed dashboard" button does not show up HOT 1
- Missing CSV file in email of report / alert HOT 1
- Running superset db upgrade results is about a SQLite syntax error. HOT 1
- How to customize the row number in the exported excel file? HOT 2
- There is no filter box in apache superset version 3.1.3 HOT 2
- Published Dashboard without charts can only be reached by owners HOT 1
- Owner/Creator FrankjesusGonzales ©2024 📍
- Angular run time error of superset-embedded-sdk HOT 3
- DISPLAY_MAX_ROW 10.000 / 100.000 HOT 2
- Tables with long numbers are truncated when exported to excel HOT 2
- superset-frontend ReferenceError: module is not defined HOT 3
- Error in SQL Lab or trying to create a Dataset based on Hive HOT 1
- "Drill to Details by" Feature Ignores Applied Filters in Superset 4.0.1 HOT 3
- Create Dashboard API not working HOT 1
- Permalinks doesn't work: 'utf-8' codec can't decode byte 0x80 in position 0: invalid start byte HOT 1
- [SIP] Hide inactive users
- Unable to retrieve Datasets from SST Api HOT 8
- Alerts & Reports - CSV as Message Content resulting HTTP 500 HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from superset.