Comments (3)
It is related to the unresolved highly upvoted issue with dagster-postgres
:
Basically put pg-bouncer infront of it or you cannot use dagster-postgres
in production.
from dagster.
Some general tips can be found in this discussion #14771
I assume you are GKE
on GCP? Do you see any issues with any of the dagster pods?
How many "stations" / dynamic partitions do you expect to interact with every tick?
all sensors are stuck
Are there multiple sensors structured the same way?
For some context what is happening under the hood is that the dagster daemon is responsible for managing the ticks of sensors/schedules but evaluating function happens in the code server. The daemon should not start the next tick for a sensor until the previous one completes, so I am surprised to see connections pile up unless
- there are multiple of these sensors running concurrently
- the processes are getting OOM killed or something and leaving dangling connections
from dagster.
Some general tips can be found in this discussion #14771
I assume you are
GKE
on GCP? Do you see any issues with any of the dagster pods?How many "stations" / dynamic partitions do you expect to interact with every tick?
all sensors are stuck
Are there multiple sensors structured the same way?
For some context what is happening under the hood is that the dagster daemon is responsible for managing the ticks of sensors/schedules but evaluating function happens in the code server. The daemon should not start the next tick for a sensor until the previous one completes, so I am surprised to see connections pile up unless
* there are multiple of these sensors running concurrently * the processes are getting OOM killed or something and leaving dangling connections
@alangenfeld
I believe it has to do with running postgres
, as the issue is reproducible. If you profile you will notice that even a trivial deployment of dagster
with some user-code (and dagster-webserver
+ dagster-deamon
) will make dagster-postgres
overwhelm any unprotected postgres
deployment, even if you bump the normal connection limit from 100 to 1000 connections. There are hundreds of connections a second being spammed by dagster due to the code in dagster-postgres
. I assume you use something different in the cloud version?
edit: cleaned up hasty response
from dagster.
Related Issues (20)
- CeleryK8sRunLauncher doesn't work with celery_executor HOT 3
- Hooks: slack_on_success and slack_on_failure fail silently to work
- [dagster-deltalake] GcsConfig ImportError and TypeError for partitioned assets
- Different UX when viewing runs locally than in production because of additional tags
- `load_asset_checks_from_module` sometimes returns `AssetsDefinition`s instead of `AssetChecksDefinition`s
- ModuleNotFoundError: No module named 'dbt.adapters.base.impl' HOT 4
- Support tags with colon (:) HOT 4
- Propagate filters when navigating through catalog search results
- dagster_pipes.DagsterPipesError: Cannot send message after pipes context is closed." HOT 2
- Cannot create asset job with BackfillPolicy.multi_run() and backfill_policy=None
- GCS IO manager connection error "AttributeError: 'NoneType' object has no attribute 'get_client'" HOT 6
- Dagster-pipes report_asset_materialization is missing `partition` versioning
- Daemon error on code location reload HOT 4
- Dagster-pipes errors out even though run was successfull HOT 1
- Allow easily backfilling all failed partitions for different assets with different partitions HOT 2
- @io_manager(config_schema=...) fails to resolve silently after pydantic update >=2.7.1
- Provide more flexible launch-time configuration composition options for Kubernetes HOT 1
- Allow multiple partition selection in launch pad
- Launchpad on asset shows all resources instead of the required ones HOT 2
- Missing environment variable causes silent failure of materialization from UI HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from dagster.