Comments (5)
worth to add, we run the same asset on Databricks, that runs on EC2 instances under the hood, and it does work with no issue in the newer version... so the problem is in when we submit it directly using the dagster_aws
from dagster.
This is very odd. On a new EMR instance, running pip freeze shows:
python-dateutil==2.8.1
But going into python and checking dateutil's version gives
>>> import dateutil
>>> dateutil.__version__
'2.2'
from dagster.
Oh weird, are there two dateutils in the default EMR python environment??
py-dateutil==2.2
python-dateutil==2.8.1
from dagster.
@HPicatto can you try pip uninstalling py-dateutil? That fixed the problem for me. Very odd that the default EMR environment includes both, py-dateutil had a single 2.2 release almost a decade ago: https://pypi.org/project/py-dateutil/#history
from dagster.
Raised this with AWS here: https://repost.aws/questions/QUl4F36LBhQu-TPbgalNJ-sQ/default-emr-spark-python-environment-contains-two-different-versions-of-the-dateutil-package
from dagster.
Related Issues (20)
- Dagster-daemon stops spawning "dagster api grpc" processes HOT 2
- Dagit UI Activity Timeline 'Now' timestamp is incorrect for 30 minute timezone offsets HOT 1
- [dagster-dbt] Only materialize views if there's a code change HOT 1
- Unknown cause for ValueError: time data 'START_DATE' does not match format '%Y-%m-%d' when using environment variable HOT 6
- Confusing documentation on run monitoring HOT 1
- Tasks created for runs on ECS deployments are not associated with a service HOT 1
- GraphQL Playground auto-complete and type boxes don't disappear in the UI when using Chrome browser HOT 1
- Dagster runs that are manually terminated in code that has overridden the default SIGINT handler move into a FAILURE state instead of a CANCELED state
- FileNotFoundError: [Errno 2] No such file or directory: '.\\dagster_home\\storage\\asset_name' HOT 1
- Kubernetes hybrid agent redeploys every code location on restart HOT 2
- execute_k8s_job fails when native sidecars are used HOT 1
- A CLI command that can check/validate definitions without loading the Dagster services/UI
- Webserver at random times for random periods makes queries to dagster DB that drive CPU usage to near maximum HOT 1
- Expose internal_asset_deps in @graph_multi_asset HOT 1
- Global concurrency_key limits do not affect asset materialization HOT 3
- dagster-snowflake doesn't support SQLAlchemy version >= 2.0.0
- Custom freshness checks on dynamically partitioned assets HOT 1
- DbtProject does not take profiles dir for dbt parse
- dagster fails to load simple code location after 1.8 upgrade HOT 19
- Dagster schedule ticks that fail and retry should indicate in the tick timeline that they retried HOT 4
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from dagster.