Comments (6)
Simplest definitions file I could create to reproduce the issue:
from dagster import Definitions, graph, op, DailyPartitionsDefinition
@op
def my_op(context):
context.log.info(f"{context.op_config=}")
@graph
def my_graph():
my_op()
definitions = Definitions(
jobs=[
my_graph.to_job(
config={"ops": {"my_op": {"config": {"key": "10:30"}}}},
partitions_def=DailyPartitionsDefinition("2024-01-01"), # Only seems to be an issue when the job is partitioned
)
]
)
Fresh conda environment
# Name Version Build Channel
_libgcc_mutex 0.1 conda_forge conda-forge
_openmp_mutex 4.5 2_gnu conda-forge
alembic 1.13.1 pyhd8ed1ab_1 conda-forge
aniso8601 9.0.1 pyhd8ed1ab_0 conda-forge
annotated-types 0.6.0 pyhd8ed1ab_0 conda-forge
anyio 4.2.0 pyhd8ed1ab_0 conda-forge
backoff 2.2.1 pyhd8ed1ab_0 conda-forge
brotli-python 1.1.0 py310hc6cd4ac_1 conda-forge
bzip2 1.0.8 hd590300_5 conda-forge
c-ares 1.26.0 hd590300_0 conda-forge
ca-certificates 2024.2.2 hbcca054_0 conda-forge
certifi 2024.2.2 pyhd8ed1ab_0 conda-forge
charset-normalizer 3.3.2 pyhd8ed1ab_0 conda-forge
click 8.1.7 unix_pyh707e725_0 conda-forge
colorama 0.4.6 pyhd8ed1ab_0 conda-forge
coloredlogs 14.0 pyhd8ed1ab_3 conda-forge
croniter 2.0.1 pyhd8ed1ab_0 conda-forge
dagster 1.6.3 pyhd8ed1ab_0 conda-forge
dagster-graphql 1.6.3 pyhd8ed1ab_0 conda-forge
dagster-pipes 1.6.3 pyhd8ed1ab_0 conda-forge
dagster-webserver 1.6.3 pyhd8ed1ab_0 conda-forge
docstring_parser 0.15 pyhd8ed1ab_0 conda-forge
exceptiongroup 1.2.0 pyhd8ed1ab_2 conda-forge
fsspec 2024.2.0 pyhca7485f_0 conda-forge
gql 3.5.0 pyhd8ed1ab_1 conda-forge
gql-with-requests 3.5.0 pyhd8ed1ab_1 conda-forge
graphene 3.3 pyhd8ed1ab_0 conda-forge
graphql-core 3.2.3 pyhd8ed1ab_0 conda-forge
graphql-relay 3.2.0 pyhd8ed1ab_0 conda-forge
greenlet 3.0.3 py310hc6cd4ac_0 conda-forge
grpcio 1.60.0 py310h1b8f574_1 conda-forge
grpcio-health-checking 1.60.0 pyhd8ed1ab_0 conda-forge
h11 0.14.0 pyhd8ed1ab_0 conda-forge
httptools 0.6.1 py310h2372a71_0 conda-forge
humanfriendly 10.0 pyhd8ed1ab_6 conda-forge
idna 3.6 pyhd8ed1ab_0 conda-forge
importlib-metadata 7.0.1 pyha770c72_0 conda-forge
importlib_resources 6.1.1 pyhd8ed1ab_0 conda-forge
jinja2 3.1.3 pyhd8ed1ab_0 conda-forge
ld_impl_linux-64 2.40 h41732ed_0 conda-forge
libabseil 20230802.1 cxx17_h59595ed_0 conda-forge
libffi 3.4.2 h7f98852_5 conda-forge
libgcc-ng 13.2.0 h807b86a_5 conda-forge
libgomp 13.2.0 h807b86a_5 conda-forge
libgrpc 1.60.0 h74775cd_1 conda-forge
libnsl 2.0.1 hd590300_0 conda-forge
libprotobuf 4.25.1 hf27288f_1 conda-forge
libre2-11 2023.06.02 h7a70373_0 conda-forge
libsqlite 3.44.2 h2797004_0 conda-forge
libstdcxx-ng 13.2.0 h7e041cc_5 conda-forge
libuuid 2.38.1 h0b41bf4_0 conda-forge
libuv 1.46.0 hd590300_0 conda-forge
libxcrypt 4.4.36 hd590300_1 conda-forge
libzlib 1.2.13 hd590300_5 conda-forge
mako 1.3.2 pyhd8ed1ab_0 conda-forge
markdown-it-py 3.0.0 pyhd8ed1ab_0 conda-forge
markupsafe 2.1.5 py310h2372a71_0 conda-forge
mdurl 0.1.2 pyhd8ed1ab_0 conda-forge
multidict 6.0.5 py310h2372a71_0 conda-forge
ncurses 6.4 h59595ed_2 conda-forge
openssl 3.2.1 hd590300_0 conda-forge
packaging 23.2 pyhd8ed1ab_0 conda-forge
pendulum 2.1.2 py310h2372a71_6 conda-forge
pip 24.0 pyhd8ed1ab_0 conda-forge
protobuf 4.25.1 py310h620c231_0 conda-forge
psutil 5.9.8 py310h2372a71_0 conda-forge
pydantic 2.6.1 pyhd8ed1ab_0 conda-forge
pydantic-core 2.16.2 py310hcb5633a_1 conda-forge
pygments 2.17.2 pyhd8ed1ab_0 conda-forge
pysocks 1.7.1 pyha2e5f31_6 conda-forge
python 3.10.13 hd12c33a_1_cpython conda-forge
python-dateutil 2.8.2 pyhd8ed1ab_0 conda-forge
python-dotenv 1.0.1 pyhd8ed1ab_0 conda-forge
python_abi 3.10 4_cp310 conda-forge
pytz 2024.1 pyhd8ed1ab_0 conda-forge
pytzdata 2020.1 pyh9f0ad1d_0 conda-forge
pywin32-on-windows 0.1.0 pyh1179c8e_3 conda-forge
pyyaml 6.0.1 py310h2372a71_1 conda-forge
re2 2023.06.02 h2873b5e_0 conda-forge
readline 8.2 h8228510_1 conda-forge
requests 2.31.0 pyhd8ed1ab_0 conda-forge
requests-toolbelt 1.0.0 pyhd8ed1ab_0 conda-forge
rich 13.7.0 pyhd8ed1ab_0 conda-forge
setuptools 69.0.3 pyhd8ed1ab_0 conda-forge
six 1.16.0 pyh6c4a22f_0 conda-forge
sniffio 1.3.0 pyhd8ed1ab_0 conda-forge
sqlalchemy 2.0.25 py310h2372a71_0 conda-forge
starlette 0.37.0 pyhd8ed1ab_0 conda-forge
structlog 23.2.0 pyhd8ed1ab_0 conda-forge
tabulate 0.9.0 pyhd8ed1ab_1 conda-forge
tk 8.6.13 noxft_h4845f30_101 conda-forge
tomli 2.0.1 pyhd8ed1ab_0 conda-forge
toposort 1.10 pyhd8ed1ab_0 conda-forge
tqdm 4.66.1 pyhd8ed1ab_0 conda-forge
typing-extensions 4.9.0 hd8ed1ab_0 conda-forge
typing_extensions 4.9.0 pyha770c72_0 conda-forge
tzdata 2024a h0c530f3_0 conda-forge
universal_pathlib 0.1.4 pyhd8ed1ab_0 conda-forge
urllib3 2.2.0 pyhd8ed1ab_0 conda-forge
uvicorn 0.27.0.post1 py310hff52083_0 conda-forge
uvicorn-standard 0.27.0.post1 hff52083_0 conda-forge
uvloop 0.19.0 py310h7d0a568_0 conda-forge
watchdog 4.0.0 py310hff52083_0 conda-forge
watchfiles 0.21.0 py310hcb5633a_0 conda-forge
websockets 11.0.3 py310h2372a71_1 conda-forge
wheel 0.42.0 pyhd8ed1ab_0 conda-forge
xz 5.2.6 h166bdaf_0 conda-forge
yaml 0.2.5 h7f98852_2 conda-forge
yarl 1.9.4 py310h2372a71_0 conda-forge
zipp 3.17.0 pyhd8ed1ab_0 conda-forge
Running with dagster dev -f <path-to-repro.py>
:
from dagster.
Reporting the different versions of dagster I tried this with:
1.4.2 - good
1.5.0 - good
1.5.2 - good
1.5.3 - bad # --
1.5.5 - bad
1.5.10 - bad
1.6.0 - bad
1.6.1 - bad
1.6.3 - bad
Looks like the regression was introduced in 1.5.3 ?
from dagster.
Checking what the front-end sends:
Best guess is that this was inadvertently introduced with the change here, but my typescript and context for the front-end is weak, so could be miles off!
from dagster.
The other effects of this are also:
- Lists in YAML config are doubled when selecting a partition, then selecting another one (due to the fault
yaml.merge
logic mentioned above. - Float values like
1.0
have their type destroyed and are transformed to ints in the launchpad and in the resulting job.
from dagster.
Thanks @mzjp2, looking into this today to see if we can get it fixed in this week's release
from dagster.
Hey gang, we've fixed a few changes in this week's release that address the most immediate concerns identified here. I think in a perfect world, we wouldn't convert the YAML to a JSON/JS representation when merging the YAML document with the run/partition defaults and that'd allow us to preserve floating point numbers -- but it'll be difficult to stop doing that so I'm going to punt it for now.
Thanks for flagging this issue, the array merging behavior in particular was really pretty bad. 🙏
from dagster.
Related Issues (20)
- Lesson 8: ImportError: cannot import name 'utc_datetime_from_timestamp' from 'dagster._utils' HOT 1
- The dagster-pandera integration cannot handle `Annotated` types
- Dagster daemon heartbeat randomly stops HOT 2
- Dagster-slack is missing py.typed marker
- Examples for k8s configuration
- dagster-dbt : unable to load dbt project of an dbt-teradata adapter (when database is None in manifest.json) HOT 2
- Asset check `@asset_check` docstring not loaded as description
- asset check cannot load an asset that has a prefix
- Implement PipesCloudWatchMessageReader
- Error in "Customizing your Kubernetes Deployment"
- Config field descriptions not loading from attribute docstrings HOT 1
- Automatic pruning for `TimeWindowPartitionsDefinition` HOT 2
- materializable looker assets
- error with 1.7.14 : 'AssetSpecWithPartitionsDef' object has no attribute 'auto_materialize_policy'
- Add a Dagster run status for warnings. HOT 2
- Enabling dbt to fetch row count data causes job failures to go unreported. HOT 1
- [dagster-dbt] `fetch_column_metadata` fails when building models installed by dbt dependencies HOT 1
- Cannot provide optional config to a partitioned asset when triggering a backfill via the UI materialize button HOT 2
- Include the current scheduledefinition on ScheduleEvaluationContext
- Extend code references git_url to include self-hosted GitLab HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from dagster.