Comments (6)
agon-shabi
commented on July 23, 2024
2
Simplest definitions file I could create to reproduce the issue:
from dagster import Definitions, graph, op, DailyPartitionsDefinition
@op
def my_op(context):
context.log.info(f"{context.op_config=}")
@graph
def my_graph():
my_op()
definitions = Definitions(
jobs=[
my_graph.to_job(
config={"ops": {"my_op": {"config": {"key": "10:30"}}}},
partitions_def=DailyPartitionsDefinition("2024-01-01"), # Only seems to be an issue when the job is partitioned
)
]
)Fresh conda environment
# Name Version Build Channel
_libgcc_mutex 0.1 conda_forge conda-forge
_openmp_mutex 4.5 2_gnu conda-forge
alembic 1.13.1 pyhd8ed1ab_1 conda-forge
aniso8601 9.0.1 pyhd8ed1ab_0 conda-forge
annotated-types 0.6.0 pyhd8ed1ab_0 conda-forge
anyio 4.2.0 pyhd8ed1ab_0 conda-forge
backoff 2.2.1 pyhd8ed1ab_0 conda-forge
brotli-python 1.1.0 py310hc6cd4ac_1 conda-forge
bzip2 1.0.8 hd590300_5 conda-forge
c-ares 1.26.0 hd590300_0 conda-forge
ca-certificates 2024.2.2 hbcca054_0 conda-forge
certifi 2024.2.2 pyhd8ed1ab_0 conda-forge
charset-normalizer 3.3.2 pyhd8ed1ab_0 conda-forge
click 8.1.7 unix_pyh707e725_0 conda-forge
colorama 0.4.6 pyhd8ed1ab_0 conda-forge
coloredlogs 14.0 pyhd8ed1ab_3 conda-forge
croniter 2.0.1 pyhd8ed1ab_0 conda-forge
dagster 1.6.3 pyhd8ed1ab_0 conda-forge
dagster-graphql 1.6.3 pyhd8ed1ab_0 conda-forge
dagster-pipes 1.6.3 pyhd8ed1ab_0 conda-forge
dagster-webserver 1.6.3 pyhd8ed1ab_0 conda-forge
docstring_parser 0.15 pyhd8ed1ab_0 conda-forge
exceptiongroup 1.2.0 pyhd8ed1ab_2 conda-forge
fsspec 2024.2.0 pyhca7485f_0 conda-forge
gql 3.5.0 pyhd8ed1ab_1 conda-forge
gql-with-requests 3.5.0 pyhd8ed1ab_1 conda-forge
graphene 3.3 pyhd8ed1ab_0 conda-forge
graphql-core 3.2.3 pyhd8ed1ab_0 conda-forge
graphql-relay 3.2.0 pyhd8ed1ab_0 conda-forge
greenlet 3.0.3 py310hc6cd4ac_0 conda-forge
grpcio 1.60.0 py310h1b8f574_1 conda-forge
grpcio-health-checking 1.60.0 pyhd8ed1ab_0 conda-forge
h11 0.14.0 pyhd8ed1ab_0 conda-forge
httptools 0.6.1 py310h2372a71_0 conda-forge
humanfriendly 10.0 pyhd8ed1ab_6 conda-forge
idna 3.6 pyhd8ed1ab_0 conda-forge
importlib-metadata 7.0.1 pyha770c72_0 conda-forge
importlib_resources 6.1.1 pyhd8ed1ab_0 conda-forge
jinja2 3.1.3 pyhd8ed1ab_0 conda-forge
ld_impl_linux-64 2.40 h41732ed_0 conda-forge
libabseil 20230802.1 cxx17_h59595ed_0 conda-forge
libffi 3.4.2 h7f98852_5 conda-forge
libgcc-ng 13.2.0 h807b86a_5 conda-forge
libgomp 13.2.0 h807b86a_5 conda-forge
libgrpc 1.60.0 h74775cd_1 conda-forge
libnsl 2.0.1 hd590300_0 conda-forge
libprotobuf 4.25.1 hf27288f_1 conda-forge
libre2-11 2023.06.02 h7a70373_0 conda-forge
libsqlite 3.44.2 h2797004_0 conda-forge
libstdcxx-ng 13.2.0 h7e041cc_5 conda-forge
libuuid 2.38.1 h0b41bf4_0 conda-forge
libuv 1.46.0 hd590300_0 conda-forge
libxcrypt 4.4.36 hd590300_1 conda-forge
libzlib 1.2.13 hd590300_5 conda-forge
mako 1.3.2 pyhd8ed1ab_0 conda-forge
markdown-it-py 3.0.0 pyhd8ed1ab_0 conda-forge
markupsafe 2.1.5 py310h2372a71_0 conda-forge
mdurl 0.1.2 pyhd8ed1ab_0 conda-forge
multidict 6.0.5 py310h2372a71_0 conda-forge
ncurses 6.4 h59595ed_2 conda-forge
openssl 3.2.1 hd590300_0 conda-forge
packaging 23.2 pyhd8ed1ab_0 conda-forge
pendulum 2.1.2 py310h2372a71_6 conda-forge
pip 24.0 pyhd8ed1ab_0 conda-forge
protobuf 4.25.1 py310h620c231_0 conda-forge
psutil 5.9.8 py310h2372a71_0 conda-forge
pydantic 2.6.1 pyhd8ed1ab_0 conda-forge
pydantic-core 2.16.2 py310hcb5633a_1 conda-forge
pygments 2.17.2 pyhd8ed1ab_0 conda-forge
pysocks 1.7.1 pyha2e5f31_6 conda-forge
python 3.10.13 hd12c33a_1_cpython conda-forge
python-dateutil 2.8.2 pyhd8ed1ab_0 conda-forge
python-dotenv 1.0.1 pyhd8ed1ab_0 conda-forge
python_abi 3.10 4_cp310 conda-forge
pytz 2024.1 pyhd8ed1ab_0 conda-forge
pytzdata 2020.1 pyh9f0ad1d_0 conda-forge
pywin32-on-windows 0.1.0 pyh1179c8e_3 conda-forge
pyyaml 6.0.1 py310h2372a71_1 conda-forge
re2 2023.06.02 h2873b5e_0 conda-forge
readline 8.2 h8228510_1 conda-forge
requests 2.31.0 pyhd8ed1ab_0 conda-forge
requests-toolbelt 1.0.0 pyhd8ed1ab_0 conda-forge
rich 13.7.0 pyhd8ed1ab_0 conda-forge
setuptools 69.0.3 pyhd8ed1ab_0 conda-forge
six 1.16.0 pyh6c4a22f_0 conda-forge
sniffio 1.3.0 pyhd8ed1ab_0 conda-forge
sqlalchemy 2.0.25 py310h2372a71_0 conda-forge
starlette 0.37.0 pyhd8ed1ab_0 conda-forge
structlog 23.2.0 pyhd8ed1ab_0 conda-forge
tabulate 0.9.0 pyhd8ed1ab_1 conda-forge
tk 8.6.13 noxft_h4845f30_101 conda-forge
tomli 2.0.1 pyhd8ed1ab_0 conda-forge
toposort 1.10 pyhd8ed1ab_0 conda-forge
tqdm 4.66.1 pyhd8ed1ab_0 conda-forge
typing-extensions 4.9.0 hd8ed1ab_0 conda-forge
typing_extensions 4.9.0 pyha770c72_0 conda-forge
tzdata 2024a h0c530f3_0 conda-forge
universal_pathlib 0.1.4 pyhd8ed1ab_0 conda-forge
urllib3 2.2.0 pyhd8ed1ab_0 conda-forge
uvicorn 0.27.0.post1 py310hff52083_0 conda-forge
uvicorn-standard 0.27.0.post1 hff52083_0 conda-forge
uvloop 0.19.0 py310h7d0a568_0 conda-forge
watchdog 4.0.0 py310hff52083_0 conda-forge
watchfiles 0.21.0 py310hcb5633a_0 conda-forge
websockets 11.0.3 py310h2372a71_1 conda-forge
wheel 0.42.0 pyhd8ed1ab_0 conda-forge
xz 5.2.6 h166bdaf_0 conda-forge
yaml 0.2.5 h7f98852_2 conda-forge
yarl 1.9.4 py310h2372a71_0 conda-forge
zipp 3.17.0 pyhd8ed1ab_0 conda-forge
Running with dagster dev -f <path-to-repro.py>:
opening in launchpad
selecting partition (config gets filled in)
during run
and viewing the run's configuration
from dagster.
agon-shabi
commented on July 23, 2024
Reporting the different versions of dagster I tried this with:
1.4.2 - good
1.5.0 - good
1.5.2 - good
1.5.3 - bad # --
1.5.5 - bad
1.5.10 - bad
1.6.0 - bad
1.6.1 - bad
1.6.3 - bad
Looks like the regression was introduced in 1.5.3 ?
from dagster.
agon-shabi
commented on July 23, 2024
Checking what the front-end sends:
in 1.5.2: the value retains its quoting
in 1.5.3: the value seems to have _lost_ its quoting
Best guess is that this was inadvertently introduced with the change here, but my typescript and context for the front-end is weak, so could be miles off!
from dagster.
mzjp2
commented on July 23, 2024
The other effects of this are also:
- Lists in YAML config are doubled when selecting a partition, then selecting another one (due to the fault
yaml.mergelogic mentioned above. - Float values like
1.0have their type destroyed and are transformed to ints in the launchpad and in the resulting job.
from dagster.
bengotow
commented on July 23, 2024
Thanks @mzjp2, looking into this today to see if we can get it fixed in this week's release
from dagster.
bengotow
commented on July 23, 2024
Hey gang, we've fixed a few changes in this week's release that address the most immediate concerns identified here. I think in a perfect world, we wouldn't convert the YAML to a JSON/JS representation when merging the YAML document with the run/partition defaults and that'd allow us to preserve floating point numbers -- but it'll be difficult to stop doing that so I'm going to punt it for now.
Thanks for flagging this issue, the array merging behavior in particular was really pretty bad. 🙏
from dagster.
Related Issues (20)
- dbt_select when building schedules not working HOT 2
- Loading dbt manifest file fails when it contains semantic_model ressource type
- [Documentation Feedback] Problem on /dagster-plus/developing-testing/code-locations page HOT 1
- AutoMaterialize "cooldown": AutoMaterializePolicy.eager(cooldown_seconds=300), with SKIP instead of DISCARD
- Recursive graph
- Please provide an example for run partial ops of a job HOT 1
- Make it more clear when "test sensor" button isn't available
- Readonly webserver not recovering from reloading temp. unavailable code location HOT 1
- Wrong example in source code documention of `build_asset_job`
- Performance problems when resolving asset job definitions
- 'Runs' page usability HOT 1
- Cannot configure partitioned job on the launchpad. HOT 1
- Global asset page not showing partitioned assets as unsynced even though partitions are out of sync HOT 10
- Updating Great Expectation Intergration
- Intermittent step failures after step success on k8s executor
- Code version of dagster dbt assets do not respect code of dependent macros and ephemeral models
- Add `dagster-cloud` and `dagster-insights` to `dagster-feedstock` on conda-forge HOT 1
- Allow metadata specification when manually reporting materialization from the UI
- Problem finding datetime_ambiguous on EMR HOT 5
- Change tags on existing operations
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from dagster.