Comments (8)
@garethbrickman Would really appreciate if if you could make some time for this, seems like a relatively major issue.
from dagster.
@ion-elgreco thanks for the reproduction, we should be able to get a fix in for next week's release
from dagster.
@garethbrickman Can someone look into this? This breaks all IO managers implementations for source assets
from dagster.
Hi @sverbruggen @ion-elgreco , are either of you able to create a minimal reproduction of this issue? My attempts haven't been successful here, generally trying things along these lines:
def test_input_manager_with_source_assets() -> None:
fancy_metadata = {"foo": "bar", "baz": 1.23}
upstream = SourceAsset("upstream", metadata=fancy_metadata)
@asset(ins={"upstream": AssetIn(input_manager_key="special_io_manager")})
def downstream(upstream) -> int:
return upstream + 1
class MyIOManager(IOManager):
def load_input(self, context) -> int:
assert context.upstream_output is not None
assert context.upstream_output.asset_key == AssetKey(["upstream"])
assert context.upstream_output.definition_metadata == fancy_metadata
return 2
def handle_output(self, context, obj) -> None: ...
defs = Definitions(
assets=[upstream, downstream],
resources={"special_io_manager": IOManagerDefinition.hardcoded_io_manager(MyIOManager())},
)
job = defs.get_implicit_job_def_for_assets([downstream.key])
assert job is not None
output = job.execute_in_process()
assert output._get_output_for_handle("downstream", "result") == 3 # noqa: SLF001
assert False
There was a change which resulted in those system-generated metadata keys (e.g. dagster/asset_execution_type
) no longer appearing, but that wouldn't impact the user-provided metadata as you're experiencing.
from dagster.
@OwenKephart I have an MRE in my code base, will share it when I arrive back home tonight!
from dagster.
Thank you!
from dagster.
@OwenKephart found a way to reproduce it with a slight modification of your example:
from dagster import (
AssetIn,
AssetKey,
ConfigurableIOManager, # noqa
DataVersion,
IOManager, # noqa
asset,
materialize,
observable_source_asset,
)
def test_input_manager_with_source_assets() -> None:
fancy_metadata = {"foo": "bar", "baz": 1.23}
@observable_source_asset(metadata=fancy_metadata)
def upstream():
return DataVersion('1')
# upstream = SourceAsset("upstream", metadata=fancy_metadata)
@asset(ins={"upstream": AssetIn(input_manager_key="special_io_manager")})
def downstream(upstream) -> int:
return upstream + 1
class MyIOManager(IOManager):
def load_input(self, context) -> int:
assert context.upstream_output is not None
assert context.upstream_output.asset_key == AssetKey(["upstream"])
assert context.upstream_output.definition_metadata == fancy_metadata
return 2
def handle_output(self, context, obj) -> None: ...
materialize(assets=[upstream,downstream], resources={"special_io_manager": MyIOManager()})
AssertionError: assert {'dagster/io_... 'io_manager'} == {'baz': 1.23, 'foo': 'bar'}
Left contains 1 more item:
{'dagster/io_manager_key': 'io_manager'}
Right contains 2 more items:
{'baz': 1.23, 'foo': 'bar'}
Use -v to get more diff
The key is to use observable source assets, and materialize it instead of a job
from dagster.
@OwenKephart thanks!! Much appreciated for the quick fix
from dagster.
Related Issues (20)
- DbtProject does not take profiles dir for dbt parse
- dagster fails to load simple code location after 1.8 upgrade HOT 19
- Dagster schedule ticks that fail and retry should indicate in the tick timeline that they retried HOT 4
- switching schedule on and off in the UI shows graphQL error HOT 2
- Problem with op build not using kwargs HOT 1
- Unable to process empty dataframe with DeltaLakePandasIOManager for partitioned asset
- JSONDecodeErrors failing jobs HOT 5
- Unexpected GraphQL Error. Error when activate/deactivate schedule via Dagster Web UI HOT 2
- Incompatible with polars >= 1.4.0 HOT 1
- `dagster api grpc` with empty Code Location fails without Warning
- Subsettable multi-assets cannot be directly invoked
- K8sRunLauncher to infer kubeconfig at runtime
- Partition mapping for each week of the current month HOT 1
- bug while turning off automation condition sensor through UI HOT 1
- New API authentication option for Airbyte OSS HOT 4
- Attempting to show unified asset health for assets with dimension of different lengths HOT 1
- `AutomationCondition` support for `dbt_assets` HOT 1
- Logs generated during evaluation of sensor not shown in UI if they originate from within a configurable resource
- DuckDB connection_config parameter not being recognized HOT 2
- retry_on_asset_or_op_failure = false is not working HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from dagster.