Git Product home page Git Product logo

Comments (8)

sverbruggen avatar sverbruggen commented on August 27, 2024 1

@garethbrickman Would really appreciate if if you could make some time for this, seems like a relatively major issue.

from dagster.

OwenKephart avatar OwenKephart commented on August 27, 2024 1

@ion-elgreco thanks for the reproduction, we should be able to get a fix in for next week's release

from dagster.

ion-elgreco avatar ion-elgreco commented on August 27, 2024

@garethbrickman Can someone look into this? This breaks all IO managers implementations for source assets

from dagster.

OwenKephart avatar OwenKephart commented on August 27, 2024

Hi @sverbruggen @ion-elgreco , are either of you able to create a minimal reproduction of this issue? My attempts haven't been successful here, generally trying things along these lines:

def test_input_manager_with_source_assets() -> None:
    fancy_metadata = {"foo": "bar", "baz": 1.23}

    upstream = SourceAsset("upstream", metadata=fancy_metadata)

    @asset(ins={"upstream": AssetIn(input_manager_key="special_io_manager")})
    def downstream(upstream) -> int:
        return upstream + 1

    class MyIOManager(IOManager):
        def load_input(self, context) -> int:
            assert context.upstream_output is not None
            assert context.upstream_output.asset_key == AssetKey(["upstream"])
            assert context.upstream_output.definition_metadata == fancy_metadata

            return 2

        def handle_output(self, context, obj) -> None: ...

    defs = Definitions(
        assets=[upstream, downstream],
        resources={"special_io_manager": IOManagerDefinition.hardcoded_io_manager(MyIOManager())},
    )
    job = defs.get_implicit_job_def_for_assets([downstream.key])
    assert job is not None
    output = job.execute_in_process()

    assert output._get_output_for_handle("downstream", "result") == 3  # noqa: SLF001
    assert False

There was a change which resulted in those system-generated metadata keys (e.g. dagster/asset_execution_type) no longer appearing, but that wouldn't impact the user-provided metadata as you're experiencing.

from dagster.

ion-elgreco avatar ion-elgreco commented on August 27, 2024

@OwenKephart I have an MRE in my code base, will share it when I arrive back home tonight!

from dagster.

OwenKephart avatar OwenKephart commented on August 27, 2024

Thank you!

from dagster.

ion-elgreco avatar ion-elgreco commented on August 27, 2024

@OwenKephart found a way to reproduce it with a slight modification of your example:

from dagster import (
    AssetIn,
    AssetKey,
    ConfigurableIOManager,  # noqa
    DataVersion,
    IOManager,  # noqa
    asset,
    materialize,
    observable_source_asset,
)


def test_input_manager_with_source_assets() -> None:
    fancy_metadata = {"foo": "bar", "baz": 1.23}

    @observable_source_asset(metadata=fancy_metadata)
    def upstream():
        return DataVersion('1')
    # upstream = SourceAsset("upstream", metadata=fancy_metadata)

    @asset(ins={"upstream": AssetIn(input_manager_key="special_io_manager")})
    def downstream(upstream) -> int:
        return upstream + 1

    class MyIOManager(IOManager):
        def load_input(self, context) -> int:
            assert context.upstream_output is not None
            assert context.upstream_output.asset_key == AssetKey(["upstream"])
            assert context.upstream_output.definition_metadata == fancy_metadata
            return 2

        def handle_output(self, context, obj) -> None: ...

    materialize(assets=[upstream,downstream], resources={"special_io_manager": MyIOManager()})
AssertionError: assert {'dagster/io_... 'io_manager'} == {'baz': 1.23, 'foo': 'bar'}
  Left contains 1 more item:
  {'dagster/io_manager_key': 'io_manager'}
  Right contains 2 more items:
  {'baz': 1.23, 'foo': 'bar'}
  Use -v to get more diff

The key is to use observable source assets, and materialize it instead of a job

from dagster.

ion-elgreco avatar ion-elgreco commented on August 27, 2024

@OwenKephart thanks!! Much appreciated for the quick fix

from dagster.

Related Issues (20)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.