Comments (3)
@temminks, Ok, I think I've found the issue with what you've shared above. Both @sling_assets
and replicate()
take a DagsterSlingTranslator implementation. You've only added it to the replicate
call. If you change this to the following it should work as expected.
# before
@sling_assets(replication_config=replication_config)
def gfdi_to_transformator_assets(context, sling: SlingResource):
yield from sling.replicate(
replication_config=replication_config,
dagster_sling_translator=DagsterSlingTranslator(target_prefix="my_prefix"),
)
# after
custom_dagster_sling_translator = DagsterSlingTranslator(target_prefix="my_prefix")
@sling_assets(replication_config=replication_config, dagster_sling_translator=custom_dagster_sling_translator)
def gfdi_to_transformator_assets(context, sling: SlingResource):
yield from sling.replicate(
replication_config=replication_config,
dagster_sling_translator=custom_dagster_sling_translator,
)
I believe that we can remove the dependency to pass the translator in two locations; I will create a ticket to track such an enhancement.
from dagster.
EDIT: disregard this message, and see the comment below!
Hi @temminks - thanks again for reporting this.
If you override the class without the @dataclass
decorator, then the overridden target_prefix
attribute will not be set, however, the use you showed with DagsterSlingTranslator(target_prefix='custom')
seems like it should work -- I will investigate further.
from dataclasses import dataclass
@dataclass
class ExampleTranslator:
target_prefix: str = "target"
t1 = ExampleTranslator()
t1.target_prefix
# 'target'
@dataclass # <-- with dataclass decorator
class Custom(ExampleTranslator):
target_prefix: str = "custom"
t2 = Custom()
t2.target_prefix
# 'custom' <-- 👍
class CustomNotDataclass(DagsterSlingTranslator):
target_prefix: str = "custom"
t3 = CustomNotDataclass()
t3.target_prefix
# 'target' <-- 👎
t4 = DagsterSlingTranslator(target_prefix='custom')
t4.target_prefix
# 'custom' <-- 👍
from dagster.
Just to follow up here, a code change was pushed that makes it so you only have to specify the translator and replication config in the asset decorator. This is to avoid the confusion of the above scenario. You can see the details here:
This change also makes it so you can use the Sling integration from an @op
directly if desired.
from dagster.
Related Issues (20)
- Conflicting names of proto files HOT 1
- Ability to add `storage_kind` tag for dlt and Sling assets HOT 1
- Robust ignore patterns for `code.zip` in pyspark step launcher
- [dagster-k8s] Dagster K8s Pipes Client hangs when Pipes pod is unexpectedly deleted
- [UI] In Asset Graph, asset cards with checks extend past the group border HOT 1
- dbt_select when building schedules not working HOT 2
- Loading dbt manifest file fails when it contains semantic_model ressource type
- [Documentation Feedback] Problem on /dagster-plus/developing-testing/code-locations page HOT 1
- AutoMaterialize "cooldown": AutoMaterializePolicy.eager(cooldown_seconds=300), with SKIP instead of DISCARD
- Recursive graph
- Please provide an example for run partial ops of a job HOT 1
- Make it more clear when "test sensor" button isn't available
- Readonly webserver not recovering from reloading temp. unavailable code location HOT 1
- Wrong example in source code documention of `build_asset_job`
- Performance problems when resolving asset job definitions
- 'Runs' page usability HOT 1
- Cannot configure partitioned job on the launchpad. HOT 1
- Global asset page not showing partitioned assets as unsynced even though partitions are out of sync HOT 10
- Updating Great Expectation Intergration
- Intermittent step failures after step success on k8s executor
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from dagster.