Assign ECS cpu/memory in tags on a per-asset basis about dagster HOT 5 OPEN

I would happily work on this with a nudge in the right direction.

from dagster.

@leejlFG, I see your point about these being only run_tags.

I hadn't considered that aspect before, good point 🤔 An ECSStepLauncher would be nice to have, but it's not a requirement for me. It's acceptable if all assets for the same run share the same resources.

What I care most about is having some sort of tag per asset so that auto-materializations/ad-hoc runs are launched with the necessary resources.

from dagster.

Would this work for you? #14544

from dagster.

The lack of this feature is a significant drawback for us because we would have to apply the highest hardware requirements of all assets to every run, regardless of whether they're needed and, therefore, overprovision.
In addition, it would also be beneficial to provide capacityProvider on an asset-level basis.

@leejlFG What I don't understand is why this isn't working. Shouldn't the tags be applied with the get_cpu_and_memory_overrides function?

from dagster.

@yuhan using op_tags would be fine for our purposes, but since we use the ECSRunLauncher the maximum resources for any asset/op in the run needs to take priority for the entire ECS task, which is exactly the issue @ammerzon describes.

I may be wrong, but op_tags and plain tags on assets don't propagate and merge with run_tags in the ECSRunLauncher. Tags applied to jobs are run_tags, nothing else is. The get_cpu_and_memory_overrides function references only the run_tags. If one new set of tags needs to be merged with run_tags I think it makes the most sense to avoid using op_tags as they are basically not used at all outside k8s and op-based workflows.

@ammerzon you really need an ECSStepLauncher instead to have different assets inside the same run spin up with different resources, right?

Getting some sort of tag on assets so ad hoc and automaterializations launch with the necessary resources instead of the default ones sounds like a different but related problem than getting individual assets within a run to spawn with fresh containers in a step launcher, also with those specified resources.

from dagster.