Backfilling of partitions when gRPC location is lost about dagster HOT 3 OPEN

I can simply overcome this by setting :

AutoMaterializePolicy.eager().without_rules(
     AutoMaterializeRule.materialize_on_missing()
)

But it is still pretty confusing.

from dagster.

Ok so, seems that restart of location is for some reason firing Upstream data has changed since last materialization if previous materializations had failed:

from dagster.

Hi @tomsej, I'm not entirely sure why this is only happening on gRPC reconnection, but if you have missing partitions (due to failed materializations in the past that have not since been backfilled) and AutoMaterializePolicy.eager(), then it is expected that the auto-materialize daemon will attempt to backfill these. Your turning off materialize_on_missing() is the correct solution.

That said, we realize many users find auto-backfilling of historical partitions confusing and are working on improvements in this area, with auto-materializations focused on the most partition only. Stay tuned!

from dagster.