`run_retries/max_retries` should not trigger on manually terminated jobs about dagster HOT 3 CLOSED

@mjkanji the intended behavior here is that run retries should only trigger for runs that failed (status "Failure" in the UI:

not runs that were manually terminated from the Dagster UI or using the GraphQL API (status "Canceled" in the UI:
)

It sounds like this is different than what you're experiencing though - can you provide debug information for a run where this happened on a manually terminated run using the "Download debug information" dropdown on the runs page?

Or if it's a run in Dagster Cloud you could also email [email protected] with a link to the run.

from dagster.

Hi @gibsondan, please see the screenshot below. I terminated the run (see the DagsterExecutionInterruptedError in the logs). However, as you can see from the right side pane, a child run was automatically started in the background, which is showing up as a "re-execution" in the Runs page like a retry normally would.

Unless I somehow pressed the "re-execute" button by mistake (which, I guess, is possible), as far as I can tell, the child run was triggered automatically. The run is also showing up as a Failure and not Canceled. I'm not sure why that's the case.

I've also emailed [email protected] with the links to the actual runs for you to review directly.

from dagster.

We identified the issue here (the pod not getting terminated cleanly in time before the process was killed, then our monitoring functionality incorrectly moving it into a failure state instead of a cancelled state, triggering a retry) - and rolled out a fix for this in the most recent cloud release. No upgrade to anything required if you're using cloud. Thanks for the report!

from dagster.