Configuring kubernetes settings on a job through tags not working about dagster HOT 9 CLOSED

Is putting this configuration on the runK8sConfig field here an option? https://github.com/dagster-io/dagster/blob/master/helm/dagster/values.yaml#L573-L591 - that would cause them to be applied to all runs by default

from dagster.

I noticed this very old PR which may be related: #9940

from dagster.

The configuration there looks correct to me. A couple of follow-up questions:

• is it possible to post a debug file for a run with these tags? You can generate a debug file from this drop-down on the runs page

• Can you post the output of "kubectl get pod POD_NAME_HERE -o yaml" where the pod name is one of your job pods for this job?

I don't think that PR is related - the tags you describe there should be working.

from dagster.

I tried creating a run with the tags that you specified and was able to see the configuration being applied in the launched run pod (excerpt from the kubectl get pod command I posted above):

  tolerations:
  - effect: NoSchedule
    key: agentpool
    operator: Equal
    value: node04
  - effect: NoSchedule
    key: agentpool
    operator: Equal
    value: node01
  - effect: NoSchedule
    key: agentpool
    operator: Equal
    value: node05
  - effect: NoSchedule
    key: agentpool
    operator: Equal
    value: node02
  - effect: NoSchedule
    key: agentpool
    operator: Equal
    value: node03
  - effect: NoExecute
    key: node.kubernetes.io/not-ready
    operator: Exists
    tolerationSeconds: 300
  - effect: NoExecute
    key: node.kubernetes.io/unreachable
    operator: Exists
    tolerationSeconds: 300

So the first thing that I would check is to verify that those tags are actually getting applied to the run. You can check that from the "View tags" button here:

You should see something like this in the dialog that opens if the tag was applied as expected:

from dagster.

Hi @gibsondan,

It occurs when triggering a backfill materialization from the asset pane instead of the job pane, see result below:

from dagster.

Got it - I think that explains what's happening here. Triggering a backfill materialization from the asset page isn't going to use that job.

from dagster.

@gibsondan In that case it's a bug, right? We have these default settings on the user deployment, however they are not picked up

from dagster.

See this issue for reference: #9924

from dagster.

Yup, we will try out tomorrow! Thanks @gibsondan ❤️

from dagster.