Expecting all purpose cluster stop be utilised when configured job cluster detail about dbt-databricks HOT 3 OPEN

This is expected behavior, as python models are integrated into the rest of your dbt project using SQL (for example, on an incremental model, the merge behavior is conducted in SQL), and that SQL would be executed on the AP Cluster. We are investigating ways for python model behavior to be more 'spark-like', but for now I would say this is an enhancement request, rather than a bug, as it is consistent with the structure imposed by dbt-core.

from dbt-databricks.

Thanks, Benc. It clears my doubts.

from dbt-databricks.

@benc-db Would it be possible to use a more simple approach when running a python model inside a job cluster like following:

1. dbt creates a new notebook for the python model
2. the new notebook is executed withing dbt using python command dbutils.notebook.run("....") (see Run a Databricks notebook from another notebook) inside a own process

I am not sure but it looks to me, that the strict seperation between execution (dbt python code) and the model execution (putting model into an isolated space) seems to be a bit oversized on Databricks job clusters, because the job will run nevertheless on spark on the master node. But maybe I am not getting the full picture of this issue...

from dbt-databricks.