HTTPSConnectionPool(host='bigquery.googleapis.com', port=443): Max retries exceeded with url about dbt-bigquery HOT 3 CLOSED

@ShahNewazKhan If possible, could you include the debug-level logs that come before the error message you've pasted above? I think you're right, that's coming from one of dbt's dependencies, and not dbt itself.

There is some logic inside the dbt-bigquery's BigQueryConnectionManager that will retry queries if they match a set of reopenable/retryable errors. This leverages logic from the retry module of google.api_core.

I'm a bit surprised to hear that you're running 10 concurrent invocations of dbt against BigQuery, from 10 separate k8s/Airflow pods. This is an unsupported pattern: dbt does not work well with concurrency that is externally orchestrated, and all bets are off if you try running the same model from multiple different invocations. This error could fall under the same umbrella. Why 10 separate invocations/instances, as opposed to one invocation/instance with 10+ threads?

from dbt-bigquery.

Why 10 separate invocations/instances, as opposed to one invocation/instance with 10+ threads?
Thanks @jtcohen6 for looking into this! The concurrent runs occur when we are running data backfills since dbt is a task in our ariflow dag that spins up dbt as a k8s pod.

I will try to gather the debug logs as this in happening in our GKE deployed instances where we are performing this backfills like staging and prod however we do not need to perform backfills in our dev or local environments.

from dbt-bigquery.

This issue has been marked as Stale because it has been open for 180 days with no activity. If you would like the issue to remain open, please remove the stale label or comment on the issue, or it will be closed in 7 days.

from dbt-bigquery.