Comments (3)
@ShahNewazKhan If possible, could you include the debug-level logs that come before the error message you've pasted above? I think you're right, that's coming from one of dbt's dependencies, and not dbt itself.
There is some logic inside the dbt-bigquery
's BigQueryConnectionManager that will retry queries if they match a set of reopenable/retryable errors. This leverages logic from the retry
module of google.api_core
.
I'm a bit surprised to hear that you're running 10 concurrent invocations of dbt against BigQuery, from 10 separate k8s/Airflow pods. This is an unsupported pattern: dbt does not work well with concurrency that is externally orchestrated, and all bets are off if you try running the same model from multiple different invocations. This error could fall under the same umbrella. Why 10 separate invocations/instances, as opposed to one invocation/instance with 10+ threads?
from dbt-bigquery.
Why 10 separate invocations/instances, as opposed to one invocation/instance with 10+ threads?
Thanks @jtcohen6 for looking into this! The concurrent runs occur when we are running data backfills since dbt is a task in our ariflow dag that spins up dbt as a k8s pod.
I will try to gather the debug logs as this in happening in our GKE deployed instances where we are performing this backfills like staging and prod however we do not need to perform backfills in our dev or local environments.
from dbt-bigquery.
This issue has been marked as Stale because it has been open for 180 days with no activity. If you would like the issue to remain open, please remove the stale label or comment on the issue, or it will be closed in 7 days.
from dbt-bigquery.
Related Issues (20)
- [Bug] many timeouts with `priority: interactive` config HOT 7
- [Bug] Incremental models don't update new elements in struct column
- [Regression] `docs generate` raises error when querying dataset with BQ ML models HOT 1
- dbt bigquery 1.3 Flake8 is failing a mypy check
- [Bug] Cannot parse partition config when specified in property YAML file HOT 2
- [Test] testing workflow
- [Tech Debt] Run all python model tests in sequence across runs
- [Feature] Retrieving and using table/column descriptions from BigQuery
- [Bug] Using `grants` on `materialized_view` raises an error
- [Bug] Build verification is failing HOT 1
- [Bug] Updating partitioning/clustering on a table self referencing itself will fail and delete the table
- [Bug] Dataproc (Python models) do not retry polling on 5xx errors HOT 1
- [Bug] Data mix-up in struct with enforced contracts
- [Bug] Nested record field is not quoted and causes build error when contract enforced enabled. HOT 1
- [Bug] `dbt show` integration test using JSON struct is failing for `1.8`
- Update the build check workflow
- [Bug] spark_catalog requires a single-part namespace in dbt python incremental model
- [Bug] Jobs timing out early regardless of job_execution_timeout_seconds
- [Bug] Bug in partitions_match method causing tables to be dropped for full refresh even though partitioning spec did not change
- [Bug] is_incremental always return False when use Project Number instead of Project ID HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from dbt-bigquery.