Comments (4)
@ivan-fishbrain Thanks for the issue. As it is, BigQuery doesn't support string-type partition columns, so I do think it makes sense to raise some kind of error here. The question is whether BigQuery should be the one to raise that error (as it's doing here), or if the dbt-bigquery
adapter should be doing it.
The BigQuery error would be clearer if this were your first time running the model, to the effect of:
PARTITION BY expression must be DATE(<timestamp_column>), DATE(<datetime_column>), DATETIME_TRUNC(<datetime_column>, DAY/HOUR/MONTH/YEAR), a DATE column, TIMESTAMP_TRUNC(<timestamp_column>, DAY/HOUR/MONTH/YEAR), DATE_TRUNC(<date_column>, MONTH/YEAR), or RANGE_BUCKET(<int64_column>, GENERATE_ARRAY(<int64_value>, <int64_value>[, <int64_value>]))
Alternatively, dbt-bigquery could be more aggressive about validating partition column types in the PartitionConfig, and raise an error if it differs from the supported values. I think that's a reasonable approach, since we already hard-code a lot of type-logic in the render
method; it just means that we'll need a PR to the dbt-bigquery
adapter if ever / whenever BigQuery adds support for partitioning by a new datatype.
I'm going to transfer the issue over to the dbt-bigquery
repo, since that's where the change will need to happen. Here's some pseudo-code to get you started:
class PartitionConfig(dbtClassMixin):
...
def validate(cls, data):
super().validate(data)
dtype = data.get('data_type')
supported = ("date", "timestamp", "datetime", "int")
if dtype not in supported:
raise ValidationError(f"Got partition data type {dtype}, expected one of {supported}")
from dbt-bigquery.
@jtcohen6 Thank you for explanation! I wasn't aware about impossibility of creation string-partitioned tables.
Then, I think dbt should at least create warn about this impossibility explicitly.
from dbt-bigquery.
This issue has been marked as Stale because it has been open for 180 days with no activity. If you would like the issue to remain open, please remove the stale label or comment on the issue, or it will be closed in 7 days.
from dbt-bigquery.
I think you can partition on anything now. And also under no circumstances should dbt-core be calling a function that's not even supported in bigquery, surely!
from dbt-bigquery.
Related Issues (20)
- [Test] testing workflow
- [Tech Debt] Run all python model tests in sequence across runs
- [Feature] Retrieving and using table/column descriptions from BigQuery
- [Bug] Using `grants` on `materialized_view` raises an error
- [Bug] Build verification is failing HOT 1
- [Bug] Updating partitioning/clustering on a table self referencing itself will fail and delete the table
- [Bug] Dataproc (Python models) do not retry polling on 5xx errors HOT 1
- [Bug] Data mix-up in struct with enforced contracts
- [Bug] Nested record field is not quoted and causes build error when contract enforced enabled. HOT 1
- [Bug] `dbt show` integration test using JSON struct is failing for `1.8`
- Update the build check workflow
- [Bug] spark_catalog requires a single-part namespace in dbt python incremental model
- [Bug] Jobs timing out early regardless of job_execution_timeout_seconds
- [Bug] Bug in partitions_match method causing tables to be dropped for full refresh even though partitioning spec did not change
- [Bug] is_incremental always return False when use Project Number instead of Project ID HOT 2
- [Feature] Support Continuous SQL Queries in dbt HOT 1
- [Bug] Reserved words cause contract enforcement query to fail
- [Bug] 'dbt docs generate' creates catalog for the relations not used in the models HOT 1
- [Regression] Python model build failed when Google bucket retention policy is set (starting from dbt version 1.7)
- [Feature] Spike on supporting Py3.12 in dbt-bigquery HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from dbt-bigquery.