Git Product home page Git Product logo

dbt_stripe_source's Introduction

Stripe Source dbt Package (Docs)

📣 What does this dbt package do?

  • Materializes Stripe staging tables which leverage data in the format described by this ERD. These staging tables clean, test, and prepare your Stripe data from Fivetran's connector for analysis by doing the following:
    • Name columns for consistency across all packages and for easier analysis
    • Adds freshness tests to source data
    • Adds column-level testing where applicable. For example, all primary keys are tested for uniqueness and non-null values.
  • Generates a comprehensive data dictionary of your Stripe data through the dbt docs site.
  • These tables are designed to work simultaneously with our Stripe transformation package.

🎯 How do I use the dbt package?

Step 1: Prerequisites

To use this dbt package, you must have the following:

  • At least one Fivetran stripe connector syncing data into your destination.
  • A BigQuery, Snowflake, Redshift, Databricks, or PostgreSQL destination.

Databricks Dispatch Configuration

If you are using a Databricks destination with this package you will need to add the below (or a variation of the below) dispatch configuration within your dbt_project.yml. This is required in order for the package to accurately search for macros within the dbt-labs/spark_utils then the dbt-labs/dbt_utils packages respectively.

dispatch:
  - macro_namespace: dbt_utils
    search_order: ['spark_utils', 'dbt_utils']

Step 2: Install the package (skip if also using the stripe transformation package)

If you are not using the Stripe transformation package, include the following package version in your packages.yml file. If you are installing the transform package, the source package is automatically installed as a dependency.

TIP: Check dbt Hub for the latest installation instructions or read the dbt docs for more information on installing packages.

packages:
  - package: fivetran/stripe_source
    version: [">=0.11.0", "<0.12.0"]

Step 3: Define database and schema variables

By default, this package runs using your destination and the stripe schema. If this is not where your stripe data is (for example, if your stripe schema is named stripe_fivetran), add the following configuration to your root dbt_project.yml file:

vars:
    stripe_database: your_destination_name
    stripe_schema: your_schema_name 

Step 4: Disable models for non-existent sources

This package takes into consideration that not every Stripe account utilizes the invoice, invoice_line_item, payment_method, payment_method_card, plan, price, subscription, or credit_note features, and allows you to disable the corresponding functionality. By default, all variables' values are assumed to be true with the exception of credit_note. Add variables for only the tables you want to disable or enable respectively:

# dbt_project.yml

...
vars:
    stripe__using_invoices:        False  #Disable if you are not using the invoice and invoice_line_item tables
    stripe__using_payment_method:  False  #Disable if you are not using the payment_method and payment_method_card tables
    stripe__using_subscriptions:   False  #Disable if you are not using the subscription and plan/price tables.
    stripe__using_credit_notes:    True   #Enable if you are using the credit note tables.

(Optional) Step 5: Additional configurations

Expand to view configurations

Unioning Multiple Stripe Connectors

If you have multiple Stripe connectors you would like to use this package on simultaneously, we have added the ability to do so. Data from disparate connectors will be unioned together and be passed downstream to the end models. The source_relation column will specify where each record comes from. To use this functionality, you will need to either set the stripe_union_schemas or stripe_union_databases variables. Please also make sure the single-source stripe_database and stripe_schema variables are removed.

# dbt_project.yml

...
config-version: 2

vars:
    stripe_union_schemas: ['stripe_us','stripe_mx'] # use this if the data is in different schemas/datasets of the same database/project
    stripe_union_databases: ['stripe_db_1','stripe_db_2'] # use this if the data is in different databases/projects but uses the same schema name

Leveraging Plan vs Price Sources

Customers using Fivetran with the newer Stripe Price API will have a price table, and possibly a plan table if that was used previously. Therefore to accommodate two different source tables we added logic to check if there exists a price table by default. If not, it will leverage the plan table. However if you wish to use the plan table instead, you may set stripe__using_price to false in your dbt_project.yml to override the macro.

# dbt_project.yml

...
config-version: 2

vars:
  stripe__using_price: false #  True by default. If true, will look `price ` table. If false, will look for the `plan` table. 

Leveraging Subscription Vs Subscription History Sources

For Stripe connectors set up after February 09, 2022 the subscription table has been replaced with the new subscription_history table. By default this package will look for your subscription data within the subscription_history source table. However, if you have an older connector then you must configure the stripe__using_subscription_history to false in order to have the package use the subscription source rather than the subscription_history table.

Please note that if you have stripe__using_subscription_history enabled then the package will filter for only active records.

vars:
    stripe__using_subscription_history: False  # True by default. Set to False if your connector syncs the `subscription` table instead. 

Running on Live vs Test Customers

By default, this package will run on non-test data (where livemode = true) from the source Stripe tables. However, you may want to include and focus on test data when testing out the package or developing your analyses. To run on only test data, add the following configuration to your root dbt_project.yml file:

vars:
    stripe_source:
        stripe__using_livemode: false  # Default = true

Including sub Invoice Line Items

By default, this package will filter out any records from the invoice_line_item source table which include the string sub_. This is due to a legacy Stripe issue where sub_ records were found to be duplicated. However, if you highly utilize these records you may wish they be included in the final output of the stg_stripe__invoice_line_item model. To do, so you may include the below variable configuration in your root dbt_project.yml:

vars:
    stripe_source:
        stripe__using_invoice_line_sub_filter: false # Default = true

Pivoting out Metadata Properties

Oftentimes you may have custom fields within your source tables that is stored as a JSON object that you wish to pass through. By leveraging the metadata variable, this package pivot out fields into their own columns. The metadata variables accept dictionaries in addition to strings.

Additionally, you may alias your field if you happen to be using a reserved word as a metadata field, any otherwise incompatible name, or just wish to rename your field. Below are examples of how you would add the respective fields.

The metadata JSON field is present within the customer, charge, card, dispute, invoice, invoice_line_item, payment_intent, payment_method, payout, plan, price, refund, subscription, and transfer source tables. To pivot these fields out and include in the respective downstream staging model, add the respective variable(s) to your root dbt_project.yml file like below.

vars: 
  stripe__account_metadata:
    - name: metadata_field
    - name: another_metadata_field
    - name: and_another_metadata_field
  stripe__charge_metadata:
    - name: metadata_field_1
  stripe__card_metadata:
    - name: metadata_field_10
  stripe__customer_metadata:
    - name: metadata_field_6
      alias: metadata_field_six
  stripe__dispute_metadata:
    - name: dispute_metadata_field
  stripe__invoice_metadata: 
    - name: metadata_field_2
  stripe__invoice_line_item_metadata: 
    - name: metadata_field_20
  stripe__payment_intent_metadata:
    - name: incompatible.field
      alias: rename_incompatible_field
  stripe__payment_method_metadata:
    - name: field_is_reserved_word
      alias: field_is_reserved_word_xyz
  stripe__payout_metadata:
    - name: 123
      alias: one_two_three
  stripe__price_plan_metadata: ## Used for both Price and Plan sources
    - name: rename_price
      alias: renamed_field_price
  stripe__refund_metadata:
    - name: metadata_field_3
  stripe__subscription_metadata:
    - name: 567
      alias: five_six_seven
  stripe__transfer_metadata:
    - name: transfer_metadata_field

Alternatively, if you only have strings in your JSON object, the metadata variable accepts the following configuration as well.

vars:
    stripe__subscription_metadata: ['the', 'list', 'of', 'property', 'fields'] # Note: this is case-SENSITIVE and must match the casing of the property as it appears in the JSON

Change the build schema

By default, this package builds the stripe staging models within a schema titled (<target_schema> + _stg_stripe) in your destination. If this is not where you would like your stripe staging data to be written to, add the following configuration to your root dbt_project.yml file:

models:
    stripe_source:
      +schema: my_new_schema_name # leave blank for just the target_schema

Change the source table references

If an individual source table has a different name than the package expects, add the table name as it appears in your destination to the respective variable:

IMPORTANT: See this project's dbt_project.yml variable declarations to see the expected names.

vars:
    stripe_<default_source_table_name>_identifier: your_table_name 

(Optional) Step 6: Orchestrate your models with Fivetran Transformations for dbt Core™

Expand to view details

Fivetran offers the ability for you to orchestrate your dbt project through Fivetran Transformations for dbt Core™. Learn how to set up your project for orchestration through Fivetran in our Transformations for dbt Core™ setup guides.

🔍 Does this package have dependencies?

This dbt package is dependent on the following dbt packages. Please be aware that these dependencies are installed by default within this package. For more information on the following packages, refer to the dbt hub site.

IMPORTANT: If you have any of these dependent packages in your own packages.yml file, we highly recommend that you remove them from your root packages.yml to avoid package version conflicts.

packages:
    - package: fivetran/fivetran_utils
      version: [">=0.4.0", "<0.5.0"]

    - package: dbt-labs/dbt_utils
      version: [">=1.0.0", "<2.0.0"]

    - package: dbt-labs/spark_utils
      version: [">=0.3.0", "<0.4.0"]

🙌 How is this package maintained and can I contribute?

Package Maintenance

The Fivetran team maintaining this package only maintains the latest version of the package. We highly recommend that you stay consistent with the latest version of the package and refer to the CHANGELOG and release notes for more information on changes across versions.

Contributions

A small team of analytics engineers at Fivetran develops these dbt packages. However, the packages are made better by community contributions!

We highly encourage and welcome contributions to this package. Check out this dbt Discourse article to learn how to contribute to a dbt package!

🏪 Are there any resources available?

  • If you have questions or want to reach out for help, please refer to the GitHub Issue section to find the right avenue of support for you.
  • If you would like to provide feedback to the dbt package team at Fivetran or would like to request a new dbt package, fill out our Feedback Form.
  • Have questions or want to be part of the community discourse? Create a post in the Fivetran community and our team along with the community can join in on the discussion!

dbt_stripe_source's People

Contributors

ashishkhaitan avatar dimoschi avatar erikmag avatar fivetran-chloe avatar fivetran-jamie avatar fivetran-joemarkiewicz avatar fivetran-reneeli avatar fivetran-sheringuyen avatar jeffsloan avatar jmussitsch avatar kristin-bagnall avatar lewisdavies avatar lijol avatar markmacardle avatar mvtemple avatar sachams avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

dbt_stripe_source's Issues

FEATURE - Update packages.yml to allow for dbt-utils 0.7.0

Are you a Fivetran customer?
Meera, Consultant, Ephemeral Tattoos

Is your feature request related to a problem? Please describe.
I'm trying to use dbt-utils 0.7.0 (the latest release) so that I can use dbt 0.20.0 with my M1 mac, but this is conflicting with the dbt-utils version requirement in my Fivetran package

Describe the solution you'd like
Change the version check to allow for dbt-utils 0.7.0

Describe alternatives you've considered

Additional context

Please indicate the level of urgency and business impact of this request

Are you interested in contributing to this package?

  • Yes, I can do this and open a PR for your review.
  • Possibly, but I'm not quite sure how to do this. I'd be happy to do a live coding session with someone to get this work implemented.
  • No, I'd prefer if someone else did this. I don't have the time and/or don't know how to incorporate the changes necessary.

FEATURE - Support History Mode

Are you a Fivetran customer?

Fivetran created FR

Is your feature request related to a problem? Please describe.

This feature request is not related to a problem. However, I could see this being an issue if certain customers have history mode enabled and want the package to run for just the latest records of the orders and subscriptions source tables.

Describe the solution you'd like

The Stripe release notes for February 2022 indicate that there are two new tables ORDER_HISTORY and SUBSCRIPTION_HISTORY. This FR is to include those two tables into the package and allow users to disable the history mode and only select the latest records.

Describe alternatives you've considered

Using the non-history ORDER and SUBSCRIPTION source tables.

FEATURE - [add metadata to customer object]

Are you a Fivetran customer?
Benoit Mayer, Stedi Inc

Is your feature request related to a problem? Please describe.
We store customer metadata in Stripe on the customers and would like it to flow through to reporting. the metadata is essential to our reporting.

Describe the solution you'd like
add metadata on customers as implemented for transactions as described https://github.com/fivetran/dbt_stripe_source/tree/v0.4.3#pivoting-out-metadata-properties

Describe alternatives you've considered

Please indicate the level of urgency and business impact of this request
not critical but time sensitive. by end of month would be ideal

Are you interested in contributing to this package?

  • Yes, I can do this and open a PR for your review.
  • Possibly, but I'm not quite sure how to do this. I'd be happy to do a live coding session with someone to get this work implemented.
  • No, I'd prefer if someone else did this. I don't have the time and/or don't know how to incorporate the changes necessary.

[Bug] Issue With Fee Unique Columns Test

Is there an existing issue for this?

  • I have searched the existing issues

Describe the issue

The test dbt_utils_unique_combination_of_columns_stg_stripe__fee_balance_transaction_id__source_relation is defined as:

- name: stg_stripe__fee
    description: The details of a fee associated with a balance transaction
    tests:
      - dbt_utils.unique_combination_of_columns:
          combination_of_columns:
            - balance_transaction_id
            - source_relation

However, the Fee table has an additional column index that is part of the composite primary key as described here:

https://docs.google.com/presentation/d/1zyxgbaOjgBt3NsY0OfsiGsWDIefcBc-R1lHWlMltCYU/edit#slide=id.g14afa1361c6_3_0

So this test is giving false positives

Relevant error log or model output

No response

Expected behavior

The expected behavior would be for this test to not give false positives and include the index column as part of the unique column set.

dbt Project configurations

name: 'chabi'
version: '1.0.0'
config-version: 2
profile: "{{ env_var('DBT_PROFILE', '') }}"
model-paths: ["models"]
analysis-paths: ["analyses"]
test-paths: ["tests"]
seed-paths: ["seeds"]
macro-paths: ["macros"]
snapshot-paths: ["snapshots"]
target-path: "target"
clean-targets:
  - "target"
  - "dbt_packages"
  - 
models:
  stripe:
    +schema:
    intermediate:
      +schema: stg
  stripe_source:
    +schema: stg
 
vars:
  stripe_schema: stripe
  stripe_timezone: Australia/Sydney
  stripe__using_credit_notes: true

Package versions

packages:
  - package: dbt-labs/dbt_utils
    version: [">=1.0.0", "<2.0.0"]
  - package: dbt-labs/codegen
    version: 0.9.0
  - package: fivetran/stripe
    version: 0.9.0

What database are you using dbt with?

snowflake

dbt Version

Core:
  - installed: 1.4.1
  - latest:    1.4.5 - Update available!

  Your version of dbt-core is out of date!
  You can find instructions for upgrading here:
  https://docs.getdbt.com/docs/installation

Plugins:
  - snowflake: 1.4.0 - Update available!

  At least one plugin is out of date or incompatible with dbt-core.
  You can find instructions for upgrading here:
  https://docs.getdbt.com/docs/installation

Additional Context

No response

Are you willing to open a PR to help address this issue?

  • Yes.
  • Yes, but I will need assistance and will schedule time during our office hours for guidance
  • No.

[Feature] Persist `phone` column in stg_stripe__customer

Is there an existing feature request for this?

  • I have searched the existing issues

Describe the Feature

We include the phone field in the get_customer_columns macro, but we do not include it in the staging model. We only include the shipping_phone but i believe both will be needed for our identity resolution project

Describe alternatives you've considered

just use shipping_phone, but shipping_phone is sometimes just the company phone, while phone is the individual's

Are you interested in contributing this feature?

  • Yes.
  • Yes, but I will need assistance and will schedule time during your office hours for guidance.
  • No.

Anything else?

No response

Disable individual tables

I want to disable the below tables from processing. Can you please let me know how to do it ?

  1. stg_stripe__card
  2. stg_stripe__balance_transaction
  3. stg_stripe__fee
  4. stg_stripe__payment_intent
  5. stg_stripe__payout

dbt_project .yml below

  • using_invoices: True #Disable if you are not using the invoice and invoice_line_item tables
  • using_subscriptions: True #Disable if you are not using the subscription and plan tables.
  • using_payment_method: False #Disable if you are not using the invoice and invoice_line_item tables
  • using_credit_notes: False #Disable if you are using the credit note tables.

[Feature] Restructure the tmp models to leverage the new empty table feature in union data

Is there an existing feature request for this?

  • I have searched the existing issues

Describe the Feature

The latest release of the fivetran_utils package allows us to instinctively infer if the source table the package is using is in fact present, if it is not the package will create an empty table to ensure the downstream transformations succeed.

However, the Stripe package is not currently compatible with this feature due to the livemode_predicate which exists in the *_tmp staging models. In order for the new empty data model creation to be successful in the Stripe package, we will need to move this predicate macro to the non *_tmp staging model.

Describe alternatives you've considered

Not leveraging the new empty model creation feature within the tmp models.

Are you interested in contributing this feature?

  • Yes.
  • Yes, but I will need assistance and will schedule time during your office hours for guidance.
  • No.

Anything else?

No response

Multiple Stripe databases

Hi,

I noticed in the docs you only have a single stripe database, is it possible to have multiple Stripe databases? For example to support multicurrency and not be charged transaction fees, you have to have multiple Stripe accounts.

So we have like:

stripe_foo
stripe_bar
stripe_baz

as our database names.

Missing rows (dbt run error)

Are you a current Fivetran customer?
Yes

Describe the bug
Found 5k rows in my raw Payment Intents table that are missing from the staging Payment Intents table. Realized they were in the tmp view, meaning that I hadn't re-run dbt.

Steps to reproduce

 select count(*) from raw_data.stripe.payment_intent raw
 left join analytics_data.dbt_meera.stg_stripe__payment_intent stg on raw.id = stg.payment_intent_id
 where stg.payment_intent_id is null

Expected behavior
I'd expect no missing rows across my raw data and staging data

Project variables configuration

version: '1.0'

require-dbt-version: ">=0.14.0"

source-paths: ["models"]
analysis-paths: ["analysis"]
test-paths: ["tests"]
data-paths: ["data"]
macro-paths: ["macros"]

target-path: "target"
clean-targets:
    - "target"
    - "dbt_modules"

quoting:
  database: false
  identifier: false
  schema: false

models:
  transient: true
  stripe_source: 
    +schema:   # Left blank so that will write to schema "dbt"

on-run-end:
  - "{{ grant_select_on_schemas(schemas, 'reporter') }}"
  - "{{ grant_select_on_schemas(schemas, 'transformer') }}"

config-version: 2

vars:
    # Defined variables for Fivetran Stripe package
    stripe_schema: stripe
    stripe_database: raw_data
    using_invoices:        False  #Disabled since we are not using the invoice and invoice_line_item tables
    using_subscriptions:   False  #Disabled since we are not using the subscription and plan tables.

Package Version

packages:
  - package: fishtown-analytics/dbt_utils
    version: 0.6.5
    
  - package: fivetran/stripe_source
    version: 0.3.1

Warehouse

  • BigQuery
  • Redshift
  • Snowflake
  • Postgres
  • Databricks
  • Other (provide details below)

[Feature] Include the subscription_id to the invoice table

Is there an existing feature request for this?

  • I have searched the existing issues

Describe the Feature

It would be beneficial to add the subscription_id field to the invoice staging table. This field could then be leveraged in downstream transformations.

Describe alternatives you've considered

Fork the package and leverage this field or use the source directly.

Are you interested in contributing this feature?

  • Yes.
  • Yes, but I will need assistance and will schedule time during your office hours for guidance.
  • No.

Anything else?

No response

stg_stripe__subscription

Is there an existing issue for this?

  • I have searched the existing issues

Describe the issue

The model is still using stg_stripe__subscription. This is throwing error as the table doesn't not exist . It has been changed to stg_stripe__subscription_history

Relevant error log or model output

STRIPE.SUBSCRIPTION' does not exist or not authorized.

Expected behavior

The model is expected to run fine

dbt Project configurations

stripe_database: FIVETRAN_DB
stripe_schema: STRIPE

Package versions

0.7.2

What database are you using dbt with?

snowflake

dbt Version

1.1.1

Additional Context

No response

Are you willing to open a PR to help address this issue?

  • Yes.
  • Yes, but I will need assistance and will schedule time during our office hours for guidance
  • No.

[Feature] Add databricks compatibility

Is there an existing feature request for this?

  • I have searched the existing issues

Describe the Feature

Internal - Add databricks compatibility

Describe alternatives you've considered

No response

Are you interested in contributing this feature?

  • Yes.
  • Yes, but I will need assistance and will schedule time during your office hours for guidance.
  • No.

Anything else?

No response

customers missing when running package on test data

Hi there, I would like to run this package on our stripe prod data aswell as the test data (using the fivetran test connector). I can override schema names accordingly. But test customers will never be pulled in because their livemode flag will be set to false. Would it be worth adding some configuration option to enable to run this package on test data?
Just wondering if there are some considerations that I'm not aware of?
many thanks,
Christoph.

[Feature] Support FAL target.type

Is there an existing feature request for this?

  • I have searched the existing issues

Describe the Feature

As identified within the dbt_stripe Issue #45, it seems that the fal target.type is not supported within conditionals found throughout the package. In particular, the get_plan_columns conditional is exclusively looking for the target.type in BigQuery, Spark, or Databricks. However, if a user has the fal adapter, then it will fail as the quoting is needed but the adapter is not included in the conditional.

This feature is to explore supporting the fal adapter and how it differs from other warehouses.

Describe alternatives you've considered

Do not use the fal adapter.

Are you interested in contributing this feature?

  • Yes.
  • Yes, but I will need assistance and will schedule time during your office hours for guidance.
  • No.

Anything else?

No response

[Feature] Add status columns to the invoice table

Is there an existing feature request for this?

  • I have searched the existing issues

Describe the Feature

The below columns to be added
STATUS_TRANSITIONS_FINALIZED_AT
STATUS_TRANSITIONS_PAID_AT
STATUS_TRANSITIONS_VOIDED_AT
STATUS_TRANSITIONS_MARKED_UNCOLLECTIBLE_AT

Describe alternatives you've considered

None

Are you interested in contributing this feature?

  • Yes.
  • Yes, but I will need assistance and will schedule time during your office hours for guidance.
  • No.

Anything else?

No response

FEATURE - subscription_item

Is there an existing feature request for this?

  • I have searched the existing issues

Describe the Feature

The new model for subscription item to be added

Describe alternatives you've considered

No response

Are you interested in contributing this feature?

  • Yes.
  • Yes, but I will need assistance and will schedule time during your office hours for guidance.
  • No.

Anything else?

No response

[Bug] Uniqueness test on unique_invoice_line_item_id

Is there an existing issue for this?

  • I have searched the existing issues

Describe the issue

An error in my Fivetran/Stripe package today with uniqueness in stg_stripe_invoice_line_items model.
Seems to be testing for uniqueness on unique_invoice_line_item_id but I have a few records where it is not unique through the data source.

Relevant error log or model output

17:49:29 3 of 3 START test unique_stg_stripe__invoice_line_item_unique_invoice_line_item_id  [RUN]
17:49:30 3 of 3 FAIL 3 unique_stg_stripe__invoice_line_item_unique_invoice_line_item_id . [FAIL 3 in 1.09s]

Expected behavior

As this stg_stripe_invoice_line_items has 2 PK's I expect the unique_stg_stripe__invoice_line_item_unique_invoice_line_item_id to not exist.

dbt_utils_unique_combination_of_columns_stg_stripe__invoice_line_item_invoice_line_item_id__invoice_id__source_relation is already there to ensure the combo is unique

dbt Project configurations

  #### Variables for Fivetran/stripe package ####
  # stripe_database: database
  stripe_schema: source_ft_stripe 

  stripe__using_invoices:        True  #Disable if you are not using the invoice and invoice_line_item tables
  stripe__using_payment_method:  False  #Disable if you are not using the payment_method and payment_method_card tables
  stripe__using_subscriptions:   True  #Disable if you are not using the subscription and plan/price tables.
  stripe__using_credit_notes:    False   #Enable if you are using the credit note tables.

Package versions

  - package: fivetran/stripe
    version: [">=0.12.0", "<0.13.0"]

What database are you using dbt with?

bigquery

dbt Version

18:16:55 Running with dbt=1.6.11

Additional Context

Screenshot 2024-04-01 at 11 21 32 AM

I think you just need to remove this test?

unnamed (2)

Are you willing to open a PR to help address this issue?

  • Yes.
  • Yes, but I will need assistance and will schedule time during our office hours for guidance
  • No.

[Feature] don't filter out deleted customers + persist the `is_deleted` field in `stg_stripe__customer`

Is there an existing feature request for this?

  • I have searched the existing issues

Describe the Feature

Remove this filter and persist the is_deleted field so that deleted customers continue downstream (and we can tell that they're deleted).

this will provide more complete data to places where we join in customer data (like here) and will allow for better matching in the current Customer360 project.

note: this will increase the # of rows in the stripe__customer_overview table, but likely not by a huge amount.

also this will not require a change to the transform package, as the is_deleted field will be automatically picked up by this

Describe alternatives you've considered

No response

Are you interested in contributing this feature?

  • Yes.
  • Yes, but I will need assistance and will schedule time during your office hours for guidance.
  • No.

Anything else?

No response

[Feature] Provide dispatch for macros

Is there an existing issue for this?

  • I have searched the existing issues

Describe the issue

We should leverage the adapter dispatch for all macros in order to allow customers to override them when needed. (livemode_predicate )

Relevant error log or model output

No response

Expected behavior

We can test that the macro can be overridden using the dispatch config.

dbt Project configurations

na

Package versions

na

What database are you using dbt with?

other (mention it in "Additional Context")

dbt Version

na

Additional Context

No response

Are you willing to open a PR to help address this issue?

  • Yes.
  • Yes, but I will need assistance and will schedule time during our office hours for guidance
  • No.

FEATURE - [Identifiers for multiple stripe connections]

Are you a Fivetran customer?
Yes. Ramiz Bozai, Senior Data Analyst at Cerebral

Is your feature request related to a problem? Please describe.
It would be great to have an identifier denoting which data/records came from which source as I union together our different stripe tables from different connections.

Describe the solution you'd like
Having a column indicating what the source of the data is (like country) would allow us to filter that data based on the column.

Describe alternatives you've considered
This is more of a nice to have, but I understand I can possible fork this package and potentially try to add this myself.

Additional context
N/A

Please indicate the level of urgency and business impact of this request
Not urgent. Just a nice to have feature.

Are you interested in contributing to this package?

  • Yes, I can do this and open a PR for your review.
  • Possibly, but I'm not quite sure how to do this. I'd be happy to do a live coding session with someone to get this work implemented.
  • No, I'd prefer if someone else did this. I don't have the time and/or don't know how to incorporate the changes necessary.

[Bug] Stripe `created` field for balance transactions should explicitly be defined as UTC

Is there an existing issue for this?

  • I have searched the existing issues

Describe the issue

The current definition of the created field within the balance_transaction source should explicitly define the timezone the timestamp is reported within. The stripe connector syncs this field as UTC and we should address it as such.

Relevant error log or model output

N/A

Expected behavior

The field definition details the UTC timezone.

dbt Project configurations

N/A

Package versions

packages:

  • package: fivetran/stripe_source
    version: [">=0.6.0", "<0.7.0"]

What database are you using dbt with?

bigquery

dbt Version

v 1.1.0

Additional Context

No response

Are you willing to open a PR to help address this issue?

  • Yes.
  • Yes, but I will need assistance and will schedule time during our office hours for guidance
  • No.

Feedback needed

where id not like 'sub%' -- ids starting with 'sub' are temporary and are replaced by permanent ids starting with 'sli'

Is is possible to provide some feedback on this staging model and the exclusion of invoice line items starting with sub_? Trying to find out in Stripe's documentation, I wasn't able to find something relevant. Also according to Stripe:

There are actually 3 types of ID that can appear in the invoice_line_item. What type of ID appears depends on where the item has been added.

sub_ - The ID associated with a subscription on a customer object.
si_ - The ID associated with a subscription item. Typically used for multiplan subscriptions, where the plan becomes the subscription item.
sli_ - The ID associated with a subscription line item. It essentially defines a subscription item on an invoice.

So, it seems that invoice line items starting with sub_ are related to subscriptions, so in stripe__subscription_line_items model of dbt_stripe those are excluded even thought it seems like they shouldn't.

[Feature] Update staging models to standard renaming practices

Is there an existing feature request for this?

  • I have searched the existing issues

Describe the Feature

Field renaming should take place in staging models (ex, the in the account staging model, rename id to account_id and name to account_name). We've done this in a few places but not all.

However note this will have downstream implications as most of the renaming occurs in the transforms package.

Describe alternatives you've considered

No response

Are you interested in contributing this feature?

  • Yes.
  • Yes, but I will need assistance and will schedule time during your office hours for guidance.
  • No.

Anything else?

No response

FEATURE - Allow for metadata fields to be included in staging models

Are you a Fivetran customer?

Original request identified by @sachams

Is your feature request related to a problem? Please describe.

No, this is to add the metadata fields to staging models.

Describe the solution you'd like

Ideally a customer would be able to add the metadata field and possible parse the field out as well.

Current PR Work
PR #18

Hardcoded `plan_interval` in `stg_stripe__plan` for BigQuery removes actual data

Hi there,

In the stg_stripe__plan model, any reason we need this bit?

Here's the direct link to the code

{% if target.type == 'bigquery' %}
    'interval' as plan_interval,
{% else %}
    interval as plan_interval,
{% endif %}

The issue with the above is I no longer have ability to see the ACTUAL interval of the plan. Instead it's just a blanket interval constant, making plan_interval useless.

Is there a particular issue this only happens for BigQuery?

Columns missing in Invoice

Is there an existing feature request for this?

  • I have searched the existing issues

Describe the Feature

Missing columns

  1. PERIOD_START
  2. PERIOD_END

Describe alternatives you've considered

NA

Are you interested in contributing this feature?

  • Yes.
  • Yes, but I will need assistance and will schedule time during your office hours for guidance.
  • No.

Anything else?

Can this be expedited ? This model will not be useful to me as they are important columns

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.