fivetran / dbt_stripe_source Goto Github PK

Is there an existing issue for this?

• I have searched the existing issues

Describe the issue

The test dbt_utils_unique_combination_of_columns_stg_stripe__fee_balance_transaction_id__source_relation is defined as:

- name: stg_stripe__fee
    description: The details of a fee associated with a balance transaction
    tests:
      - dbt_utils.unique_combination_of_columns:
          combination_of_columns:
            - balance_transaction_id
            - source_relation

However, the Fee table has an additional column index that is part of the composite primary key as described here:

https://docs.google.com/presentation/d/1zyxgbaOjgBt3NsY0OfsiGsWDIefcBc-R1lHWlMltCYU/edit#slide=id.g14afa1361c6_3_0

So this test is giving false positives

Relevant error log or model output

No response

Expected behavior

The expected behavior would be for this test to not give false positives and include the index column as part of the unique column set.

dbt Project configurations

name: 'chabi'
version: '1.0.0'
config-version: 2
profile: "{{ env_var('DBT_PROFILE', '') }}"
model-paths: ["models"]
analysis-paths: ["analyses"]
test-paths: ["tests"]
seed-paths: ["seeds"]
macro-paths: ["macros"]
snapshot-paths: ["snapshots"]
target-path: "target"
clean-targets:
  - "target"
  - "dbt_packages"
  - 
models:
  stripe:
    +schema:
    intermediate:
      +schema: stg
  stripe_source:
    +schema: stg
 
vars:
  stripe_schema: stripe
  stripe_timezone: Australia/Sydney
  stripe__using_credit_notes: true

Package versions

packages:
  - package: dbt-labs/dbt_utils
    version: [">=1.0.0", "<2.0.0"]
  - package: dbt-labs/codegen
    version: 0.9.0
  - package: fivetran/stripe
    version: 0.9.0

What database are you using dbt with?

snowflake

dbt Version

Core:
  - installed: 1.4.1
  - latest:    1.4.5 - Update available!

  Your version of dbt-core is out of date!
  You can find instructions for upgrading here:
  https://docs.getdbt.com/docs/installation

Plugins:
  - snowflake: 1.4.0 - Update available!

  At least one plugin is out of date or incompatible with dbt-core.
  You can find instructions for upgrading here:
  https://docs.getdbt.com/docs/installation

Additional Context

No response

Are you willing to open a PR to help address this issue?

• Yes.
• Yes, but I will need assistance and will schedule time during our office hours for guidance
• No.

Is there an existing issue for this?

• I have searched the existing issues

Describe the issue

An error in my Fivetran/Stripe package today with uniqueness in stg_stripe_invoice_line_items model.
Seems to be testing for uniqueness on unique_invoice_line_item_id but I have a few records where it is not unique through the data source.

Relevant error log or model output

17:49:29 3 of 3 START test unique_stg_stripe__invoice_line_item_unique_invoice_line_item_id  [RUN]
17:49:30 3 of 3 FAIL 3 unique_stg_stripe__invoice_line_item_unique_invoice_line_item_id . [FAIL 3 in 1.09s]

Expected behavior

As this stg_stripe_invoice_line_items has 2 PK's I expect the unique_stg_stripe__invoice_line_item_unique_invoice_line_item_id to not exist.

dbt_utils_unique_combination_of_columns_stg_stripe__invoice_line_item_invoice_line_item_id__invoice_id__source_relation is already there to ensure the combo is unique

dbt Project configurations

  #### Variables for Fivetran/stripe package ####
  # stripe_database: database
  stripe_schema: source_ft_stripe 

  stripe__using_invoices:        True  #Disable if you are not using the invoice and invoice_line_item tables
  stripe__using_payment_method:  False  #Disable if you are not using the payment_method and payment_method_card tables
  stripe__using_subscriptions:   True  #Disable if you are not using the subscription and plan/price tables.
  stripe__using_credit_notes:    False   #Enable if you are using the credit note tables.

Package versions

  - package: fivetran/stripe
    version: [">=0.12.0", "<0.13.0"]

What database are you using dbt with?

bigquery

dbt Version

18:16:55 Running with dbt=1.6.11

Additional Context

I think you just need to remove this test?

Are you willing to open a PR to help address this issue?

• Yes.
• Yes, but I will need assistance and will schedule time during our office hours for guidance
• No.

Hi there,

In the stg_stripe__plan model, any reason we need this bit?

Here's the direct link to the code

{% if target.type == 'bigquery' %}
    'interval' as plan_interval,
{% else %}
    interval as plan_interval,
{% endif %}

The issue with the above is I no longer have ability to see the ACTUAL interval of the plan. Instead it's just a blanket interval constant, making plan_interval useless.

Is there a particular issue this only happens for BigQuery?

Are you a Fivetran customer?
Meera, Consultant, Ephemeral Tattoos

Is your feature request related to a problem? Please describe.
I'm trying to use dbt-utils 0.7.0 (the latest release) so that I can use dbt 0.20.0 with my M1 mac, but this is conflicting with the dbt-utils version requirement in my Fivetran package

Describe the solution you'd like
Change the version check to allow for dbt-utils 0.7.0

Describe alternatives you've considered

Additional context

Please indicate the level of urgency and business impact of this request

Are you interested in contributing to this package?

• Yes, I can do this and open a PR for your review.
• Possibly, but I'm not quite sure how to do this. I'd be happy to do a live coding session with someone to get this work implemented.
• No, I'd prefer if someone else did this. I don't have the time and/or don't know how to incorporate the changes necessary.

Is is possible to provide some feedback on this staging model and the exclusion of invoice line items starting with sub_? Trying to find out in Stripe's documentation, I wasn't able to find something relevant. Also according to Stripe:

There are actually 3 types of ID that can appear in the invoice_line_item. What type of ID appears depends on where the item has been added.

sub_ - The ID associated with a subscription on a customer object.
si_ - The ID associated with a subscription item. Typically used for multiplan subscriptions, where the plan becomes the subscription item.
sli_ - The ID associated with a subscription line item. It essentially defines a subscription item on an invoice.

So, it seems that invoice line items starting with sub_ are related to subscriptions, so in stripe__subscription_line_items model of dbt_stripe those are excluded even thought it seems like they shouldn't.

Is there an existing feature request for this?

• I have searched the existing issues

Describe the Feature

The below columns to be added
STATUS_TRANSITIONS_FINALIZED_AT
STATUS_TRANSITIONS_PAID_AT
STATUS_TRANSITIONS_VOIDED_AT
STATUS_TRANSITIONS_MARKED_UNCOLLECTIBLE_AT

Describe alternatives you've considered

None

Are you interested in contributing this feature?

• Yes.
• Yes, but I will need assistance and will schedule time during your office hours for guidance.
• No.

Anything else?

No response

Are you a current Fivetran customer?
Yes

Describe the bug
Found 5k rows in my raw Payment Intents table that are missing from the staging Payment Intents table. Realized they were in the tmp view, meaning that I hadn't re-run dbt.

Steps to reproduce

 select count(*) from raw_data.stripe.payment_intent raw
 left join analytics_data.dbt_meera.stg_stripe__payment_intent stg on raw.id = stg.payment_intent_id
 where stg.payment_intent_id is null

Expected behavior
I'd expect no missing rows across my raw data and staging data

Project variables configuration

version: '1.0'

require-dbt-version: ">=0.14.0"

source-paths: ["models"]
analysis-paths: ["analysis"]
test-paths: ["tests"]
data-paths: ["data"]
macro-paths: ["macros"]

target-path: "target"
clean-targets:
    - "target"
    - "dbt_modules"

quoting:
  database: false
  identifier: false
  schema: false

models:
  transient: true
  stripe_source: 
    +schema:   # Left blank so that will write to schema "dbt"

on-run-end:
  - "{{ grant_select_on_schemas(schemas, 'reporter') }}"
  - "{{ grant_select_on_schemas(schemas, 'transformer') }}"

config-version: 2

vars:
    # Defined variables for Fivetran Stripe package
    stripe_schema: stripe
    stripe_database: raw_data
    using_invoices:        False  #Disabled since we are not using the invoice and invoice_line_item tables
    using_subscriptions:   False  #Disabled since we are not using the subscription and plan tables.

Package Version

packages:
  - package: fishtown-analytics/dbt_utils
    version: 0.6.5
    
  - package: fivetran/stripe_source
    version: 0.3.1

Warehouse

• BigQuery
• Redshift
• Snowflake
• Postgres
• Databricks
• Other (provide details below)

Is there an existing feature request for this?

• I have searched the existing issues

Describe the Feature

Internal - Add databricks compatibility

Describe alternatives you've considered

No response

Are you interested in contributing this feature?

• Yes.
• Yes, but I will need assistance and will schedule time during your office hours for guidance.
• No.

Anything else?

No response

Is there an existing feature request for this?

• I have searched the existing issues

Describe the Feature

Field renaming should take place in staging models (ex, the in the account staging model, rename id to account_id and name to account_name). We've done this in a few places but not all.

However note this will have downstream implications as most of the renaming occurs in the transforms package.

Describe alternatives you've considered

No response

Are you interested in contributing this feature?

• Yes.
• Yes, but I will need assistance and will schedule time during your office hours for guidance.
• No.

Anything else?

No response

Is there an existing feature request for this?

• I have searched the existing issues

Describe the Feature

The latest release of the fivetran_utils package allows us to instinctively infer if the source table the package is using is in fact present, if it is not the package will create an empty table to ensure the downstream transformations succeed.

However, the Stripe package is not currently compatible with this feature due to the livemode_predicate which exists in the *_tmp staging models. In order for the new empty data model creation to be successful in the Stripe package, we will need to move this predicate macro to the non *_tmp staging model.

Describe alternatives you've considered

Not leveraging the new empty model creation feature within the tmp models.

Are you interested in contributing this feature?

• Yes.
• Yes, but I will need assistance and will schedule time during your office hours for guidance.
• No.

Anything else?

No response

Is there an existing feature request for this?

• I have searched the existing issues

Describe the Feature

The new model for subscription item to be added

Describe alternatives you've considered

No response

Are you interested in contributing this feature?

• Yes.
• Yes, but I will need assistance and will schedule time during your office hours for guidance.
• No.

Anything else?

No response

Is there an existing issue for this?

• I have searched the existing issues

Describe the issue

The model is still using stg_stripe__subscription. This is throwing error as the table doesn't not exist . It has been changed to stg_stripe__subscription_history

Relevant error log or model output

STRIPE.SUBSCRIPTION' does not exist or not authorized.

Expected behavior

The model is expected to run fine

dbt Project configurations

stripe_database: FIVETRAN_DB
stripe_schema: STRIPE

Package versions

0.7.2

What database are you using dbt with?

snowflake

dbt Version

1.1.1

Additional Context

No response

Are you willing to open a PR to help address this issue?

• Yes.
• Yes, but I will need assistance and will schedule time during our office hours for guidance
• No.

Hi,

I noticed in the docs you only have a single stripe database, is it possible to have multiple Stripe databases? For example to support multicurrency and not be charged transaction fees, you have to have multiple Stripe accounts.

So we have like:

stripe_foo
stripe_bar
stripe_baz

as our database names.

Are you a Fivetran customer?

Fivetran created FR

Is your feature request related to a problem? Please describe.

This feature request is not related to a problem. However, I could see this being an issue if certain customers have history mode enabled and want the package to run for just the latest records of the orders and subscriptions source tables.

Describe the solution you'd like

The Stripe release notes for February 2022 indicate that there are two new tables ORDER_HISTORY and SUBSCRIPTION_HISTORY. This FR is to include those two tables into the package and allow users to disable the history mode and only select the latest records.

Describe alternatives you've considered

Using the non-history ORDER and SUBSCRIPTION source tables.

Are you a Fivetran customer?

Original request identified by @sachams

Is your feature request related to a problem? Please describe.

No, this is to add the metadata fields to staging models.

Describe the solution you'd like

Ideally a customer would be able to add the metadata field and possible parse the field out as well.

Current PR Work
PR #18

Are you a Fivetran customer?
Benoit Mayer, Stedi Inc

Is your feature request related to a problem? Please describe.
We store customer metadata in Stripe on the customers and would like it to flow through to reporting. the metadata is essential to our reporting.

Describe the solution you'd like
add metadata on customers as implemented for transactions as described https://github.com/fivetran/dbt_stripe_source/tree/v0.4.3#pivoting-out-metadata-properties

Describe alternatives you've considered

Please indicate the level of urgency and business impact of this request
not critical but time sensitive. by end of month would be ideal

Are you interested in contributing to this package?

• Yes, I can do this and open a PR for your review.
• Possibly, but I'm not quite sure how to do this. I'd be happy to do a live coding session with someone to get this work implemented.
• No, I'd prefer if someone else did this. I don't have the time and/or don't know how to incorporate the changes necessary.

Is there an existing issue for this?

• I have searched the existing issues

Describe the issue

The current definition of the created field within the balance_transaction source should explicitly define the timezone the timestamp is reported within. The stripe connector syncs this field as UTC and we should address it as such.

Relevant error log or model output

N/A

Expected behavior

The field definition details the UTC timezone.

dbt Project configurations

N/A

Package versions

packages:

• package: fivetran/stripe_source
  version: [">=0.6.0", "<0.7.0"]

What database are you using dbt with?

bigquery

dbt Version

v 1.1.0

Additional Context

No response

Are you willing to open a PR to help address this issue?

• Yes.
• Yes, but I will need assistance and will schedule time during our office hours for guidance
• No.

Is there an existing feature request for this?

• I have searched the existing issues

Describe the Feature

We include the phone field in the get_customer_columns macro, but we do not include it in the staging model. We only include the shipping_phone but i believe both will be needed for our identity resolution project

Describe alternatives you've considered

just use shipping_phone, but shipping_phone is sometimes just the company phone, while phone is the individual's

Are you interested in contributing this feature?

• Yes.
• Yes, but I will need assistance and will schedule time during your office hours for guidance.
• No.

Anything else?

No response

Is there an existing feature request for this?

• I have searched the existing issues

Describe the Feature

As identified within the dbt_stripe Issue #45, it seems that the fal target.type is not supported within conditionals found throughout the package. In particular, the get_plan_columns conditional is exclusively looking for the target.type in BigQuery, Spark, or Databricks. However, if a user has the fal adapter, then it will fail as the quoting is needed but the adapter is not included in the conditional.

This feature is to explore supporting the fal adapter and how it differs from other warehouses.

Describe alternatives you've considered

Do not use the fal adapter.

Are you interested in contributing this feature?

• Yes.
• Yes, but I will need assistance and will schedule time during your office hours for guidance.
• No.

Anything else?

No response

Are you a Fivetran customer?
Yes. Ramiz Bozai, Senior Data Analyst at Cerebral

Is your feature request related to a problem? Please describe.
It would be great to have an identifier denoting which data/records came from which source as I union together our different stripe tables from different connections.

Describe the solution you'd like
Having a column indicating what the source of the data is (like country) would allow us to filter that data based on the column.

Describe alternatives you've considered
This is more of a nice to have, but I understand I can possible fork this package and potentially try to add this myself.

Additional context
N/A

Please indicate the level of urgency and business impact of this request
Not urgent. Just a nice to have feature.

Are you interested in contributing to this package?

• Yes, I can do this and open a PR for your review.
• Possibly, but I'm not quite sure how to do this. I'd be happy to do a live coding session with someone to get this work implemented.
• No, I'd prefer if someone else did this. I don't have the time and/or don't know how to incorporate the changes necessary.

Is there an existing feature request for this?

• I have searched the existing issues

Describe the Feature

Remove this filter and persist the is_deleted field so that deleted customers continue downstream (and we can tell that they're deleted).

this will provide more complete data to places where we join in customer data (like here) and will allow for better matching in the current Customer360 project.

note: this will increase the # of rows in the stripe__customer_overview table, but likely not by a huge amount.

also this will not require a change to the transform package, as the is_deleted field will be automatically picked up by this

Describe alternatives you've considered

No response

Are you interested in contributing this feature?

• Yes.
• Yes, but I will need assistance and will schedule time during your office hours for guidance.
• No.

Anything else?

No response

Is there an existing feature request for this?

• I have searched the existing issues

Describe the Feature

It would be beneficial to add the subscription_id field to the invoice staging table. This field could then be leveraged in downstream transformations.

Describe alternatives you've considered

Fork the package and leverage this field or use the source directly.

Are you interested in contributing this feature?

• Yes.
• Yes, but I will need assistance and will schedule time during your office hours for guidance.
• No.

Anything else?

No response

Is there an existing issue for this?

• I have searched the existing issues

Describe the issue

We should leverage the adapter dispatch for all macros in order to allow customers to override them when needed. (livemode_predicate )

Relevant error log or model output

No response

Expected behavior

We can test that the macro can be overridden using the dispatch config.

dbt Project configurations

na

Package versions

na

What database are you using dbt with?

other (mention it in "Additional Context")

dbt Version

na

Additional Context

No response

Are you willing to open a PR to help address this issue?

• Yes.
• Yes, but I will need assistance and will schedule time during our office hours for guidance
• No.

I want to disable the below tables from processing. Can you please let me know how to do it ?

1. stg_stripe__card
2. stg_stripe__balance_transaction
3. stg_stripe__fee
4. stg_stripe__payment_intent
5. stg_stripe__payout

dbt_project .yml below

• using_invoices: True #Disable if you are not using the invoice and invoice_line_item tables
• using_subscriptions: True #Disable if you are not using the subscription and plan tables.
• using_payment_method: False #Disable if you are not using the invoice and invoice_line_item tables
• using_credit_notes: False #Disable if you are using the credit note tables.

Hi there, I would like to run this package on our stripe prod data aswell as the test data (using the fivetran test connector). I can override schema names accordingly. But test customers will never be pulled in because their livemode flag will be set to false. Would it be worth adding some configuration option to enable to run this package on test data?
Just wondering if there are some considerations that I'm not aware of?
many thanks,
Christoph.

Is there an existing feature request for this?

• I have searched the existing issues

Describe the Feature

Missing columns

1. PERIOD_START
2. PERIOD_END

Describe alternatives you've considered

NA

Are you interested in contributing this feature?

• Yes.
• Yes, but I will need assistance and will schedule time during your office hours for guidance.
• No.

Anything else?

Can this be expedited ? This model will not be useful to me as they are important columns