This package models Fivetran Log data from our free internal connector. It uses account-level data in the format described by this ERD.

This package helps you understand:

• How you are spending money in Fivetran according to our consumption-based pricing model. We display consumption data at the table, connector, destination, and account levels.
• How your data is flowing in Fivetran:
  • Connector health and sync statuses
  • Transformation run statuses
  • Daily API calls, schema changes, and records modified
  • Table-level details of each connector sync

The package's main goals are to:

• Create a history of measured monthly active rows (MAR), credit consumption, and the relationship between the two
• Enhance the connector table with sync metrics and relevant alert messages
• Enhance the transformation table with run metrics
• Create a history of vital daily events for each connector
• Create an audit log of records inserted, deleted, an updated in each table during connector syncs

Note: An earlier version of this package unioned destination-level connector data to the account level. As of December 2020, the Fivetran Log now supports the creation of account-level connectors. We have removed the Fivetran Log dbt package's unioning functionality and recommend that users resync their Log connectors at the account level.

dbt_fivetran_log currently supports dbt 0.20.x.

Check dbt Hub for the latest installation instructions, or read the dbt docs for more information on installing packages.

Include in your packages.yml

packages:
  - package: fivetran/fivetran_log
    version: [">=0.4.0", "<0.5.0"]

The Fivetran team maintaining this package only maintains the latest version. We highly recommend you keep your packages.yml updated with the dbt hub latest version. You may refer to the CHANGELOG and release notes for more information on changes across versions.

By default, this package will run using your target database and the fivetran_log schema. If this is not where your Fivetran Log data is, add the following configuration to your dbt_project.yml file:

# dbt_project.yml

...
config-version: 2

vars:
  fivetran_log:
    fivetran_log_database: your_database_name
    fivetran_log_schema: your_schema_name 

If you have never created Fivetran-orchestrated basic SQL transformations, your source data will not contain the transformation and trigger_table tables. Moreover, if you have only created scheduled basic transformations that are not triggered by table syncs, your source data will not contain the trigger_table table (though it will contain transformation).

To disable the corresponding functionality in the package, you must add the following variable(s) to your dbt_project.yml file. By default, all variables are assumed to be true:

# dbt_project.yml

...
config-version: 2

vars:
  fivetran_log:
    fivetran_log_using_transformations: false # this will disable all transformation + trigger_table logic
    fivetran_log_using_triggers: false # this will disable only trigger_table logic 

Some users may wish to exclude Fivetran error and warnings messages from the final fivetran_log__connector_status model due to the length of the message. To disable the errors_since_last_completed_sync and warnings_since_last_completed_sync fields from the final model you may add the following variable to you to your dbt_project.yml file. By default, this variable is assumed to be true:

# dbt_project.yml

...
config-version: 2

vars:
  fivetran_log:
    fivetran_log_using_sync_alert_messages: false # this will disable only the sync alert messages within the connector status model

By default this package will build the Fivetran Log staging models within a schema titled (<target_schema> + _stg_fivetran_log) and the Fivetran Log final models within your <target_schema> + _fivetran_log in your target database. If this is not where you would like you Fivetran Log staging and final models to be written to, add the following configuration to your dbt_project.yml file:

# dbt_project.yml

...
models:
  fivetran_log:
    +schema: my_new_final_models_schema # leave blank for just the target_schema
    staging:
      +schema: my_new_staging_models_schema # leave blank for just the target_schema

Read more about using custom schemas in dbt here.

Don't see a model or specific metric you would have liked to be included? Notice any bugs when installing and running the package? If so, we highly encourage and welcome contributions to this package! Please create issues or open PRs against main. See the Discourse post for information on how to contribute to a package.

This package has been tested on BigQuery, Snowflake and Redshift.

• Provide feedback on our existing dbt packages or what you'd like to see next
• Have questions, feedback, or need help? Book a time during our office hours using Calendly or email us at [email protected]
• Find all of Fivetran's pre-built dbt packages in our dbt hub
• Learn how to orchestrate dbt transformations with Fivetran
• Learn more about Fivetran overall in our docs
• Check out Fivetran's blog
• Learn more about dbt in the dbt docs
• Check out Discourse for commonly asked questions and answers
• Join the chat on Slack for live discussions and support
• Find dbt events near you
• Check out the dbt blog for the latest news on dbt's development and best practices