[Feature] Improve run time of package with additional ftiler about dbt_mixpanel HOT 3 CLOSED

Hi @CraigWilson-ZOE ! Thanks for bringing this to our attention, I think you made a great point as to what the approach should be here. We agree that reducing package run time should be crucial in your development sandbox, particularly if you are not planning to use this data.

Also, thanks for submitting the proposed PR! We will review this PR in a future sprint and see if there are more holistic changes we can incorporate to meet resourcing requirements in running dbt.

We will keep you in the loop about further updates here, and changes we plan to make to optimize package runs.

from dbt_mixpanel.

Hello @CraigWilson-ZOE, thank you for your suggestion regarding the filter. I am working on enhancing the overall performance of this Mixpanel package and have added additional updates on a test branch. I've eliminated the stg_mixpanel__event_tmp model in favor of transforming stg_mixpanel__event into an incremental model, with the intention that subsequent runs after the initial would not incur much compute. I've also made adjustments to some of the clustering aspects and updated the incremental strategies for all the models.

If you would be able to try out the test branch and let me know your thoughts and feedback, it would be appreciated! I am curious to know how much of an impact these updates make. Thank you!

To install the test branch, you can use the following in your packages.yml:

packages:
  - git: https://github.com/fivetran/dbt_mixpanel.git
    revision: feature/performance-enhancement
    warn-unpinned: false

from dbt_mixpanel.

Hi @CraigWilson-ZOE, following up to my last message, we still removed the tmp view, but decided to keep the stg model as ephemeral. This prevents duplicate tables while also allowing the start date to be set by the user. I have gone ahead and released v0.9.0 and will mark this issue as closed, however if you have any additional feedback, please feel free to tag us in this thread or open a new issue!

from dbt_mixpanel.