microsoft / dbt-fabricspark Goto Github PK

License: MIT License

Makefile 0.89% Python 98.99% Shell 0.13%

dbt-fabricspark's Introduction

[dbt](https://www.getdbt.com/) enables data analysts and engineers to transform their data using the same practices that software engineers use to build applications.

dbt is the T in ELT. Organize, cleanse, denormalize, filter, rename, and pre-aggregate the raw data in your warehouse so that it's ready for analysis.

dbt-fabricspark

The dbt-fabricspark package contains all of the code enabling dbt to work with Synapse Spark in Microsoft Fabric. For more information, consult the docs.

Getting started

Install dbt
Read the introduction and viewpoint

Running locally

Use livy endpoint to connect to Synapse Spark in Microsoft Fabric. The binaries required to setup local environment is not possiblw with Synapse Spark in Microsoft Fabric. However, you can configure profile to connect via livy endpoints.

Create a profile like this one:

fabric-spark-test:
  target: fabricspark-dev
  outputs:
    fabricspark-dev:
        authentication: CLI
        method: livy
        connect_retries: 0
        connect_timeout: 10
        endpoint: https://api.fabric.microsoft.com/v1
        workspaceid: bab084ca-748d-438e-94ad-405428bd5694
        lakehouseid: ccb45a7d-60fc-447b-b1d3-713e05f55e9a
        lakehouse: test
        schema: test
        threads: 1
        type: fabricspark
        retry_all: true

Reporting bugs and contributing code

Want to report a bug or request a feature? Let us know on Slack, or open an issue.

Code of Conduct

Everyone interacting in the Microsoft project's codebases, issue trackers, and mailing lists is expected to follow the PyPA Code of Conduct.

Join the dbt Community

Be part of the conversation in the dbt Community Slack
Read more on the dbt Community Discourse

Reporting bugs and contributing code

Want to report a bug or request a feature? Let us know on Slack, or open an issue
Want to help us build dbt? Check out the Contributing Guide

Code of Conduct

Everyone interacting in the dbt project's codebases, issue trackers, chat rooms, and mailing lists is expected to follow the dbt Code of Conduct.

dbt-fabricspark's People

Contributors

Stargazers

Watchers

Forkers

ekote nixent jcvdodson

dbt-fabricspark's Issues

Action required: migrate or opt-out of migration to GitHub inside Microsoft

Migrate non-Open Source or non-External Collaboration repositories to GitHub inside Microsoft

In order to protect and secure Microsoft, private or internal repositories in GitHub for Open Source which are not related to open source projects or require collaboration with 3rd parties (customer, partners, etc.) must be migrated to GitHub inside Microsoft a.k.a GitHub Enterprise Cloud with Enterprise Managed User (GHEC EMU).

Action

✍️ Please RSVP to opt-in or opt-out of the migration to GitHub inside Microsoft.

❗Only users with admin permission in the repository are allowed to respond. Failure to provide a response will result to your repository getting automatically archived.🔒

Instructions

Reply with a comment on this issue containing one of the following optin or optout command options below.

✅ Opt-in to migrate

@gimsvc optin --date <target_migration_date in mm-dd-yyyy format>

Example: @gimsvc optin --date 03-15-2023

❌ Opt-out of migration

@gimsvc optout --reason <staging|collaboration|delete|other>

Example: @gimsvc optout --reason staging

Options:

staging : This repository will ship as Open Source or go public

collaboration : Used for external or 3rd party collaboration with customers, partners, suppliers, etc.

delete : This repository will be deleted because it is no longer needed.

other : Other reasons not specified

Need more help? 🖐️

Email [email protected]. ✉️
Post your questions in GitHub inside Microsoft Team in Microsoft Teams. 🗨️

Documentation for Livy REST API

Is there any documentation for the Livy endpoint, something similar to one for Synapse?

Fabric REST APi describes other elements of Fabric but not Livy.

Incorrect declaration of assert_not_null macro

assert_not_null macro is declared as fabricassert_not_null in dbt/include/fabricspark/macros/utils/assert_not_null.sql

It results in 'assert_not_null' is undefined error when calling dateadd macro. Suggest replacing fabricassert_not_null with assert_not_null

This repo is missing important files

There are important files that Microsoft projects should all have that are not present in this repository. A pull request has been opened to add the missing file(s). When the pr is merged this issue will be closed automatically.

Microsoft teams can learn more about this effort and share feedback within the open source guidance available internally.

Merge this pull request

Unable to establish connection to Livy endpoint with error 408

Trying to materialize a model with the dbt-fabric adapter. It fails to establish connection to Livy endpoint and times out at create_session

Error message:
Http Error: 408 Client Error: Request Timeout for url: https://api.fabric.microsoft.com/v1/workspaces/{workspace_id}/lakehouses/{lakehouse_id}/livyapi/versions/2023-12-01/sessions

Call to other endpoints, like List tables works fine.

Is there any configuration that needs to be done to make it to work?

This repo is missing important files

Microsoft teams can learn more about this effort and share feedback within the open source guidance available internally.

Merge this pull request

microsoft / dbt-fabricspark Goto Github PK

dbt-fabricspark's Introduction

dbt-fabricspark

Getting started

Running locally

Reporting bugs and contributing code

Code of Conduct

Join the dbt Community

Reporting bugs and contributing code

Code of Conduct

dbt-fabricspark's People

Contributors

Stargazers

Watchers

Forkers

dbt-fabricspark's Issues

Migrate non-Open Source or non-External Collaboration repositories to GitHub inside Microsoft

Action

Instructions

Need more help? 🖐️

Recommend Projects

Recommend Topics

Recommend Org