tap-dataverse is a Singer tap for Dataverse.

Built with the Meltano Tap SDK for Singer Taps.

Install from GitHub:

pipx install git+https://github.com/mjsqu/tap-dataverse.git@main

• catalog
• state
• discover
• about
• stream-maps
• schema-flattening
• batch

A full list of supported settings and capabilities for this tap is available by running:

tap-dataverse --about

This Singer tap will automatically import any environment variables within the working directory's .env if the --config=ENV is provided, such that config values will be considered if a matching environment variable is set either in the terminal context or in the .env file.

This tap uses the client_credentials method of authentication and requires an App Registration and PowerApp setup steps. Note, these steps have been copied from internal documentation with all identifying values removed, please refer to the official Dataverse API documentation for further assistance, or do a PR if you would like to help improve the docs.

• Login to https://portal.azure.com using your Azure Admin account
• Open App registrations
• Click + New registration
• Enter a Name (e.g. tap_dataverse_powerplatformaccess)
• Select Accounts in this organizational directory only (single tenant)
• Click Register
• Click Certificates & secrets
• Click + New client secret
• Description - (suggestion Tap-Dataverse Power Platform Client Secret)
• Select Expiry of choice
• Click Add and record client_secret in config
• Click Expose an API
• Click Set
• Update Application ID URI
• Enter the OAuth Endpoint
• Click + Add a scope
• Enter:
• Scope name = session:role-any
• Who can consent? = Admin and users
• Click Add scope
• Click Save

• Login to https://admin.powerplatform.microsoft.com

• Click Environments

• Click Data Provider environment

• Click S2S apps See all

• Click + New app user

• Click + Add an app

• Select tap_dataverse_powerplatformaccess or the name selected for the App Registration earlier

• Click Add

• Enter: Business unit from the PowerPlatform developer settings url e.g. from https://.api.crm6.dynamics.com the business unit value comes before .api.crm in the url

• Once you have completed all these steps, you should have:

• client_id - GUID

• tenant_id - GUID <> client_id

• client_secret

You can easily run tap-dataverse by itself or in a pipeline using Meltano.

tap-dataverse --version
tap-dataverse --help
tap-dataverse --config CONFIG --discover > ./catalog.json

Follow these instructions to contribute to this project.

pipx install poetry
poetry install

Create tests within the tests subfolder and then run:

poetry run pytest

You can also test the tap-dataverse CLI interface directly using poetry run:

poetry run tap-dataverse --help

Note: This tap will work in any Singer environment and does not require Meltano. Examples here are for convenience and to streamline end-to-end orchestration scenarios.

Next, install Meltano (if you haven't already) and any needed plugins:

# Install meltano
pipx install meltano
# Initialize meltano within this directory
cd tap-dataverse
meltano install

Now you can test and orchestrate using Meltano:

# Test invocation:
meltano invoke tap-dataverse --version
# OR run a test `elt` pipeline:
meltano elt tap-dataverse target-jsonl

See the dev guide for more instructions on how to use the SDK to develop your own taps and targets.