gouline / dbt-metabase Goto Github PK
View Code? Open in Web Editor NEWdbt + Metabase integration
Home Page: https://pypi.org/project/dbt-metabase/
License: MIT License
dbt + Metabase integration
Home Page: https://pypi.org/project/dbt-metabase/
License: MIT License
When using a config.yml file and specifying metabase_use_http: True
(tried upper and lower case), I end up with the following SSL error (note: running Metabase via http://localhost:3000
):
requests.exceptions.SSLError: HTTPSConnectionPool(host='localhost', port=3000): Max retries exceeded with url: /api/session (Caused by SSLError(SSLError(1, '[SSL: WRONG_VERSION_NUMBER] wrong version number (_ssl.c:1122)')))
After digging into it, the issue was that it was still sending the request via https
. I found that here the self.protocol
attribute was still being passed as https
resulting in the error.
response = requests.request(
method, f"{self.protocol}://{self.host}{path}", verify=self.verify, **kwargs
)
Using --metabase_http as a command line argument fixed the issue for me, I believe it just overrides what's here
@click.option(
"--metabase_http/--metabase_https",
"metabase_use_http",
default=False,
help="use HTTP or HTTPS to connect to Metabase. Default HTTPS",
)
Currently using dbt-metabase==0.8.5
ERROR HTTP request failed. Payload: {'display_name': None, 'description': 'My description here.', 'visibility_type': 'normal'}. metabase.py:885
Response: {"errors":{"visibility_type":"value may be nil, or if non-nil, value must be one of: `cruft`, `hidden`, `technical`."}}
Traceback (most recent call last):
File "/usr/local/opt/[email protected]/Frameworks/Python.framework/Versions/3.9/bin/dbt-metabase", line 4, in <module>
__import__('pkg_resources').run_script('dbt-metabase==0.9.1.dev1+g68f584e', 'dbt-metabase')
File "/usr/local/lib/python3.9/site-packages/pkg_resources/__init__.py", line 672, in run_script
self.require(requires)[0].run_script(script_name, ns)
File "/usr/local/lib/python3.9/site-packages/pkg_resources/__init__.py", line 1479, in run_script
exec(script_code, namespace, namespace)
File "/usr/local/lib/python3.9/site-packages/dbt_metabase-0.9.1.dev1+g68f584e-py3.9.egg/EGG-INFO/scripts/dbt-metabase", line 5, in <module>
File "/usr/local/lib/python3.9/site-packages/dbt_metabase-0.9.1.dev1+g68f584e-py3.9.egg/dbtmetabase/__init__.py", line 754, in main
File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1130, in __call__
return self.main(*args, **kwargs)
File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1055, in main
rv = self.invoke(ctx)
File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1657, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/usr/local/lib/python3.9/site-packages/dbt_metabase-0.9.1.dev1+g68f584e-py3.9.egg/dbtmetabase/__init__.py", line 127, in invoke
File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1404, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/usr/local/lib/python3.9/site-packages/click/core.py", line 760, in invoke
return __callback(*args, **kwargs)
File "/usr/local/lib/python3.9/site-packages/dbt_metabase-0.9.1.dev1+g68f584e-py3.9.egg/dbtmetabase/__init__.py", line 288, in wrapper
File "/usr/local/lib/python3.9/site-packages/dbt_metabase-0.9.1.dev1+g68f584e-py3.9.egg/dbtmetabase/__init__.py", line 624, in models
File "/usr/local/lib/python3.9/site-packages/dbt_metabase-0.9.1.dev1+g68f584e-py3.9.egg/dbtmetabase/metabase.py", line 216, in export_models
File "/usr/local/lib/python3.9/site-packages/dbt_metabase-0.9.1.dev1+g68f584e-py3.9.egg/dbtmetabase/metabase.py", line 268, in export_model
File "/usr/local/lib/python3.9/site-packages/dbt_metabase-0.9.1.dev1+g68f584e-py3.9.egg/dbtmetabase/metabase.py", line 880, in api
File "/usr/local/lib/python3.9/site-packages/requests/models.py", line 1022, in raise_for_status
raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 400 Client Error: Bad Request for url: http://localhost:3000/api/table/10
When running the dbt-metabase exposures
command it fails when it hits an item referencing a deactivated user. In particular, this line returns 404, causing the whole script to crash:
dbt-metabase/dbtmetabase/metabase.py
Line 596 in 7d8c97e
Possible solutions:
/api/user?include_deactivated=true
endpoint to list all users once and search by ID. This endpoint has the advantage of also retrieving deactivated user's details, whilst the /api/user/:id
one returns 404.dbt model names are case insensitive, and so the user should be able to enter the names of included models in lowercase, which is often how models are typed.
Currently, because model_name
in dbt_manifest.py
is uppercased, model names that are sent to the includes
variable as lowercase strings are ignored.
A simple uppercasing of the includes
list solves this.
I'll open a PR.
Note: I think a similar issue may exist with dbt_excludes
, but I don't need to use that at the moment, and when I tried to include it as a quick fix, it didn't work expected and I don't have time to dig into it, so I'm leaving it for now.
I'm using the dbt alias
feature in many of my models, which means that my model name doesn't match the respective object in my database and dbt-metabase
won't be able to correctly match them when retrieving the schema from Metabase.
This is because dbt-metabase
retrieves the model information/name directly from the YAML files written by the dbt users instead of using the dbt compiled artifacts which are now much more stable: https://docs.getdbt.com/reference/artifacts/dbt-artifacts. I also use multiple schemas in my project and the schemas can be defined in multiple places (config block in the model, dbt_project.yml, etc). The compiled manifest.json is the final source of truth of where my model will be materialized (schema, alias, etc).
I propose to expand the CLI and Python interfaces to, besides supporting the existing dbt_path
(for backward compatibility), also support a dbt_manifest_path
to a manifest.json
file (mutually exclusive arguments).
I'm working on a PR for that, but I'm creating this issue here to gather early feedback and see what you think.
PS: Thanks a lot for this project! It will be very useful in my organization.
Support for timeline events would be cool (API).
Hi, i'm trying to use this awesome project with my dbt setup (bigquery) but keep getting this AttributeError: 'NoneType' object has no attribute 'upper'.
I'm running on dbt v0.20.0 and Metabase 0.40.1
Traceback (most recent call last):
File "/home/edbizarro/.pyenv/versions/analytics-venv-3.7.8/bin/dbt-metabase", line 6, in <module>
dbtmetabase.main(sys.argv[1:])
File "/home/edbizarro/.pyenv/versions/3.7.8/envs/analytics-venv-3.7.8/lib/python3.7/site-packages/dbtmetabase/__init__.py", line 248, in main
include_tags=parsed.include_tags,
File "/home/edbizarro/.pyenv/versions/3.7.8/envs/analytics-venv-3.7.8/lib/python3.7/site-packages/dbtmetabase/__init__.py", line 114, in export
mbc.export_models(metabase_database, models, reader.catch_aliases)
File "/home/edbizarro/.pyenv/versions/3.7.8/envs/analytics-venv-3.7.8/lib/python3.7/site-packages/dbtmetabase/metabase.py", line 156, in export_models
table_lookup, field_lookup = self.build_metadata_lookups(database_id)
File "/home/edbizarro/.pyenv/versions/3.7.8/envs/analytics-venv-3.7.8/lib/python3.7/site-packages/dbtmetabase/metabase.py", line 377, in build_metadata_lookups
table_schema = table.get("schema", "public").upper()
AttributeError: 'NoneType' object has no attribute 'upper'
Command I'm using to test
dbt-metabase export \
--dbt_manifest_path ./analytics/target/manifest.json \
--dbt_database analytics \
--metabase_host ...\
--metabase_user ... \
--metabase_password "..." \
--metabase_database "Analytics" \
--schema public
Adding this per suggestion from @z3z1ma from the dbt slack discussion ๐
This feature would be helpful for anyone with a large Metabase deployment - where adding exposures might add a lot of noise to the dbt project.
We could add functionality to assign maturity or tags based on total use, recent use, or some combination of those two. Tagging may be better as it's a bit more granular -- you could have things like tag:broken-exposure or tag:no-views-last-month and use that to find and clean up unused or lesser used exposures.
Hello
There seems to be a size limit somewhere
dbt-metabase used to work but as the model grew it suddently stopped working
I get
Traceback (most recent call last):
File "/usr/local/bin/dbt-metabase", line 5, in <module>
dbtmetabase.main()
File "/usr/local/lib/python3.8/dist-packages/dbtmetabase/__init__.py", line 720, in main
cli(max_content_width=600) # pylint: disable=unexpected-keyword-arg
File "/usr/lib/python3/dist-packages/click/core.py", line 764, in __call__
return self.main(*args, **kwargs)
File "/usr/lib/python3/dist-packages/click/core.py", line 717, in main
rv = self.invoke(ctx)
File "/usr/lib/python3/dist-packages/click/core.py", line 1137, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/usr/local/lib/python3.8/dist-packages/dbtmetabase/__init__.py", line 113, in invoke
return super().invoke(ctx)
File "/usr/lib/python3/dist-packages/click/core.py", line 956, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/usr/lib/python3/dist-packages/click/core.py", line 555, in invoke
return callback(*args, **kwargs)
File "/usr/local/lib/python3.8/dist-packages/dbtmetabase/__init__.py", line 262, in wrapper
return func(*args, **kwargs)
File "/usr/local/lib/python3.8/dist-packages/dbtmetabase/__init__.py", line 568, in models
dbt_models, aliases = dbt.read_models(
File "/usr/local/lib/python3.8/dist-packages/dbtmetabase/models/interface.py", line 134, in read_models
return self.parser.read_models(self, include_tags, docs_url)
File "/usr/local/lib/python3.8/dist-packages/dbtmetabase/parsers/dbt_manifest.py", line 112, in read_models
self._read_model(
File "/usr/local/lib/python3.8/dist-packages/dbtmetabase/parsers/dbt_manifest.py", line 213, in _read_model
depends_on_id = list(
IndexError: list index out of range
Hi!
The docs state that this amazing library "Requires Python 3.6 or above", and also setup.py
states the same.
However, some recently introduced changes (https://github.com/gouline/dbt-metabase/blob/master/dbtmetabase/models/metabase.py#L28) broke that. Using typing.Literal
requires Python 3.8+. Running the library with Python 3.7 for example will raise:
ImportError: cannot import name 'Literal' from 'typing' (/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/typing.py)
Alternatives on what to do:
Literal
type hint and instead use EnumOrthogonal to that, we could:
- uses: actions/setup-python@v2
with:
python-version: '3.6.x'
tox
I'm happy to contribute with a PR once the maintainer and contributors align on how to move forward ๐ .
When I run the following command, the session is established but then the process ends without any message:
dbt-metabase models \
--dbt_manifest_path ./target/manifest.json \
--dbt_schema dbt_prod \
--metabase_host $url \
--metabase_user $metabase_email \
--metabase_password $metabase_pw \
--metabase_database $dbname \
--dbt_database dbt_prod
2021-09-14 09:42:07,987 - INFO - Session established successfully
Am I doing something wrong?
Not sure if this is the right place for this but I have built functionality capable of synchronizing dbt tests to a designated collection in Metabase. The flip side of the process uses Metabase API to query each question in the designated collection flagging failures and collecting results to in turn be displayed on Metabase in a unit test dashboard.
Example collection with synchronized unit tests / schema tests / great expectations from dbt Jinja compiled tests.
Example Dashboard from API collected results:
Table links to Metabase questions where I can view the failing records:
While I dont know if this falls into the scope of this project, I wanted to ask from the perspective of a deeper integration what your thoughts were. I have these processes running via Jenkins in a CI/CD env in production with good success. Even without collection synchronization to quickly view results, I havent seen much dbt test query parsing and tabular result generation. Maybe a mention of it from someone else who built an application wrapper.
While running dbt-metabase config
it suggested -1
as a timeout value
But after running dbt-metabase exposures
it failed with
File "/home/psegatori/.local/lib/python3.8/site-packages/dbtmetabase/metabase.py", line 121, in sync_and_wait
raise exceptions.MetabaseUnableToSync(
dbtmetabase.models.exceptions.MetabaseUnableToSync: Timeout provided -1 secs, must be at least 5
In 646a914, you fixed it so that if we haven't specified a metabase.semantic_type.
This is good, but now, how do we explicitly clear a semantic type set by the automagic rules that Metabase applies on schema sync?
Setting metabase.semantic_type: null
seems to have no effect. Can we look for the presence of this key, or maybe use a special value?
Traceback (most recent call last):
File "/usr/local/bin/dbt-metabase", line 5, in <module>
dbtmetabase.main()
File "/usr/local/lib/python3.9/site-packages/dbtmetabase/__init__.py", line 751, in main
cli(max_content_width=600) # pylint: disable=unexpected-keyword-arg
File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1130, in __call__
return self.main(*args, **kwargs)
File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1055, in main
rv = self.invoke(ctx)
File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1657, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/usr/local/lib/python3.9/site-packages/dbtmetabase/__init__.py", line 127, in invoke
return super().invoke(ctx)
File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1404, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/usr/local/lib/python3.9/site-packages/click/core.py", line 760, in invoke
return __callback(*args, **kwargs)
File "/usr/local/lib/python3.9/site-packages/dbtmetabase/__init__.py", line 288, in wrapper
return func(*args, **kwargs)
TypeError: models() got an unexpected keyword argument 'metabase_session_id'
Lets consider our usual flow of
dbt compile -m my_model
navigate to compiled model
copy paste sql
find somewhere to run it to test it
OR
dbt run -m my_model
check in datagrip, dbeaver, dbvisualizer, vscode sql ext, etc.
repeat x?...
which the latter implies potentially multiple CTAS, tables being dropped and rewritten, etc. or the inability to test in production vs compile and pasting somewhere you can run it.
I think we can easily offer an ability to invoke something like:
dbt-metabase explore my_model
And on our end conceptually, we use a subprocess to invoke dbt compile -m my_model
Read the compiled sql from the manifest.json or file.
Construct and pass it into a json object that looks something like {dataset_query: "SELECT * FROM ..."}
with your compiled model query inputted
Base64 encode the json dumped string
And construct a URL like this as many Metabase aficionados may be familiar with (its how metabase caches your in progress queries in the editor).
https://my.metabase/question#eyJkYXRhc2V0X3F1ZXJ5Ijp7InR5cGUiO...c2V0dGluZ3MiOnt9fQ==
This will allow, with a simple invocation, using metabase as a front end graphical dbt model explorer without the above workflow being an impedance. I have a similar goal in my project dbt-osmosis which uses a dbt-osmosis workbench
invocation with streamlit as a backend for rapidly developing/prototyping and diffing the data output of model changes. I think for Metabase power users, this is a simple and effective alternative too.
We empower users to explore models, all without actually ever asking them for db creds. Interesting right.
Opening this now to gauge feedback prior to implementation.
Not clear what the env var should be called. Readme and first place in the code says MB_PASS:
ENV_VARS = [
"DBT_DATABASE",
"DBT_PATH",
"DBT_MANIFEST_PATH",
"MB_USER",
"MB_PASS",
But the actual env var handling code says:
"DBT_PATH",
"DBT_MANIFEST_PATH",
"MB_USER",
"MB_PASS",
@click.option(
"--metabase_password",
metavar="PASS",
envvar="MB_PASSWORD",
Is the latter wrong?
To make code exploration and review easier, would it be possible to replicate metabase file and folder hierarchy when generating exposures ? Like reusing the slug
/metabase-exposures/question/42-my-awesome-question.yml
/metabase-exposures/dashboard/84-my-super-dashboard
Or maybe just the ID if to avoid too many file changes when just renaming something ?
/metabase-exposures/cards/1.yml
/metabase-exposures/cards/2.yml
/metabase-exposures/dashboards/1.yml
Currently all exposures are sent to a single file and it's messy
On Metabase, it's possible to define permissions restrictions to tables for certain groups (Such as in the picture)
In the Open Source plan for Metabase, we can define if this group has either "Unrestricted" or "No self-Service" permission to the table, I am not sure if this changes for the other plans.
Is the automation for the permission system, something planned/wanted for this package?
One suggestion that could make it possible is to add this in the meta
section of the table definition, and define which groups have which level of permission:
I suggest to use /api/database/{id}/metadata instead of /api/database/{id}/fields.
Currently, the code is leveraging the /fields API to retrieve field metadata. However, the response I am getting is like:
{ "id": 39, "name": "Companies House No", "base_type": "type/Text", "special_type": null, "table_name": "Dim Account", "schema": null }
with no stable identifier for the field name (name field is basically the display name in Metabase).
I suggest to use the /metadata API instead, which will return a nested json of tables and fields, where a field will look like:
{ "description": null, "database_type": "STRING", "special_type": "type/Company", "display_name": "Company Number", "base_type": "type/Text", }
(I have summarised the response, which is a lot larger).
I'd be happy to implement this change.
Metabase 0.40 has breaking changes:
/api/database
does not anymore return an array but now returns a dictionary.
/api/fields/{id}
does not anymore return an array but now returns a dictionary.
Hey, thanks for sharing you work on this.
I'm having some problems using the library to parse my project. Neither the the manifest parser nor the DbtFolderReader
is able to parse it without failing. For my project I'm using dbt 0.20.0
, is this version supported by this package? The manifest schema has slightly changed the last couple of versions and I'm wondering if you might be working with another version.
I think its very valid for someone to want to parse exposures in CI/CD taking a manifest.json as an input, modifying it, and outputting an updated manifest.json. As of today, we output a yaml which is easily readable, can be committed to source control, etc. but might not be desired by a user who just wishes to see exposures in docs without the additional yml. Lets add a cli flag to make the execution optionally manipulate the artifact instead of generating yml to be consumed by subsequent dbt command.
This paves way for my next PR which involves making the data catalogue explorable through metabase table explorer. Stay tuned for these. Will updates this issue as I go if needed.
Would be nice to expose all Metabase's visuals and dashboards in the DBT documentation as exposures
Metabase Metrics (and in particular custom aggregations) are very useful to our end-users. I'm finding myself spending a lot of time in Metabase managing Metrics, the UI is very inconvenient.
I'd like to be able to define a list of Metabase Metrics on DBT models as schema model meta
properties. I wonder if this fits this project's scope. The primary challenge is parsing the Metabase expressions which are defined as string (ex: Distinct(case([step] = "hired", [candidate_user_id])
) into trees that the API can handle. I think it's easiest to maintain if I compile the JS parser from metabase source code into a small node script that can be committed and called from this project.
This would add a good amount of complexity so it's probably beyond the scope of the project. I thought I would share for the sake of discussion.
Moving the discussion to an issue to avoid polluting the discussion on the PR.
@z3z1ma wrote on #19 that he has a follow-up PR in mind on adding a feature to automatically create dbt exposures based on Metabase questions, making it possible to see on the dbt lineage graph which Metabase questions are using which dbt models. More details on this message as well: #19 (review)
Just today I open-sourced an internal project that does the same, but between dbt and Tableau: https://github.com/voi-oss/dbt-exposures-crawler.
One thing I would like to ask is what's @gouline scope for dbt-metabase
. Do you want to keep it narrow and focus on documentation syncing, or do you want to be a comprehensive Metabase <> dbt toolkit and think that @z3z1ma feature would fit on dbt-metabase
as well?
@z3z1ma, how much of the existing dbt-metabase
codebase you actually use in the exposures feature? Mostly the Metabase API client I guess?
Hello,
I've got a KeyError: 'fk_target_table'
when propagating data from dbt table to Metabase.
This error occurs in two specific cases:
type/FK
but it has not a tests
section within it.type/FK
but its tests
section contains a dbt_utils.relationships_where
test within it.Hello,
I've tried to follow the usage and installation instructions but got the following error
File "/home/xxxx/.local/lib/python3.8/site-packages/dbtmetabase/__init__.py", line 99, in process_value
and ctx.get_parameter_source(self.name)
AttributeError: 'Context' object has no attribute 'get_parameter_source'
I upgraded my local click
package and it worked.
It upgraded from Click 7.0
to click 8.1.3
I'm a PHP developer with no experience in Python so my suggestion might be off but could the error be avoided if the project declared that it can only work with a specific version range of click
package in the requirements.txt
?
The sync timeout should have a default value otherwise the behavior is not as described
Sync is only attempted if metabase_sync_timeout is defined (default is None)
metabase_sync_timeout (Optional[int], optional): Synchronization timeout (in secs). If set, we will fail hard on synchronization failure; if not set, we will proceed after attempting sync regardless of success. Only valid if sync is enabled. Defaults to None.
dbt-metabase/dbtmetabase/__init__.py
Line 546 in e7c7832
It appears to me that this logic fails for dbt==1.0.0
.
It seems that relationship tests don't contain the related model on the depends_on
iterable anymore. It does appear in the refs
iterable both pre and post-1.0.0
, however only as simple ref string ('my_model_name'
) instead of a full unique identified (model.[project_name].[model_name]
).
Has this been tested yet? Is there a favored approach?
I'd be happy to give this a go on the weekend with a PR, but I don't know the repository well and am wondering if I am simply missing something :)
DBT recently added meta
definitions for models and columns.
It'd be nice to be able to use these, rather than tests, to specify metabase column types.
Using meta
would remove any impact on test runtime when running DBT, and remove the need for the DBT package specified in this project.
The docs mention models
and exposures
commands, but the CLI appears to only have the export
command as of version 0.7.1.
For example, running dbt-metabase exposures
returns the following:
usage: dbt-metabase [-h] --dbt_path PATH --mb_host HOST --mb_user USER --mb_password PASS [--mb_https HTTPS] --database DB --schema SCHEMA [--sync ENABLE] [--sync_timeout SECS] [--includes [MODELS [MODELS ...]]]
[--excludes [MODELS [MODELS ...]]]
{export}
dbt-metabase: error: argument command: invalid choice: 'exposures' (choose from 'export')
Steps to reproduce:
pip install pipenv
(creates a new env, ignore if using venv)pipenv install dbt-metabase
pipenv run dbt-metabase exposures
Version currently installed:
dbt-metabase==0.7.1
- PyYAML [required: Any, installed: 6.0]
- requests [required: Any, installed: 2.26.0]
- certifi [required: >=2017.4.17, installed: 2021.10.8]
- charset-normalizer [required: ~=2.0.0, installed: 2.0.7]
- idna [required: >=2.5,<4, installed: 3.3]
- urllib3 [required: >=1.21.1,<1.27, installed: 1.26.7]
We are using multiple schemas specified in dbt_project.yml. It would be great if dbt-metabase follows the schema settings and does not send everything to the schema from the command line.
models:
my_project:
marketing:
schema: marketing
product:
schema: product
events:
schema: events
thanks @gouline :)
Hi!
I have am using bigquery and have trouble synchronizing metabase when I try to synchronize with dbt-metabase export
.
The console prints a bunch of
...
2021-09-13 17:26:21,324 - WARNING - Model API_DATASET not found
2021-09-13 17:26:21,324 - WARNING - Model API_DATASETITEM not found
2021-09-13 17:26:21,324 - CRITICAL - Sync timeout reached, models still not compatible
None of my models are upper case - could that be the reason why the models are not found?
Docs say you can use:
dbt_schema_excludes:
- development
- testing
And there's also support for a --dbt_schema_excludes
command-line option.
However, you must actually upcase the schema name, unlike the example before, because the code that actually looks at this is:
if schema_excludes and node["schema"].upper() in schema_excludes:
I'm not really clear why the upper()
is there. If it's just for the sake of normalising, then we also need to normalise the arguments?
In my personal opinion, if semantic_type is not configured in the dbt schema, the value currently set in Metabase should be retained.
I was surprised when I ran dbt-metabase (in order to upload column descriptions) and found that Metabase's initial guesses at the semantic types (which were mostly correct), plus a few that I'd done manually, had all been reset.
To avoid this, IMHO the semantic type should only be set if metabase.semantic_type is explicitly specified. Changing this behavior would have the downside that if you had previously set metabase.semantic_type and then removed it, the type would not be un-set. But in my opinion that is not surprising behavior.
@gouline do you agree? I won't try and have a go this one if you wouldn't accept it.
I got the following error : Error: No such option: --schema Did you mean --dbt_schema?
when running the README example
dbt-metabase models \
--dbt_path . \
--dbt_database business \
--metabase_host metabase.example.com \
--metabase_user [email protected] \
--metabase_password Password123 \
--metabase_database business \
--schema public
It's seems that the readme is outdated, schema
flag must be replaced by dbt_schema
as follow :
dbt-metabase models \
--dbt_path . \
--dbt_database business \
--metabase_host metabase.example.com \
--metabase_user [email protected] \
--metabase_password Password123 \
--metabase_database business \
--dbt_schema public
dbt-metabase==0.9.3
See discussion on PR#29.
The current CLI arguments --mb_https
and --sync
can only be set to false from the command line by passing an empty string. This isn't intuitive.
Suggested changes:
mb_https
and propagate mb_http
downstream to make programmatic interface consistent with CLI interface--sync
Whether I try dbt-metabase programmatically or with CLI, each time it seems that it cannot find the appropriate database to reference. I could put any string there or even leave it as a blank string and the same issue arises; nothing is being read.
Is this the correct way to load these variables? The DbtInterface seems to be the only portion not funcitoning properly, but I'm unsure how exactly to resolve the issue of it not reading things properly
dbt = DbtInterface(
path='/home/bradmcglynn/vm_dbt_package/rvezy-dbt-1/models/reports/dimensions',
manifest_path='/home/bradmcglynn/vm_dbt_package/rvezy-dbt-1/target/manifest.json',
database='rvezy-1379',
schema='dimensions.yml',
schema_excludes=None,
#includes=dbt_includes,
excludes=None,
)
We've just started using dbt-metabase exposures
and I've noticed that the output YAML replaces all spaces in the Metabase card names with underscores (e.g. a card with name "Votes by country" becomes "Votes_by_country"). This makes it a bit difficult to read when published in the docs since we wouldn't have expected that title/name to be edited in the process. Is there an option to avoid this?
It appears that the code to pull exposures from native queries is unable to recognize model names when they're qualified.
I'd expect the below metabase query to generate an exposure dependent on my_model
, but it doesn't.
select * from db.schema.my_model
I think the bug is that the regex used to find from
and join
clauses doesn't allow for .
or "
characters, despite the subsequent code appearing to account for those cases.
I'd propose that the regex should instead allow .
and "
characters.
r"[FfJj][RrOo][OoIi][MmNn]\s+\b([\w.\"]+)\b"
This logic does not account for self-referencing tables:
dbt-metabase/dbtmetabase/parsers/dbt_manifest.py
Lines 209 to 217 in e7c7832
Could you please add an option to connect to metabase API with HTTP ? Currently HTTPS is required.
Seems like there was a commit to add metabase_session_id
as a CLI option, but it was added as a parameter only for model
and not for exposures
. Adding the corresponding lines to exposures
resolved the issue for me.
I'd wager that the usability and prevalence of this marvelous packages would be greatly advanced if there was a basic documentation to go along with the two major models (MetabaseInterface
and DbtInterface
). For instance, it took me a few hours to realize that my implementation wasn't working because schema
is, in fact, not a mandatory field but rather a field limiting the amount of models being parsed and synced.
I think this is a great tool that is easily usable if you know how to, where a bit of docs could go a long way :)
Hello,
when a user is deactivated, metabase returns 404 for any API requests regarding this user, and this causes an exception during exposures sync:
Traceback (most recent call last):
File "/usr/local/lib/python3.8/site-packages/dbtmetabase/metabase.py", line 847, in api
response.raise_for_status()
File "/usr/local/lib/python3.8/site-packages/requests/models.py", line 940, in raise_for_status
raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 404 Client Error: Not Found for url: https://metabase-internal.gatserver.com/api/user/15
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/bin/dbt-metabase", line 5, in <module>
dbtmetabase.main()
File "/usr/local/lib/python3.8/site-packages/dbtmetabase/__init__.py", line 720, in main
cli(max_content_width=600) # pylint: disable=unexpected-keyword-arg
File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1128, in __call__
return self.main(*args, **kwargs)
File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1053, in main
rv = self.invoke(ctx)
File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1659, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/usr/local/lib/python3.8/site-packages/dbtmetabase/__init__.py", line 113, in invoke
return super().invoke(ctx)
File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1395, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/usr/local/lib/python3.8/site-packages/click/core.py", line 754, in invoke
return __callback(*args, **kwargs)
File "/usr/local/lib/python3.8/site-packages/dbtmetabase/__init__.py", line 262, in wrapper
return func(*args, **kwargs)
File "/usr/local/lib/python3.8/site-packages/dbtmetabase/__init__.py", line 709, in exposures
metabase.client.extract_exposures(
File "/usr/local/lib/python3.8/site-packages/dbtmetabase/metabase.py", line 583, in extract_exposures
creator = self.api("get", f"/api/user/{exposure['creator_id']}")
File "/usr/local/lib/python3.8/site-packages/dbtmetabase/metabase.py", line 849, in api
if "password" in kwargs["json"]:
There's missing error handler at:
dbt-metabase/dbtmetabase/metabase.py
Line 595 in 275a72d
creator = self.api("get", f"/api/user/{exposure['creator_id']}")
creator_email = creator["email"]
creator_name = creator["common_name"]
I'm considering submitting a PR where 404 errors would be handled and if is happening, both creator email and name would be empty. I wanted to confirm if that would be an acceptable solution in this case?
It would be useful to automatically hide/show models in Metabase. The metabase API endpoint PUT /api/table/:id
accepts a visibility_type
value that may be nil, or if non-nil, value must be one of: cruft
, hidden
, technical
. Can we allow the
meta:
metabase.visibility_type: hidden
to be defined on models?
BigQuery doesn't have the concept of schema. For this reason, I am getting a lot of errors on the upper() functions applied to NoneType (this happens when retrieving metadata for tables and fields).
I suggest to assume the schema is public
when the API returns None.
An Issue to keep track of what should be updated in the docs before releasing 0.8.0:
export models
and export exposures
in the CLI commandsAnything else you are aware?
I can open a PR if needed. I just wanted to do many things in one-go ๐
The export_model
function supports setting the display name for models from a display_name
property on MetabaseModel
.
However, this isn't accessible, as the meta:
metabase.display_name
property is not parsed and populated in the construction of the MetabaseModel
in _read_model
.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.