- make sure that
python
points topython>=3.8
and you havepip
andgit
thenpip install datazimmer
- run
dz init project-name
- pulls project-template
- add a remote
- both to git and dvc (can run
dz build-meta
to see available dvc remotes) - git remote can be given with
dz init
- both to git and dvc (can run
- create, register and document steps in a pipeline you will run in different environments
- build metadata to exportable and serialized format with
dz build-meta
- if you defined importable data from other artifacts in the config, you can import them with
load-external-data
- ensure that you import envs that are served from sources you have access to
- if you defined importable data from other artifacts in the config, you can import them with
- build and run pipeline steps by running
dz run
- validate that the data matches the datascript description with
dz validate
- a project as a whole has a cron expression in
zimmer.yaml
to determine the schedule of reruns - additionally, aswan projects within the dz project can have different cron expressions for scheduling new runs of the aswan projects
TODO: document dogshow and everything else much better here
- overlapping names convention
- resolve naming confusion with colassigner, colaccessor and table feature / composite type / index base classes
- abstract composite type + subclass of entity class
- import ACT, inherit from it and specify
- importing composite type is impossible now if it contains foreign key :(
- add option to infer data type of assigned feature
- can be problematic b/c pandas int/float/nan issue
- create similar sets of features in a dry way
- overlapping in entities
- detect / signal the same type of entity
- exports: postgres, postgis , superset
- test suite for compliance: https://w3c.github.io/csvw/publishing-snapshots/PR-earl/earl.html
- https://github.com/w3c/csvw
@article{tennison2015model,
title={Model for tabular data and metadata on the web},
author={Tennison, Jeni and Kellogg, Gregg and Herman, Ivan},
year={2015}
}
@article{pollock2015metadata,
title={Metadata vocabulary for tabular data},
author={Pollock, Rufus and Tennison, Jeni and Kellogg, Gregg and Herman, Ivan},
journal={W3C Recommendation},
volume={17},
year={2015}
}