petehunt / dagster-poor-mans-data-lake Goto Github PK

Python 94.63% Shell 5.37%

dagster-poor-mans-data-lake's Introduction

jaffle

This is a Dagster project scaffolded with dagster project scaffold.

Getting started

./install-deps.sh

Then, start the Dagit web server:

dagit

Open http://localhost:3000 with your browser to see the project.

You can start writing assets in jaffle/assets/. The assets are automatically loaded into the Dagster repository as you define them.

Development

Adding new Python dependencies

You can specify new Python dependencies in setup.py.

Unit testing

Tests are in the jaffle_tests directory and you can run tests using pytest:

pytest jaffle_tests

Schedules and sensors

If you want to enable Dagster Schedules or Sensors for your jobs, start the Dagster Daemon process in the same folder as your workspace.yaml file, but in a different shell or terminal.

The $DAGSTER_HOME environment variable must be set to a directory for the daemon to work. Note: using directories within /tmp may cause issues. See Dagster Instance default local behavior for more details.

dagster-daemon run

Once your Dagster Daemon is running, you can start turning on schedules and sensors for your jobs.

Deploy on Dagster Cloud

The easiest way to deploy your Dagster project is to use Dagster Cloud.

Check out the Dagster Cloud Documentation to learn more.

dagster-poor-mans-data-lake's People

Contributors

Stargazers

Watchers

Forkers

opendgps basalganglia sksundaram-learning tuananh ezpzbz benpankow sabman dataspeaks2020 skanduru lenapheno alexgutierr lllong33 tangguoqiang172528725 edwardjackchandler tintinrevient mathaix

dagster-poor-mans-data-lake's Issues

Error: The "distutils" standard module, which is required for the installation of Docutils, could not be found.

Both on my local machine and in a gitpod I am getting this error below. I tried to add it:

gitpod /workspace/empty/jaffle (main) $ sudo apt-get install python3-distutils 
Reading package lists... Done
Building dependency tree       
Reading state information... Done
python3-distutils is already the newest version (3.8.10-0ubuntu1~20.04).
0 upgraded, 0 newly installed, 0 to remove and 46 not upgraded.

But I still get the error:

INFO: pip is looking at multiple versions of packaging to determine which version is compatible with other requirements. This could take a while.
  Using cached docutils-0.15.1-post1.tar.gz (1.7 MB)
  Preparing metadata (setup.py) ... error
  error: subprocess-exited-with-error
  
  × python setup.py egg_info did not run successfully.
  │ exit code: 1
  ╰─> [4 lines of output]
      Error: The "distutils" standard module, which is required for the
      installation of Docutils, could not be found.  You may need to
      install a package called "python-devel" (or similar) on your
      system using your package manager.
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed

× Encountered error while generating package metadata.
╰─> See above for output.

note: This is an issue with the package mentioned above, not pip.
hint: See above for details.

Actually using AWS S3?

I have it sort of working - I am getting "something" uploaded to S3 but I am getting a multipart error. Has anyone tried to connect this to "real" S3 and what tricks do I need to know? Thanks!

error:
duckdb.Error: Invalid Error: IO Error: Unexpected response during S3 multipart upload finalization

I don't think the error is with DuckDB however, because I can connect it directly to my S3 bucket using the exact same credentials and it works just fine. I suspect it's with our hand-rolled DuckPondIOManager but I don't know how to debug it.

ImportError when running on gitpod

Hi all, I have followed your Poor man data lake tutorial but encounter this error/bug when run dagit ,
ImportError: cannot import name 'introspection_query' from 'graphql' (/workspace/.pyenv_mirror/user/current/lib/python3.8/site-packages/graphql/__init__.py).

However, according to tutorial, dagster should run properly.

Are there any fixes for this? I have looked around but still not found any solutions.

run dagit AttributeError

AttributeError: module 'pendulum' has no attribute 'Pendulum'