niniiws / orchest Goto Github PK

Website — Docs — Quickstart — Slack

Orchest is a web based data science tool that works on top of your filesystem allowing you to use your editor of choice. With Orchest you get to focus on visually building and iterating on your pipeline ideas. Under the hood Orchest runs a collection of containers to provide a scalable platform that can run on your laptop as well as on a large scale cloud cluster.

Orchest lets you

• Interactively build data science pipelines through its visual interface.
• Automatically run your pipelines in parallel.
• Develop your code in your favorite editor. Everything is filesystem based.
• Tag the notebooks cells you want to skip when running a pipeline. Perfect for prototyping as you do not have to maintain a perfectly clean notebook.
• Run experiments by parametrizing your pipeline. Easily try out all of your modeling ideas.

• Table of contents
• Installation
• Quickstart
  • Build your pipeline.
  • Write your code.
  • Run your pipeline and see the results come in.
• Contributing
• We love your feedback

Requirements

• Docker (tested on 19.03.9)

Linux/macOS/Windows(through WSL 2)

git clone https://github.com/orchest/orchest.git
cd orchest
./orchest.sh start

Note! on Windows Docker should be configured to use WSL 2. Make sure you clone inside the Linux environment. More info about Docker + WSL 2 can be found here: https://docs.docker.com/docker-for-windows/wsl/.

Please refer to our docs for a more comprehensive quickstart tutorial.

Each pipeline step executes a file (.ipynb, .py, .R, .sh) in a containerized environment.

Iteratively edit and run your code for each pipeline step with an interactive JupyterLab session.

Outputs (both stdout and stderr) are directly viewable and stored on disk.

The software in this repository is licensed as follows:

• All content residing under the "orchest-sdk/" directory of this repository is licensed under the "Apache-2.0" license as defined in "orchest-sdk/LICENSE".
• Content outside of the above mentioned directory is available under the "AGPL-3.0" license.

Contributions are more than welcome! Please see our contributer guides for more details.

We would love to hear what you think and potentially add features based on your ideas. Come chat with us on Slack.