Please visit github.com/scaleoutsystems/stackn for a maintained fork of the LEAN AI STACK.

The Lean AI Stack is a open source project aiming to be a complete solution for working on End to End machine learning. From experiments and exploring datasets to large-scale training to end user serving and monitoring of models and their performance in production. In addition to supporting the basic workflow of finding good models it also support extensive customization and adding value through setting up automated machine learning pipelines. This solution builds on best of breed open source software to provide a complete but totally configurable solution for organizational, institutional or individual needs. We are happy to have your support on this project. See Contributing.

As this project is still in early development expect to see a high paced iteration and added functionality in the coming months. For additional feature requests or bugs please use the issue ticketing system. A rough roadmap is still in development but your input is very welcome. See Contributing.

We aim to provide documentation for the general case setup. If your are a moderate to advanced user and/or have already a K8s cluster running or other prereqs already met you can skip steps below and start on the step corresponding to your current state.

Read more about the license here

Read more on how to contribute here

Follow the guide to setup the reference cluster infrastructure. Infrastructure Setup Guide

Follow along in the readme in Cert

1. Ensure you have a cluster ready. From instructions above or other.
2. Ensure you have a loaded $KUBECONFIG from env or other place.
3. Ensure you have installed and configured helm, check that helm version shows also the server version and you are ready to go!

helm repo add leanai https://leanaiorg.github.io/leanaistack/helm-charts/leanaistack/
helm repo update

To refresh dependencies before installing run the following command from this "root" directory

helm dep up

cp values.yaml values-local.yaml

Edit as appropriate.

from "root" directory and override values with your values file.

helm install leanai/leanaistack -n leanai  --values=values-local.yaml

helm install --upgrade leanai/leanaistack -n leanai --values=values-local.yaml

helm delete --purge leanai

JupyterHub is provided as a hub for your experiments and collaboration.

Minio is provided as a S3 compatible storage backend for your datasets and files.

The default cluster sets up a dynamic storage provisioner that can be utilized for your services and workflows to store datasets and files.

Docker registry provides a storage location for your docker image harbouring needs.

The workflow engine powered by Argoproj enables versatile workflow definitions to complete arbitrary tasks. In examples there will be applied usages of workflows for ML/AI.

The signals system powered by Argoproj enables versatile extensions and allow for sensor and triggering events customizable to allow for event-action coupling of workflows. In examples there will be applied usages of eventing and worklows for ML/AI.

For serving models the OpenFaaS project solution is used that can serve models packaged as docker containers and can scale up based on usage or scale to zero on long periods of non use.

The provided examples are meant to be run in already secured environments as this solution is experimental at the moment. There are however configuration options to allow for adding basic-auth protection to services and allow for TLS wrapped communication.

The user of this open source project is fully aware that this project comes with absolutely no warranty or insurance.

Ingress is provided if required and can be configured. See example values.yaml.

Several examples are in the making and adopting from real world applications. The examples archive you can find here and will be added to continiously: On example is to try out the basic workflow engine with hello-world.yamlfrom Examples.

Create docker secret based on your credentials for pulling images from private repos.

kubectl create secret docker-registry regcred --docker-server=<your-registry-server> --docker-username=<your-name> --docker-password=<your-pword> --docker-email=<your-email>