Konan is your one-stop platform to bring your AI models to life. Typically, one would be required to do some tedious boring work in order to prepare the machine learning or data science model for deployment. Not with us! With Konan, gone are the days of worrying about a model's dependencies, battling with containerization or obsessively hand-holding the infrastructure. This repo will guide you in preparing your machine learning model in as seamless a manner as possible! Of course, don't forget to also checkout Konan's rich documentation for additional info.
This repository contains the source code and tooling necessarily to generate the konan-housing-model
docker image, a Konan-ready ML model packaged and distributed as a Docker image. The model aims to estimate the sale price of house based on the popular Ames Housing Dataset and the Jupyter notebook titled #1 House Prices Solution [top 1%], generously provided to the public by Nanashi.
You can also find the built image and its tags on Dockerhub directly.
notebook.ipynb
contains the original Jupyter notebook as provided by Nanashi. It is used as a reference for thepython
scripts produced by this repo in thehousing-model/src/playground
directory.- The
.konan.example
environment file is provided as a template for end users, should they wish to replicate building this repository and generate a different Konan-compatible docker image from it. build.sh
is the script that the Github Workflow for this repo uses to generate the new ML-model to be used in the docker images.- The
tests/
directory contains the unit and integration tests that the Github Workflow will use to test any new changes added to the repo. - The
data/
directory contains the datasets used by this repository- The
data/kaggle
directory contains a copy of the original Ames Housing Dataset, sourced from Kaggle directly. - The
data/final
directory contains cleaned.csv
files that are Konan-compatible. In particular, some columns are renamed, others are dropped. This is the dataset that is used totrain
,validate
andtest
the Konan-models generated by this repo. - The
data/metadata.yml
file contains some crucial information for the encoders used by the ML-model.
- The
housing-model
is the main directory for the repo. Some highlights about its contents include:- The
housing-model/retrain.sh
script file includes the command to trigger a Model Retraining on Konan for the generated model.- Check konan-docs for more information on Model Retraining.
- The
housing-model/src/playground
directory contains thepython
scripts that generates the cleaned datasets (data/final/*
files), given the original datasets from Kaggle. (data/kaggle/*
files) and thedata/metadata.yml
file. - The
housing-model/src/app
directory contains thepython
scripts for the Konan model's web-server, including theretrain.py
andserver.py
files.- Check konan-docs for more information on the requirements for generating a Konan Model.
- The
housing-model/src/serving
directory containspython
scripts that useskonan-sdk
to interact with a Konan deployment created using akonan-housing-model
image.- Check the documentation of
konan-sdk
to learn more about how to use it. - Make sure to generate a
.env
file from the provided.env.example
example file.
- Check the documentation of
- The
To ensure that your local environment contains all of the required dependencies, run poetry install
to install the dependencies in a virtual environment. You can learn more about poetry here.
- Go to the Dockerhub page for this image and select the tag you want to deploy, and make note of the image url. The image url of your chosen tag is the part after the corresponding
docker pull
.-
For example, if you are using the
latest
tag, then the image url will be:konanai/konan-housing-model:latest
-
Don't forget to updated the
latest
tag. It is always recommended to use a specific, versioned tag and notlatest
.
-
- Go to Konan, login and deploy this image as a new Deployment! Perhaps you'd find the relevant instructions on Konan docs useful.
- Use the
image url
that you took note of in the previous step. - Leave the Container Registry Credentials blank, as you are using a publicly-available image.
- Use the
After following Konan's documentation on deploying the konan-housing-model
on Konan
, you may wish to use the provided housing-model/src/serving/main.py
script to interact with your newly-created deployment.
-
Log in to Konan and retrieve the
UUID
of the Deployment you created using thekonan-housing-model
. -
Create a
.env
file in thehousing-model/src/serving
directory using the provided.env.example
example file there. -
Export the needed environment variables
. ./housing-model/src/serving/.env
-
Trigger some predictions and provide their feedback using the
housing-model/src/serving/main.py
script filepython housing-model/src/serving/main.py
python housing-model/src/playground/prepare.py
-
Export some needed environment variables
export KONAN_MODEL_REGRESSOR_NAME="svr" # only svr and xgboost are currently supported
-
Create an empty
artifacts/<ml-algorithm>
directorymkdir -p artifacts/"svr" # must match the name of the above regressor name
-
Run the
train
scriptpython housing-model/src/playground/train.py
-
Ensure you have trained the model, as described in the above steps
-
Export the same needed environment variables, with the same values
export KONAN_MODEL_REGRESSOR_NAME="svr" # only svr and xgboost are currently supported
-
Run the
validate
scriptpython housing-model/src/playground/validate.py
Although you may use the official Konan docker image for your use, you may of course use the provided source to build (and optionally) publish your own version of the docker image.
-
Create a
.konan
environment variable from the provided.konan.example
example, and don't forget to update it with your container registry credentials and desired image name and tag. -
Use the
Makefile
to build the imagemake build
-
Use the
Makefile
to login to Dockerhubmake repo-login
-
Use the
Makefile
to publish the imagemake publish-version
Of course, feel free to run make --help
to familiarize yourself with the other options the Makefile
provides.
You are more than welcome to contribute to this repo's source code, documentation, and tooling scripts. We ask, however, that you keep your commits self-contained and to adhere to the conventional commits standard. This helps us automate pushing new tags and builds through semantic versioning.
If you have any questions, or feel that this repo needs some improvement, feel free to raise an issue with the repository's Issue Tracker.