konradbachusz / aws-mlops-module Goto Github PK

Summary problem:
When you run apply the first time, the sagemaker notebook is populated with all of our files from the local mlops_ml_models folder which are pushed to an s3 bucket (you can check the lifecycle config part of the sagemaker terraform file for more information), however, the files do not refresh when you run apply a second time unless you stop the notebook and then restart it.

We need to find a way to restart the notebook even when rerunning apply so as to take out the manual process of stopping and restarting the notebook instance.

Currently, the sagemaker notebook is not built with a container, hence, we need to install all of the libraries manually in the sagemaker notebook. We need to have a container that contains all of the libraries that we will be using and then use that container to build the sagemaker notebook so we do not have to install anything on the notebook

The lightgbm model currently produces this error "AttributeError: 'Booster' object has no attribute 'handle'" for models that were built with it. This error occurs when you try to make predictions with this model using the sagemaker endpoint.

We need to research a possible solution to this and also try to test all the other model endpoints to be sure that we don't have a similar error.

As a DevOps engineer,
When I try to build a docker file in ECR,
The ECR repository should be available for me.

Acceptance Criteria:

• The ECR repository exists in AWS
• It is created with terraform.

As a developer
When I use Terraform apply in the streaming data platform repository
All of the resources that I have in this repo should be deployed.

Acceptance Criteria:

• All of the resources that currently exist in this repo and in AWS should be deployed using the streaming data platform repository.