hiroshi18 / cloud-data-recap-dm Goto Github PK

cp .env.sample .env
direnv allow
direnv reload

make run_preprocess
make run_train
make run_evaluate

python get_new_data.py jan

python get_new_data.py jan

prefect auth login -k YOUR_KEY

prefect agent local start

make run_workflow

make run_workflow

make run_workflow

Let's track the evolution of the performance of our model accross time using MLflow.

Let's create a .env file:

cp .env.sample .env

And setup the local data path in the .env:

LOCAL_DATA_PATH=~/.lewagon/mlops/data
LOCAL_REGISTRY_PATH=~/.lewagon/mlops/training_outputs

Then direnv allow . and direnv reload and retrieve the latest version of the data using either:

• make reset_sources_all in order to reset datasets of all sizes in local disk + Big Query
• make reset_sources_env in order to reset datasets of size DATASET_SIZE / VALIDATION_DATASET_SIZE in local disk + Big Query

Create an account on Prefect Cloud if you do not have one.

Let's train the model with an initial 500k dataset with:

DATASET_SIZE=500k
VALIDATION_DATASET_SIZE=500k
CHUNK_SIZE=1000000

DATA_SOURCE=local
MODEL_TARGET=mlflow

👉 We boosted the chunk size and use local CSV in order to improve the speed of the trainings

Make sure to update the MLflow and GCP (for Big Query) parameters:

• MLFLOW_EXPERIMENT
• MLFLOW_MODEL_NAME
• PROJECT
• DATASET

Update and reload your .env, then train the model and observe the raw performance in mlflow:

• make run_preprocess
• make run_train

Now that the initial model is trained and stored in MLflow, let's inject new data and see how the model behaves.

Update and reload your .env to play with the new data source:

DATASET_SIZE=new
VALIDATION_DATASET_SIZE=new

For January, February and March:

• Inject new data with python get_new_data.py jan
• Preprocess the data with make run_preprocess
• Observe how the model in production performes on the new data with make run_evaluate
• If the model performance degrade more than $.1, train a new model from the model in production with make run_train
• Observe the performance of the new model
• If the performance of the new model is good enough, annotate the new model to be in production in mlflow

👉 Performance should be stable in January and start to evolve from February

Now that we understand how to track the performance of our model, let's automate the model lifecycle.

Authenticate to Prefect Cloud with prefect auth login -k YOUR_KEY.

Start a local agent so that the outcome of the workflow runs on your machine are persisted in Prefect.

Let's go through our datasets all over again, but this time using Prefect in order to automate everything.

We want to start over from the latest model having been trained from scratch on the 500k dataset.

Go to MLflow and mark the latest model trained from scratch on the 500k dataset as in Production.

Update and reload your .env with parameters for Prefect Cloud:

PREFECT_BACKEND=production
PREFECT_FLOW_NAME=taxifare_lifecycle_<user.github_nickname>

👉 Make sur your project exists in Prefect Cloud (the default is taxifare_project)

Register your workflow:

make run_workflow

Connect to Prefect Cloud:

• 👀 Navigation: verify that your workflow has been uploaded
• 👀 Agents: verify that your local agent is running

Let's inject the January dataset with python get_new_data.py jan.

Then run the workflow once through the Prefect Cloud UI:

• 👀 Quick run

👉 Now that everything is connected, we can schedule the workflow to run automatically

Configure the channel in taxifare.flow.flow module.

Schedule the workflow in the Prefect Cloud UI:

• 👀 Schedules

Have a look at the notification board: https://wagon-chat.herokuapp.com/

The sequential workflow is pretty slow, let's see how we can run it faster.

In the taxifare.flow.flow module, uncomment the LocalDaskExecutor line.

Register the new workflow using make run_workflow.

In the Prefect Cloud UI:

• 👀 Task run: have a look at the logs