👉 From now on, you will start each new challenge with the solution of the previous challenge

👉 Each new challenge will bring in an additional set of features

Here are the main files of interest:

.
├── .env                            # ⚙️ Single source of all config variables
├── .envrc                          # 🎬 .env automatic loader (used by direnv)
├── Makefile                        # New commands "run_train", "run_process", etc..
├── README.md
├── requirements.txt
├── setup.py
├── taxifare
│   ├── __init__.py
│   ├── interface
│   │   └── main_local.py           # 🚪 (OLD) entry point
│   │   └── main.py                 # 🚪 (NEW) entry point: No more chunks 😇 - Just process(), train()
│   ├── ml_logic
│       ├── data.py                 # (UPDATED) Loading and storing data from/to Big Query !
│       ├── registry.py             # (UPDATED) Loading and storing model weights from/to Cloud Storage!
│       ├── ...
│   ├── params.py                   # Simply load all .env variables into python objects
│   └── utils.py
└── tests

This file is a template designed to help you create a .env file for each challenge. The .env.sample file contains the variables required by the code and expected in the .env file. 🚨 Keep in mind that the .env file should never be tracked with Git to avoid exposing its content, so we have added it to your .gitignore.

Bye bye taxifare.interface.main_local module, you served us well ❤️

Long live taxifare.interface.main, our new package entry point ⭐️ to:

• preprocess: preprocess the data and store data_processed
• train: train on processed data and store model weights
• evaluate: evaluate the performance of the latest trained model on new data
• pred: make a prediction on a DataFrame with a specific version of the trained model

🚨 One main change in the code of the package is that we chose to delegate some of its work to dedicated modules in order to limit the size of the main.py file. The main changes concern:

• The project configuration: Single source of truth is .env

  • .envrc tells direnv to loads the .env as environment variables
  • params.py then loads all these variable in python, and should not be changed manually anymore

• registry.py: the code evolved to store the trained model either locally or - spoiler alert - in the cloud

  • Notice the new env variable MODEL_TARGET (local or gcs)

• data.py has refactored 2 methods that we'll use heavily in main.py

  • get_data_with_cache() (get some data from Big Query or cached CSV if exists)
  • load_data_to_bq() (upload some data to BQ)

💻 Install the new package version

make reinstall_package # always check what make do in Makefile

🧪 Check the package version

pip list | grep taxifare
# taxifare               0.0.7

Our goal is to be able to configure the behavior of our package 📦 depending on the value of the variables defined in a .env project configuration file.

💻 In order to do so, we will install the direnv shell extension. Its job is to locate the nearest .env file in the parent directory structure of the project and load its content into the environment.

# MacOS
brew install direnv

# Ubuntu (Linux or Windows WSL2)
sudo apt update
sudo apt install -y direnv

Once direnv is installed, we need to tell zsh to load direnv whenever the shell starts

code ~/.zshrc

The list of plugins is located in the beginning of the file and should look like this when you add direnv:

plugins=(...direnv)

Start a new zsh window in order to load direnv

💻 At this point, direnv is still not able to load anything, as there is no .env file, so let's create one:

• Duplicate the env.sample file and rename the duplicate as .env
• Enable the project configuration with direnv allow . (the . stands for current directory)

🧪 Check that direnv is able to read the environment variables from the .env file:

echo $DATA_SIZE
# 1k --> Let's keep it small!

From now on, every time you need to update the behavior of the project:

1. Edit .env, save it
2. Then

direnv reload . # to reload your env variables 🚨🚨

☝️ You will forget that. Prove us wrong 😝

# Ok so, for this unit, alway keep data size values small (good practice for dev purposes)
DATA_SIZE=1k
CHUNK_SIZE=200

Google Cloud Platform will allow you to allocate and use remote resources in the cloud. You can interact with it via:

• 🌐 console.cloud.google.com
• 💻 Command Line Tools
  • gcloud
  • bq (big query - SQL)
  • gsutils (cloud storage - buckets)

• Find the gcloud command that lists your own GCP project ID.
• 📝 Fill in the GCP_PROJECT variable in the .env project configuration with the ID of your GCP project
• 🧪 Run the tests with make test_gcp_project

The second CLI tool that you will use often allows you to deal with files stored within buckets on Cloud Storage.

We'll use it to store large & unstructured data such as model weights :)

💻 Create a bucket in your GCP account using gsutil

• Make sure to create the bucket where you are located yourself (use GCP_REGION in the .env)
• Fill also the BUCKET_NAME variable with the name of your choice (must be globally unique and lower case!)

e.g.

BUCKET_NAME = taxifare_<user.github_nickname>

• direnv reload . ;)

Tips: The CLI can interpolate .env variables by prefix them with a $ sign (e.g. $GCP_REGION)

gsutil ls                               # list buckets

gsutil mb \                             # make bucket
    -l $GCP_REGION \
    -p $GCP_PROJECT \
    gs://$BUCKET_NAME                     # make bucket

gsutil rm -r gs://$BUCKET_NAME               # delete bucket

🧪 Run the tests with make test_gcp_bucket

Biq Query is a data-warehouse, used to store structured data, that can be queried rapidly.

💡 To be more precise, Big Query is an online massively-parallel Analytical Database (as opposed to Transactional Database)

• Data is stored by columns (as opposed to rows on PostGres for instance)
• It's optimized for large transformation such as group-by, join, where etc...easily
• But it's not optimized for frequent row-by-row insert/delete

Le WagonCab is actually using a managed postgreSQL (e.g. Google Cloud SQL) as its main production database on which it's Django app is storing / reading hundred thousands of individual transactions per day!

Every night, Le WagonCab launch a "database replication" job that applies the daily diffs of the "main" postgresSQL into the "replica" Big Query warehouse. Why?

• Because you don't want to run queries directly against your production-database! That could slow down your users.
• Because analysis is faster/cheaper on columnar databases
• Because you also want to integrate other data in your warehouse to JOIN them (e.g marketing data from Google Ads...)

👉 Back to our business:

💻 Let's create our own dataset where we'll store & query preprocessed data !

• Using bq and the following env variables, create a new dataset called taxifare on your own GCP_PROJECT

BQ_DATASET=taxifare
BQ_REGION=...
GCP_PROJECT=...

• Then add 3 new tables processed_1k, processed_200k, processed_all

bq mk \
    --project_id $GCP_PROJECT \
    --data_location $BQ_REGION \
    $BQ_DATASET

bq mk --location=$GCP_REGION $BQ_DATASET.processed_1k
bq mk --location=$GCP_REGION $BQ_DATASET.processed_200k
bq mk --location=$GCP_REGION $BQ_DATASET.processed_all

bq show
bq show $BQ_DATASET
bq show $BQ_DATASET.processed_1k

🧪 Run the tests with make test_big_query

🎁 Look at make reset_all_files directive --> It resets all local files (csvs, models, ...) and data from bq tables and buckets, but preserve local folder structure, bq tables schema, and gsutil buckets.

Very useful to reset state of your challenge if you are uncertain and you want to debug yourself!

👉 Run make reset_all_files safely now, it will remove files from unit 01 and make it clearer

👉 Run make show_sources_all to see that you're back from a blank state!

✅ When you are all set, track your results on Kitt with make test_kitt (don't wait, this takes > 1min)

🎯 Your goal is to fill-up taxifare.interface.main so that you can run every 4 routes one by one

if __name__ == '__main__':
    # preprocess()
    # train()
    # evaluate()
    # pred()

To do so, you can either:

• 🥵 Uncomment the routes above, one after the other, and run python -m taxifare.interface.main from your Terminal

• 😇 Smarter: use each of the following make commands that we created for you below

💡 Make sure to read each function docstring carefully 💡 Don't try to parallelize route completion. Fix them one after the other. 💡 Take time to read carefully the tracebacks, and add breakpoint() to your code or to the test itself (you are 'engineers' now)!

Preprocess

💡 Feel free to refer back to main_local.py when needed! Some of the syntax can be re-used

# Call your preprocess()
make run_preprocess
# Then test this route, but with all combinations of states (.env, cached_csv or not)
make test_preprocess

Train

💡 Be sure to understand what happens when MODEL_TARGET = 'gcs' vs 'local' 💡 We advise you to set verbose=0 on model training to shorter your logs!

make run_train
make test_train

Evaluate

Be sure to understand what happens when MODEL_TARGET = 'gcs' vs 'local'

make run_evaluate
make test_evaluate

Pred

This one is easy

make run_pred
make test_pred

✅ When you are all set, track your results on Kitt with make test_kitt

🏁 Congrats for the heavy refactoring! You now have a very robust package that can be deployed in the cloud to be used with DATA_SIZE='all' 💪

In GCP, many services are not enabled by default. The service to activate in order to use virtual machines is Compute Engine.

❓How do you enable a GCP service?

Find the gcloud command to enable a service.

The taxifare package is ready to train on a machine in the cloud. Let's create our first Virtual Machine instance!

❓Create a Virtual Machine

Head over to the GCP console, specifically the Compute Engine page. The console will allow you to easily explore the available options. Make sure to create an Ubuntu instance (read the how-to below and have a look at the hint after it).

INSTANCE=taxi-instance
IMAGE_PROJECT=ubuntu-os-cloud
IMAGE_FAMILY=ubuntu-2204-lts

gcloud compute instances create $INSTANCE --image-project=$IMAGE_PROJECT --image-family=$IMAGE_FAMILY

💻 Fill in the INSTANCE variable in the .env project configuration

You have access to virtually unlimited computing power at your fingertips, ready to help with trainings or any other task you might think of.

❓How do you connect to the VM?

The GCP console allows you to connect to the VM instance through a web interface:

You can disconnect by typing exit or closing the window.

A nice alternative is to connect to the virtual machine right from your command line 🤩

All you need to do is to gcloud compute ssh on a running instance and to run exit when you want to disconnect 🎉

INSTANCE=taxi-instance

gcloud compute ssh $INSTANCE

❓How do you setup the VM to run your python code?

Let's run a light version of the Le Wagon setup.

💻 Connect to your VM instance and run the commands of the following sections

sudo apt update
sudo apt install -y zsh
sh -c "$(curl -fsSL https://raw.github.com/ohmyzsh/ohmyzsh/master/tools/install.sh)"

git clone https://github.com/pyenv/pyenv.git ~/.pyenv
git clone https://github.com/pyenv/pyenv-virtualenv.git ~/.pyenv/plugins/pyenv-virtualenv

nano ~/.zshrc

cat ~/.zshrc | grep "plugins="

cat << EOF >> ~/.zprofile
export PYENV_ROOT="\$HOME/.pyenv"
export PATH="\$PYENV_ROOT/bin:\$PATH"
eval "\$(pyenv init --path)"
EOF

sudo apt-get update; sudo apt-get install make build-essential libssl-dev zlib1g-dev \
libbz2-dev libreadline-dev libsqlite3-dev wget curl llvm \
libncursesw5-dev xz-utils tk-dev libxml2-dev libxmlsec1-dev libffi-dev liblzma-dev \
python3-dev

zsh --login

# e.g. with 3.10.6
pyenv install 3.10.6
pyenv global 3.10.6
pyenv virtualenv 3.10.6 taxifare-env
pyenv global taxifare-env

INSTANCE=taxi-instance

# scp stands for secure copy (cp)
gcloud compute scp ~/.ssh/id_ed25519 $USER@$INSTANCE:~/.ssh/

eval "$(ssh-agent -s)"
ssh-add ~/.ssh/id_ed25519

gcloud auth application-default login

pip install -U pip
pip install google-cloud-storage

python -c "from google.cloud import storage; \
    buckets = storage.Client().list_buckets(); \
    [print(b.name) for b in buckets]"

Your VM is now fully operational with:

• A python venv (lewgon) to run your code
• The credentials to connect to your GitHub account
• The credentials to connect to your GCP account

The only thing that is missing is the code of your project!

🧪 Let's run a few tests inside your VM Terminal before we install it:

• Default shell is /usr/bin/zsh

  echo $SHELL

• Python version is 3.10.6

  python --version

• Active GCP project is the same as $GCP_PROJECT in your .env file

  gcloud config list project

Your VM is now a data science beast 🔥

Let's run your first training in the cloud!

❓How do you setup and run your project on the virtual machine?

💻 Clone your package, install its requirements

git clone [email protected]:<user.github_nickname>/{{local_path_to("07-ML-Ops/02-Cloud-training/01-Cloud-training")}}

cd <path/to/the/package/model/dir>

make reset_local_files

cp .env.sample .env

nano .env

sudo apt update
sudo apt install -y direnv

zsh --login

direnv allow .

pip install .

🔥 Run the preprocessing and the training in the cloud 🔥!

make run_all

Project not set error from GCP services? You can add a GCLOUD_PROJECT environment variable that should be the same as your GCP_PROJECT

🧪 Track your progress on Kitt to conclude (from your VM)

make test_kitt

🏋🏽‍♂️ Go Big: re-run everything with DATA_SIZE = 'all' CHUNK_SIZE=100k chunks for instance 🏋🏽‍♂️!

🏁 Switch OFF your VM to finish 🌒

You can easily start and stop a VM instance from the GCP console, which allows you to see which instances are running.

INSTANCE=taxi-instance

gcloud compute instances stop $INSTANCE
gcloud compute instances list
gcloud compute instances start $INSTANCE

🚨 Computing power does not grow on trees 🌳, do not forget to switch the VM off whenever you stop using it! 💸