Welcome to the beginning of your journey to becoming an ML Operations (MLOps) Engineer! ๐ Follow these steps to get your development environment teed up! After you've finished this set-up, feel free to go through the associated Whodunit?
! ๐ต๏ธโโ๏ธ
We will be using some terminal commands, so let's make sure you know what they are and what they do!
Command | Stands For | Description |
---|---|---|
ls |
long listing | lists all files and directories in the present working directory |
ls -a |
long listing all | lists hidden files as well |
cd {dirname} |
change directory | to change to a particular directory |
cd ~ |
change directory home | navigate to HOME directory |
cd .. |
change directory up | move one level up |
cat {filename} |
concatenate | displays the file content |
sudo |
superuser | allows regular users to run programs with the security privileges of the superuser or root |
mv {filename} {newfilename} |
move | renames the file to new filename |
clear |
clear | clears the terminal screen |
mkdir {dirname} |
make directory | create new directory in present working directory or at specified path |
rm {filename} |
remove | remove file with given filename |
touch {filename}.{ext} |
touch | create new empty file |
rmdir {dirname} |
remove directory | deletes a directory |
ssh {username}@{ip-address} or {hostname} |
secure shell | login into a remote Linux machine using SSH |
We will also be using a few tools such as git
, conda
, and pip
.
Git
Git is a free and open source distributed version control system designed to handle everything from small to very large projects. These are the commands we will be using with git
:
git clone
-> clone a remote repository to your local computer
git add
-> add files to a commit
git commit -m {message}
-> commit changes with a message
git push
-> push commit to remote repository
Conda & Pip
Conda is an open-source, cross-platform, language-agnostic package manager and environment management system. We will use pip
within conda
environments to manage our package installations. pip
is Python's package management system. conda
comes with Anaconda. And Anaconda is a convenient way to set up your Python programming environment since it comes with an enviornment management tool (conda
) and comes with extra packages that are commonly used in data science and ML.
Some commands we will use in this lesson when it comes to conda
and pip
:
conda create --name mlops-course python=3.8 pip
-> This creates a virtual environment. A virtual environment is a Python environment such that the Python interpreter, libraries, amnd scripts installed into it are isolated from those installed on other environments and any libraries installed on the system. So basically, this allows you to keep all your project's code/dependencies/libraries separated from other projects. You are specifically saying to create said environment with the name mlops-course
, use python
version 3.8, and use pip
as your package manager. The command conda
invokes the underlying logic to actually make the virtual environment and manages said environments for you.
conda activate mlops-course
-> This activates the virtual environment you made with the above command for your current terminal session.
pip install numpy pandas matplotlib jupyter
-> This installs the four packages mentioned - numpy
, pandas
, jupyter
and matplotlib
. numpy
is used for scientific computing, pandas
is used for data analysis, and matplotlib
is used for data graphics. jupyter
is discussed later in this tutorial in depth! pip
is the Python package manager and you are telling it to install
the listed packages to your environment.
Let's start off by setting up our environment! Review the environment setup instructions for the local environment that you'll be using in this course.
Windows
- Install Windows Subsystem for Linux using Powershell
wsl --install -d Ubuntu-20.04
-
Install Windows Terminal (You can even make it your default!)
-
Install Ubuntu
-
Make sure you've install the correct version with the command
wsl -l -v
(If you find yourself getting stuck on the WSL2 install, here is a link to video instructions)
Give it a test drive!
Continue by installing the following tools using Windows Terminal to setup your environment. When prompted, make sure to add conda
to init
.
Tool | Purpose | Command |
---|---|---|
๐ Anaconda | Python & ML Toolkits | wget https://repo.anaconda.com/archive/Anaconda3-2021.11-Linux-x86_64.sh bash Anaconda3-2021.11-Linux-x86_64.sh source ~/.bashrc |
![]() |
Version Control | sudo apt update && sudo apt upgrade sudo apt install git-all |
๐ VS Code | Development Environment | Download |
Linux (Debian/Ubuntu)
Open terminal using Ctrl+Shift+T. Enter the following commands in terminal to setup your environment. When prompted, make sure to add conda
to init
.
Tool | Purpose | Command |
---|---|---|
๐ Anaconda | Python & ML Toolkits | wget https://repo.anaconda.com/archive/Anaconda3-2021.11-Linux-x86_64.sh bash Anaconda3-2021.11-Linux-x86_64.sh source ~/.bashrc |
![]() |
Version Control | sudo apt update && sudo apt upgrade sudo apt install git-all |
๐ VS Code | Development Environment | Download |
macOS
To get started, we need to download the MacOS package manager, Homebrew ๐บ, so that we can download the tools we'll be using in the course. If you don't already have Homebrew installed, run the following commands:
-
Open terminal using โ+Space and type
terminal
. -
Install Homebrew using the command below, following the command prompts:
/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"
-
Update Homebrew (This may take a few minutes)
git -C /usr/local/Homebrew/Library/Taps/homebrew/homebrew-core fetch --unshallow
git -C /usr/local/Homebrew/Library/Taps/homebrew/homebrew-cask fetch
-
Install the
wget
command to continue following alongbrew install wget
Enter the following commands in terminal to setup your environment. When prompted, make sure to add conda
to init
.
Tool | Purpose | Command |
---|---|---|
๐ Anaconda | Python & ML Toolkits | wget https://repo.anaconda.com/archive/Anaconda3-2021.11-MacOSX-x86_64.sh bash Anaconda3-2021.11-MacOSX-x86_64.sh source ~/.bashrc |
![]() |
Version Control | brew install git |
๐ VS Code | Development Environment | Download |
Install the IntelliCode Extension
IntelliCode is an AI-powered code completion extension to boost coding productivity. ๐
Install the Python and Jupyter Notebook Extensions
Set the Python Interpreter
-
Open VS Code and click on
New File...
-
Open the Command Pallette (Mac: Shiftโ+P , Windows: Ctrl+Shift+P)
-
Type "Python" in the search bar.
-
Click on
New Python File
-
Open the Command Pallette again. Can you remember the shortcut? If not, see #2 above again.
-
Type "Python Interpreter".
-
Click on
Python: Select Interpreter
-
Select the
Conda
environment that you installed earlier.
- Now you're ready to start coding!
If you don't already have one, make an account on Github!
Github SSH Setup
Secure Shell Protocol (SSH) provides a secure communication channel of an unsecured network. Let's set it up!- Generate a Private/Public SSH Key Pair.
ssh-keygen -o -t rsa -C "your email address for github"
-
Save file pair. Default location
~/.ssh/id_rsa
is fine! -
At the prompt, type in a secure passphrase.
-
Copy the contents of the public key that we will share with GitHub.
-
Mac:
pbcopy < ~/.ssh/id_rsa.pub
-
Windows (WSL):
clip.exe < ~/.ssh/id_rsa.pub
-
Linux:
xclip -sel c < ~/.ssh/id_rsa.pub
-
-
Go to your GitHub account and go to
Settings
. -
Under
Access
, click on theSSH and GPG keys
tab on the left.
- Click on the
New SSH Key
button.
- Name the key, and paste the public key that you copied. Click the
Add SSH Key
button
Creating a New Repository
When viewing the respository page, click on New
and proceed to create your repo.
Filling Respository Details
Create the repository by inputting the following:
Repo name
Repo description
- Make repo
public
- Add a
README
- Add
.gitignore
(Python template) - Add
license
(choose MIT)
Then click Create Repository
.
Clone Your Repo
- Open your terminal and navigate to a place where you would like to make a directory to hold all your files for this class using the command
cd
.
cd {directory name}
- Once there, make a top level directory using
mkdir
.
mkdir {directory name}
cd
into it and make another directory calledcode
.
cd {directory name}
mkdir code
cd
into it and run yourgit clone {your repo url}
command.
cd code
git clone {your repo url}
Adding The FourthBrain Whodunit? Content to Your Repo
cd
into your repo and check your remote git.
cd {your repo name}
git remote -v
At this point, you should just have access to your own repo with an origin branch with both fetch and push options.
- Let's setup our global configuration:
git config --global user.email "your email address"
git config --global user.name "your name"
- Let's add a local branch for development.
git checkout -b LocalDev
You can change anything here in this branch!
git add .
Commit the changes with the branch addition.
git commit -m "Adding a LocalDev branch."
- Let's push our local changes to our remote repo.
git checkout main
git merge LocalDev
git push origin main
- Add the Whodunit (WD) repo as an extra remote repo:
git remote add WD [email protected]:FourthBrain/whodunit.git
Let's check our remote repos:
git remote -v
At this point, you should have access to both your own repo and FourthBrain and should see something like this:
WD [email protected]:FourthBrain/whodunit.git (fetch)
WD [email protected]:FourthBrain/whodunit.git (push)
origin [email protected]:rafatisina/TestRepo.git (fetch)
origin [email protected]:rafatisina/TestRepo.git (push)
Let's update our local repos:
git fetch --all
Make a new branch for the Whodunit material (WDBranch).
git checkout --track -b WDBranch WD/main
You should see something like this:
Branch 'WDBranch' set up to track remote branch 'main' from 'WD'.
You can visually check whether you are in that branch:
git log --all --graph
Now let's push our updated local repo to our remote repo!
git checkout main
git merge WDBranch --allow-unrelated-histories
If there are any conflicts you'll need to resolve them.
git add .
git commit -m "message-here"
git push origin main
From now on... after each release follow these steps to update your repo with new content:
git fetch --all
git checkout WDBranch
git merge --ff-only @{u}
git add .
git commit -m "branch is updated"
git checkout main
git merge WDBranch --allow-unrelated-histories
You will be asked to add a comment about why this change is necessary --> add a message.
git push origin main
Activating Your Conda Environment
- Now, let's activate the environment we set-up earlier with the command
conda activate mlops-course
. If you were successful, you could see(mlops-course)
preceeding your terminal commands.
Adding a Feature Branch
Let's add a feature branch to our local repo. Earlier, we showed you how to add a feature branch and content to your repo via the Terminal. This time we are going to show you how to do it using the VS Code GUI.
-
Click on the
main
branch in the lower left side of the screen
-
You will then see a drop-down menu with some branch-level option commands. Select the
Create a new branch
option. -
You will be prompted to enter the name for the branch. Let's give our branch an informative name
feature-hello-world
. Thefeature
pre-fix is a common Git convention and let's our collaborators know what the purpose of the branch and the name of the feature. -
Now that we have a feature branch to work on, let's add some code to it!
Hello World! - Part 1
1. Next we will review some terminal commands and make some additions to our repo. Do these in your terminal where your current working directory is your repo.-
Check your current working directory:
pwd
-
Create a new file:
touch hello_world.py
-
Create new directory:
mkdir app
-
Move file to directory:
mv hello_world.py app/hello_world.py
-
Check that the move command worked:
cd app
and thenls
, you should see yourhello_world.py
file -
Lastly, lets clear our terminal screen:
clear
print("hello world! let's do some ml ops!")
- Save. And now go to the integrated terminal by clicking
CTRL + ~
. In the terminal run your first program of the class by doingcd app
->python hello_world.py
. Congrats, we are off to a great start!
Hello World! - Part 2 - Notebook Edition
-
Create a new file under
app
by clicking on theAdd file
buttonand let's name this file
hello_world.ipynb
. The.ipynb
extension is a notebook extension which will allow you to interact with your code via a notebook in VS Code, instead of a vanilla Python file. You might need to select your kernel in the top right of the notebook file, if so, choose the one we created previously. -
In the first cell of
hello_world.ipynb
lets do our imports.import pandas as pd import numpy as np import matplotlib.pyplot as plt
-
Run the cell by either clicking the play button or by doing
CTRL + ENTER
. -
Create a new cell and in that put the following code:
np.random.seed(0) values = np.random.randn(100) # array of normally distributed random numbers s = pd.Series(values) # generate a pandas series s.plot(kind='hist', title='Normally distributed random values') # hist computes distribution plt.show()
-
Run the cell and you should see your histogram plot! Well done.
- Now let's commit our code to our remote repository. This can be done one of two ways - either through the terminal or through VS Code's GUI. I'll explain both ways and you can choose which you'll use.
-
Add a message to your commit by typing in the message field.
-
Click the check mark
button under changes to add your files to this commit. If you haven't saved your changes, you will be prompted to
Save All and Commit
. ClickSave All and Commit
. -
Click the elipsis in
Source Control
ribbon and click
Push
. You may also be prompted toSync Changes
. This will do Pull and Push, which will fetch new changes to the code and push your updates as well. -
You can then put in a pull request in GitHub
to merge into the branch that you pulled from, in this case the main branch. In real life, you would then review the code changes with another developer/team lead/supervisor and address any potential code conflicts.
Now let's practice what you have learned by playing the Whodunit? game!