This project involves building a Github repository from scratch and creating a scaffolding that will be used to perform both Continuous Integration and Continuous Delivery for a Python-based machine learning application using the Flask web framework. It requires the use of Github Actions along with a Makefile, requirements.txt and application code to perform an initial lint, test, and install cycle. Further, the project is integrated with Azure Pipelines to enable Continuous Delivery to Azure App Service.
Below are the links to Trello board and spreadsheet for the project.
Below is the architectural diag the project.
Setup the project repository on Github with the initial project scaffold:
- Makefile
- Tests
- code
- requirements.txt
- Launch an Azure Cloud Development Shell Development Environment
- create ssh-keys
- upload public ssh key to the github account hosting the project repository
Below is the screenshot of the project repo cloned into the azure shell development environment.
-
Update the project files:
Makefile
,requirements.txt
and include the project code files -
Create a Python virtual environment as follows.
python3 -m venv ~/.myrepo
-
Activate the virtual environment by running the fowwing command.
-
source ~/.myrepo/bin/activate
-
Create the project sript file and test files
- Test the code by running
make all
- A screenshot showing passing the test is shown below:
Go to the Github repo and do the following:
- Enable Github Actions in the project
- Replace
yml
code autogenerated by enabling Github Actions with the providedyml
code - Verify remote tests pass in Github Actions UI as shown below:
Configure Azure Pipelines to deploy the Flask starter code to Azure App Services as follows.
- Replace scaffolding code with Flask ML code
- Run the app locally as follows:
-
activate virtual environmen by running the following code:
source ~/.myrepo/bin/activate
followed bypython app.py
-
Open another cloud shell environment
- acitvate the virtual environment and run
./make_prediction.sh
The expected output is
-
- Deploy the app by running
az webapp up -n bostonpredictionservice
- here, bostonpredictionservice is the name of the app
- copy our the generated url:
https://bostonpredictionservice.azurewebsites.net
to the browser and the following message will be displayed.
- To make remote prediction: modify the project code file
make_predict_azure_app.sh
with your chosen app name. - Verify prediction by running the code
./make_predict_azure_app.sh
- The output message is as follows:
- Get log from the deployed app by running the command
az webapp log tail
and it displays the following output.
- Checking performance validation with
locust
- modify the host name in the
locustfile.py
with the url of the deployed app. - Ensure the key names in the json under the
predict
function is the same as those inapp.py
.
- modify the host name in the
- Run
locust -f locustfile.py --headless -u 20 -r 5 -t 20s
- The expected output is
-
Go to the Azure DevOps organization in the portal
-
Click on
My Azure DeOps Organizations
-
Sign in
-
Create a new project and give it a name, for e.g
ci_cd_project2
-
Make sure that
Git
is selected as the version control and create -
Create a new service connection under the
Pipelines
tab after clicking on project settings -
choose
Create service connection
-->Azure Resource Manager
-->Service principal (automatic) -->
-->subsription
-
Choose
Service connection name
-
login
-
Choose the same resource group where the app is deployed
-
Make sure the
ci_cd_prGrant access permission to all pipelines is checked
and save -
Go to
Pieplines
and create a new pipeline -
Choose
Github
-
Configure the pipeline as
Python to Linux Wepp App on Azure
-
Choose MSDN Platform Supsription and continue
-
Sign in to Github
-
Choose the Project Repo in Github
-
Validate and configure
-
Check the package version then save and run
-
The output is as follows.
Some of the ways the project can be improved in the future are:
- Ensure that all needed packages are specified in the
requirements.txt
file. - Provide starter codes for deep learming frameworks such as Pytorch and TensorFlow.