Find me here:

This is the Regular Edition. It's the most basic version of Automate All The Things. For more complex features (logging, monitoring, service mesh, etc.) check out the other versions.

1. Regular Edition
2. Hardcore Edition
3. Insane Edition
4. Braindamage Edition
5. Transcendence Edition
6. Nirvana Edition

• Introduction
  • Prerequisites
  • What we'll be doing
  • Tools we'll be using
  • Disclaimer
• Local Setup
• Azure DevOps Setup
  • Create project
  • Install required plugins
  • Get your AWS keys
  • Create AWS service connection
  • Create DockerHub service connection
  • Create AWS-keys variable group
  • Create an Azure self-hosted agent
• AWS Infrastructure Deployment Pipeline
  • Description
  • Instructions
• ArgoCD Deployment Pipeline
  • Description
  • Instructions
• Backend Service Build & Deploy Pipeline
  • Description
  • Instructions
• Frontend Service Build & Deploy Pipeline
  • Description
  • Instructions
• Destroy All The Things Pipeline
  • Description
  • Instructions
• Conclusion
  • On the next edition

I believe in a world where all that's expected of me is to enjoy life, lay on the couch, play COD and have existential crises.

I wish I could automate cooking, cleaning, working, doing taxes, making friends, dating and even writing stupid READMEs.

But technology hasn't quite caught up to my level of laziness yet, so I've taken some inspiration from Thanos and said "Fine... I'll do it myself".

Here's my attempt at making the world a better place. People in the future will look back at heroes like me and enjoy their time playing video games and fighting the war against AI, in peace.

• Git installed
• Python3 installed
• Active GitHub account
• Active DockerHub account
• Active AWS account
• Active Azure DevOps account

The purpose of this repo is not to give you an in depth explanation of the tools we'll be using, but to demonstrate how they can interact with each other to make the deployment of a whole infrastructure (with an application) as efficient and streamlined as possible.

I want to show how IaC (Infrastructure as Code), Gitops and CI/CD (Continuous Integration/Continuous Deployment) can be merged for unlimited power.

As you can see in the diagram, we'll be deploying an EKS Kubernetes cluster in AWS. Inside the cluster we'll have three environments where our app will be deployed. The app is made up of two microservices: frontend and backend. Each frontend will be accesible to the public internet through a Load Balancer.

Along with the cluster, we'll deploy an ElastiCache (Redis) database for each environment and one EC2 instance for running configuration tasks on the databases. We'll also create an S3 bucket which will store our terraform state file.

Our app is a very simple static website, but I'm not spoiling it for you. You'll have to deploy it to see it.

• Code Versioning -> Git
• Source Code Management -> GitHub
• Cloud Infrastructure -> Amazon Web Services
• Infrastructure as Code -> Terraform
• Containerization -> Docker
• Container Orchestration -> Kubernetes
• Continuous Integration -> Azure DevOps
• Continuous Deployment -> Helm & ArgoCD
• Scripting -> Python
  

This is not a free project, it will cost you between $1 US dollars and $10 depending on how long you run the resources for. That's assuming you run them for a few hours tops, not days. Always remember to run the destroy-all-the-things pipeline when you are done.

Some things could have been further automated but I prioritized modularization and separation of concerns.

For example, the EKS cluster could have been deployed with ArgoCD installed in one pipeline, but I wanted to have them separated so that each module is focused on its specific task, making each of them more recyclable.

Also, please do submit an issue if you find any errors or you have any good ideas on how to improve this, I would love to hear them.

Let's begin...

In order to turn this whole deployment into your own thing, we need to do some initial setup:

1. Fork this repo. Keep the repository name "automate-all-the-things".
2. Clone the repo from your fork:

git clone https://github.com/<your-github-username>/automate-all-the-things.git

2. Move into the directory:

cd automate-all-the-things

2. Run the initial setup script. Come back when you are done:

python3 python/initial-setup.py

4. Hope you enjoyed the welcome script! Now push your customized repo to GitHub:

git add .
git commit -m "customized repo"
git push

5. Awesome! You can now proceed with the Azure DevOps setup.

Before creating our pipelines we need to get a few things set up:

1. Sign in Azure DevOps.
2. Go to "New project" on the top-right.
3. Write the name for your project and under "Visibility" select "Private".
4. Click "Create".

These plugins are required for the pipelines we'll be creating. Click on "Get it free", select your organization and then click "Install". https://marketplace.visualstudio.com/items?itemName=JasonBJohnson.azure-pipelines-tasks-terraform

1. Install Terraform Tasks plugin for Azure Pipelines
2. Install AWS Toolkit plugin for Azure Pipelines

These will be required for Azure DevOps to connect to your AWS account.

1. Open the IAM console at https://console.aws.amazon.com/iam/.
2. On the search bar look up "IAM".
3. On the IAM dashboard, select "Users" on the left side menu. If you are root user and haven't created any users, you'll find the "Create access key" option on IAM > My security credentials. You should know that creating Access Keys for the root user is a bad security practice. If you choose to proceed anyway, click on "Create access key" and skip to point 6.
4. Choose your IAM user name (not the check box).
5. Open the Security credentials tab, and then choose "Create access key".
6. To see the new access key, choose Show. Your credentials resemble the following:

• Access key ID: AKIAIOSFODNN7EXAMPLE
  
• Secret access key: wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY

7. Copy and save these somewhere safe.

This service connection is required for our Azure DevOps pipelines to interact with AWS.

1. Go back to Azure DevOps and open your project.
2. Go to Project settings on the left side menu (bottom-left corner).
3. On the left side menu, under "Pipelines", select "Service connections".
4. Click on "Create service connection".
5. Select AWS, click "Next".
6. Paste your Access Key ID and Secret Access Key.
7. Under "Service connection name", write "aws".
8. Select the "Grant access permission to all pipelines" option.
9. Save.

This service connection is required for our Azure DevOps pipelines to be able to push images to your DockerHub registry.

1. While on the "Service connections" screen, click on "New service connection" on the top-right.
2. Select "Docker Registry", click "Next".
3. Under "Registry type" select "Docker Hub".
4. Input your Docker ID and Password.
5. Under "Service connection name", write "dockerhub".
6. Select the "Grant access permission to all pipelines" option.
7. Click on "Verify and save".

These are needed for Terraform to be able to deploy our AWS infrastructure.

1. On the left side menu under "Pipelines" go to "Library"
2. Click on "+ Variable group".
3. Under "Variable group name" write "aws-keys".
4. Add the following variables pasting on the "Value" field your AWS keys:

• aws_access_key_id
• aws_secret_access_key

5. Click on the lock icon on each variable.
6. Click on "Save".
7. Click on "Pipeline permissions" and give it "Open access". This means all our pipelines will be able to use these variables.

If you have a hosted parallelism, you can skip this step.

A hosted parallelism basically means that Azure will spin up a server in which to run your pipelines. You can purchase one or you can request a free parallelism by filling out this form.

If you don't have a hosted parallelism, you will have to run the pipeline in a self-hosted agent. This means you'll install an Azure DevOps Agent on your local machine, which will receive and execute the pipeline jobs.

To install a self-hosted agent on your machine, you can follow the official documentation here.

Our first pipeline, the one that will provide us with all the necessary infrastructure.

What does this pipeline do? If you take a look at the 00-deploy-infra.yml file, you'll see that the first thing we do is use the Terraform plugin we previously installed to deploy a S3 Bucket and DynamoDB table. These two resources will allow us to store our terraform state remotely and give it locking functionality.

Why do we need to store our tf state remotely and locking it? Well, this is probably not necessary for this exercise but it's a best practice when working on a team.
Storing it remotely means that everyone on the team can access and work with the same state file, and locking it means that only one person can access it at a time, this prevents state conflicts.

Before we proceed with deploying our actual infrastructure, the pipeline will move the state file to the terraform/aws/ directory, so our backend resources (the Bucket and DynamoDB Table) will also be tracked as part of our whole infrastructure. If you want to understand how this works, I suggest you watch this video where Sid from DevOps Directive explains it better than I ever could.

Now that the backend is set, we will deploy our actual infrastructure!

So, what is our infra? Well, the main parts are the networking resources, the ElastiCache databases, the EC2 instance and the EKS cluster, along with the Cluster Autoscaler and an AWS Load Balancer Controller which will act as our Kubernetes Ingress Controller.

Having this AWS Load Balancer Controller means that for every Ingress resource we create in our cluster, an AWS Application Load Balancer will be automatically created. This is the native way to do it in EKS and it has a lot to benefits, but it creates an issue for us.
We want to track everything in our infra as IaC, but these automatically created Application Load Balancers won't be tracked in our Terraform... No worries, we'll take care of this issue in the Destroy All The Things Pipeline.
For more info on the AWS Load Balancer Controller you can watch this excellent video by Anton Putra.

If you want to know exactly what is being deployed, you can check out the terraform files. Here you can modify the resources to be deployed to AWS. Let's say you want to add a second EC2 Instance, you can add the following block in the ec2.tf file:

resource "aws_instance" "ec2_instance" {
    ami = "ami-01107263728f3bef4"
    subnet_id = aws_subnet.public-subnet-a.id
    instance_type = "t2.micro"
}

Commit the changes and run the pipeline again. The backend deployment step will fail, so the pipeline will finish with a warning, you can ignore it.

The pipeline will also modify the /helm/my-app/backend/environments files on the repo. It will get the endpoints for each ElastiCache DB from terraform outputs and include them in the values of each environment.

Oh and lastly... it will export an artifact with the instructions on how to connect to the EC2 instance.

1. On your Azure DevOps project, go to "Pipelines" on the left side menu.
2. Select "Pipelines" under "Pipelines" on the left side menu.
3. Click on "Create Pipeline".
4. Select "Github".
5. You might get a screen to authorize Azure Pipelines to access your GitHub account, if so, go ahead and click the green button.
6. Select the repo, it should be "your-github-username/automate-all-the-things"
7. You might also get a screen to install the Azure Pipelines App on your GitHub account, if so, go ahead and click the green button and follow the instructions.
8. Select "Existing Azure Pipelines YAML file".
9. Under "Branch" select "main" and under "Path" select "/azure-devops/00-deploy-infra.yml". Click "Continue".
10. If you have hosted parallelism skip to point 11. If you DON'T have a hosted parallelism, you need to tell Azure DevOps to use your self-hosted agent. In order to do this, you'll need to go to the repo and modify the 00-deploy-infra.yml file.
    Under "pool" you need to edit it so that it looks like this:

pool:
  # vmImage: 'ubuntu-latest'
  name: <agent-pool-name> # Insert here the name of the agent pool you created
  demands:
    - agent.name -equals <agent-name> # Insert here the name of the agent you created

11. Click on "Run".
12. When it's done, the EC2 instance public IP address will be exported as an artifact. You'll find it in the pipeline run screen. Download it to see the instructions to access the instance.

We won't go into what ArgoCD is, for that you have this video by the #1 DevOps youtuber, Nana from TechWorld with Nana.

This pipeline will use the ArgoCD Helm Chart in our repo to deploy ArgoCD into our EKS.
The first thing it will do is run the necessary tasks to connect to our the cluster. After this, ArgoCD will be installed, along with its Ingress.

It will get the access info (URL, user and pass) and export them as an artifact. As I explained before, the Ingress will automatically create an AWS Application Load Balancer. This LB takes a few moments to become active, so you might have to wait a little bit for the URL to be ready, just hit refresh on the browser every few seconds.

Finally, it will create the ArgoCD application resources for our app, which will be watching the /helm/my-app/backend and /helm/my-app/frontend directories in our repo, and automatically create all the resources it finds and apply any future changes me make there. The /helm/my-app directory simulates what would be our K8S infrastructure repository.

If you want to know more about Helm, here's another Nana video.

1. Go to "Pipelines" under "Pipelines" on the left side menu.
2. Click on "New pipeline".
3. Select "GitHub".
4. Select the repo, it should be "your-github-username/automate-all-the-things"
5. Select "Existing Azure Pipelines YAML file".
6. Under "Branch" select "main" and under "Path" select "/azure-devops/01-deploy-argocd.yml". Click "Continue".
7. If you DON'T have a hosted parallelism, you'll need to do the same thing as in point 10 from the infrastructure deployment pipeline.
8. Click on "Run".
9. When it's done, the access file will be exported as an artifact. You'll find it in the pipeline run screen. Download it to see the URL and credentials.

10. You can now access the ArgoCD UI, if it's not ready just hit refresh every few seconds. Here you should find six applications running but in a "Progressing/Degraded" state. This is because we haven't built our app and pushed it to DockerHub yet. Let's take care of that next.

Time to actually start the deployment of our app.

Our app is made of two microservices (backend and frontend) and a database. Let's start with the backend.

The /my-app/backend directory on the repo is meant to represent the backend microservice application code repository. Here you'll find the code files and the corresponding Dockerfile for the backend service.

There's four stages on this pipeline:

On the Build stage we will use Docker to build a container image from the Dockerfile, tag it with the number of the pipeline run and push it to your DockerHub registry.

On the Deploy Dev stage, we will checkout the repo and modify the helm/my-app/backend/environments/values-dev.yaml file and push the change to GitHub. But why?
Remember how we just pushed the image to DockerHub with the new tag? And remember how ArgoCD is watching the helm/my-app directory? Well, this is how we tell ArgoCD that a new version of the backend microservice is available and should be deployed. We modify the image.tag value in the values-dev.yaml file and wait for ArgoCD to apply the changes.

This is how gentlemen manage their K8S resources. We are not some cavemen creating and deleting stuff manually with kubectl. We manage our infrastructure with GitOps.

After the Deploy Dev stage is done, and only if it was successful, the Deploy Stage stage will commence. It will do the same thing as the previous stage, but this time modifying the helm/my-app/backend/environments/values-stage.yaml file.

We'll repeat the same process for Prod, but since Prod should be a more delicate environment, the Deploy Prod stage will require authorization from the top level excecutives (in this case it's you, congrats boss) to be executed. You'll receive an email with a link asking you to verify and approve the deployment to Prod. Go ahead and approve it.

That's it! Your app was deployed to all environments! Good job buddy!

This pipeline is automatically triggered everytime there are any changes commited inside the "backend service application code repository" (meaning the /my-app/backend directory). In this manner, if the backend developers commit any changes to the backend service, they will be automatically built and deployed to the cluster. That's some delicious CI/CD for you baby.

Now, if the infrastrucure team needs to make changes to the cluster resources, they would work on the "K8S infrastructure repository" (meaning the /helm/my-app directory). Let's say they need to increase the number of pod replicas for the backend service in the prod environment, then they'd change the value of deployment.replicas in the helm/my-app/backend/environments/values-prod.yaml file, commit the change and wait for ArgoCD to apply the changes on the cluster. There's some tasty Gitops for you too.

1. Go to "Pipelines" under "Pipelines" on the left side menu.
2. Click on "New pipeline".
3. Select "GitHub".
4. Select the repo, it should be "your-github-username/automate-all-the-things"
5. Select "Existing Azure Pipelines YAML file".
6. Under "Branch" select "main" and under "Path" select "/azure-devops/02-build-and-deploy-backend.yml". Click "Continue".
7. If you DON'T have a hosted parallelism, you'll need to do the same thing as in point 10 from the infrastructure deployment pipeline.
8. Click on "Run".

We are almost there! In this pipeline we will build and deploy our frontend.

The /my-app/frontend directory on the repo is meant to represent the frontend microservice code repository. Here you'll find the code files and the corresponding Dockerfile for the frontend service.

Just as in the backend pipeline, there's four stages on this pipeline:

1. We build and push our frontend image to DockerHub.
2. We deploy to Dev environment in the same manner we did with the backend (modifying the image.tag value in the helm/my-app/frontend/environments/values-dev.yaml file).
3. If deployment to Dev was OK, we deploy to Stage.
4. If deployment to Stage was OK, we'll get an email requesting authorization for deployment to Prod. Give it the thumbs up and our app will be deployed to Prod.

That's it! Your entire app was deployed to all environments! Good job buddy!

Just as with the backend, this pipeline is automatically triggered everytime there are any changes commited inside the "frontend service application code repository" (meaning the /my-app/frontend directory). In this manner, if the frontend developers commit any changes to the frontend service, they will be automatically built and deployed to the cluster.

For the infrastructure, same as before. If the infrastrucure team needs to, for example, change the number of pod replicas for the frontend service in the dev environment, then they'd change the value of deployment.replicas in the helm/my-app/frontend/environments/values-dev.yaml file, commit the change and wait for ArgoCD to apply the changes on the cluster.

1. Go to "Pipelines" under "Pipelines" on the left side menu.
2. Click on "New pipeline".
3. Select "GitHub".
4. Select the repo, it should be "your-github-username/automate-all-the-things"
5. Select "Existing Azure Pipelines YAML file".
6. Under "Branch" select "main" and under "Path" select "/azure-devops/03-build-and-deploy-frontend.yml". Click "Continue".
7. If you DON'T have a hosted parallelism, you'll need to do the same thing as in point 10 from the infrastructure deployment pipeline.
8. Click on "Run".
9. Each deployment stage will export an artifact file with the URL of each environment. You'll find them in the pipeline run screen. Download them to see the URLs.

10. If you go to the URLs too quickly you will get a "503 Service Temporarily Unavailable". We need to give ArgoCD a little time to notice the changes in the /helm/my-app/frontend directory. By default ArgoCD pulls for changes every three minutes. You can either wait like an adult or go into the ArgoCD web UI and hit "Refresh Apps" like the impatient child that you are.
11. Check the URLs again.
12. On the top left of the website you'll see the "Visit count". This number is being stored in the ElatiCache DB and accessed through the backend.
13. That's it! Hope you like the web I made for you. If you did, go leave a star on my repo.

Let's burn it all to the ground.

Remember how the AWS Load Balancer Controller created this problem for us where some Applications Load Balancers were created automatically in AWS but were not tracked by our Terraform? Well, in this pipeline, the first thing we need to do it take care of this.

The pipeline will first eliminate ArgoCD from our cluster and then delete all Ingress resources. This will automatically get rid of any Application Load Balancers in AWS.

After this, the pipeline will be able to run "terraform destroy" with no issues. Our infra will be obliterated and we won't be giving any more of our precious money to Bezos.

The pipeline will finish with a warning, worry not, this is because the "terraform destroy" command will have also deleted our terraform backend (the Bucket and DyamoDB Table), so Terraform won't be able to push the updated state back there. We can ignore this warning. I wish there was a more elegant way of finishing the project but I couldn't find any so deal with it.

1. Go to "Pipelines" under "Pipelines" on the left side menu.
2. Click on "New pipeline".
3. Select "GitHub".
4. Select the repo, it should be "your-github-username/automate-all-the-things"
5. Select "Existing Azure Pipelines YAML file".
6. Under "Branch" select "main" and under "Path" select "/azure-devops/04-destroy-all-the-things.yml". Click "Continue".
7. If you DON'T have a hosted parallelism, you'll need to do the same thing as in point 10 from the infrastructure deployment pipeline.
8. Click on "Run".
9. There's two AWS resources that for some reason don't get destroyed: a DHCP Option Set and an Auto Scaling Managed Rule. I'm pretty sure these don't generate any expenses but you can go and delete them manually just in case. I'm really sorry about this... I have brought shame upon my family...

Our journey comes to an end... Congratulations! You made it!

I hope this proved useful (and fun), that you learned something and that you can take some pieces of this to use in your own projects.

You now possess the power of of CI/CD, GitOps, and Infrastructure as Code. You know what this means, use it carefully.

Special thanks to all these wonderful YouTube people. This wouldn't have been possible without them:

• Nana Janashia from Techworld With Nana
• Viktor Farcic from DevOps Toolkit
• Marcer Dempers from That DevOps Guy
• Sid Palas from DevOps Directive
• Mumshad Mannambeth and his guys from KodeKloud
• Anton Putra

Automate All The Things Hardcore Edition:

• We'll make ArgoCD self-managed through GitOps
• We'll implement the ArgoCD App of Apps pattern
• We'll implement observability tools:
  • Monitoring with Prometheus
  • Logging with Loki
  • Visualization with Grafana
• We'll start using AWS Elastic Block Store which is required to manage persistent volumes
• We'll make the backend service log every time it receives a request
• We'll make the ElastiCache DBs password protected