A high throughput, horizontally scalable URL shortener solution using Go, Redis, MongoDB, Terraform, and Kubernetes on EKS.

• Golang ✔️
• MongoDB ✔️
• Redis ✔️
• Terraform ✔️
• Kubernetes on EKS ✔️

This section aims to go through the technical decisions behind the solution architecture.

First of all, an URL Shortener should handle a lot of URL redirections concurrently, avoiding downtime as much as possible, in order to keep those short URLs working. It's a read-heavy system and has a specific room for optimization: hot URLs. Some URLs will be fetched much more frequently than others, so it makes sense to keep them in some kind of cache.

To achieve the requirements, the system stores hot URLs in an in-memory database: a Redis instance. When an URL is created, it is cached with an expiration lifespan. If the URL is fetched before this span ends, the expiration span is refreshed, so that a commonly used URL keeps having a fast response time. When the expiration span is reached, the URL will be only available at the main data storage, in this case, a MongoDB instance. Because there's no need for strong consistency or relational dependency, a NoSQL database is a good choice to store users' and URLs' information. Golang is a very good language to handle concurrency, so it's great for building fast servers.

To handle millions of users, the application needs to scale horizontally. By running everything as containers inside a kubernetes cluster, we can easily start up new instances for the application backend API or MongoDB. Although in this example we don't use shards, MongoDB supports sharding, which is a strong solution if our data grows really large.

The production infrastructure was built as code using terraform. To reproduce this environment yourself, there are a few requirements needed:

• An AWS account, with programatic access.
• Terraform CLI installed (For this project I've used version 0.12.24).
• Kubernetes CLI installed.

After everything is set up properly, go through the following steps:

Enter the terraform source code directory

cd terraform

Initialize the working directory and download all dependecies with

terraform init

Generate an execution plan

terraform plan -out=plan.apply

Apply

terraform apply plan.apply

Generally, EKS cluster takes about 15 minutes to create. When is done, a new file named kubeconfig_url-shortener-production-cluster should be seen in the current directory. This file will be used to access the brand-new Kubernetes cluster.

To get access to the EKS cluster

export KUBECONFIG="${PWD}/kubeconfig_url-shortener-production-cluster"

Now let's start up our pods and get everything running

kubectl apply -f ./kubernetes 

When it's complete, three newly created pods should be running: A Redis instance for cashing hot URLs, a MongoDB as the main data storage and the URL Shortener API

kubectl get pods 

On the EKS interface, you can also see if everything went correctly by entering the url shortener cluster

In order to test it out, get the load balance endpoint by

kubectl describe service url-shortener

That's it! Now you can start generating short URLs by calling the API endpoints. 🚀

For local development, you can start everything up using docker-compose by running

make all

You can also run the application locally with kubernetes using minikube.

First of all, start the minikube cluster

minikube start

Apply everything by running

kubectl apply -f kubernetes

You can watch all pods coming alive with

kubectl get pods -w

To find the application URL exposed in our cluster by the service, type

minikube service url-shortener

When you're done, delete the application with

kubectl delete -f kubernetes

And also delete the minikube cluster

minikube delete --all

curl -XPOST http://localhost:3000/v1/users --header "Content-Type: application/json"  --data '{
    "name": "Jack",
    "email": "[email protected]"
}'

{
  "id": "5fd6ac5c6884b412d6ec1475",
  "name": "Jack",
  "email": "[email protected]",
  "updated_at": 1607904348,
  "created_at": 1607904348
}

curl http://localhost:3000/v1/users/1

{
  "id": "5fd6ac5c6884b412d6ec1475",
  "name": "Jack",
  "email": "[email protected]",
  "updated_at": 1607904348,
  "created_at": 1607904348
}

curl -XPOST http://localhost:3000/v1/short_urls --header "Content-Type: application/json"  --data '{
    "original_url": "https://really-long-website-url.com",
    "user_id": "5fd6ac5c6884b412d6ec1475",
    "expires_at": 100
}'

{
  "hash": "DWv53EaGg",
  "original_url": "https://really-long-website-url.com",
  "user_id": "5fd6ac5c6884b412d6ec1475",
  "expires_at": 100,
  "created_at": 1607904409
}

curl http://localhost:3000/v1/short_urls/DWv53EaGg

{ "original_url": "https://really-long-website-url.com" }

curl http://localhost:3000/v1/short_urls/DWv53EaGg/redirect

It redirects to the original URL