A distributed logging service, a largely-scalable, highly available, secure, and millisecond latency system to stream data. Streams data securely!

• Simple reason is that distributed systems are cool because of many reasons and some of them are large-scalable, high availability, secure, and fast.
• With this project, I attempt to go in-depth on how to build a system that grows in functionality as well as users and the team developing it.
• It's my attempt to broaden my knowledge and make it strong by developing this real-world end-to-end product.

• Simplicity
• Strongly typed and compiled
• Compiles to a single binary with no external dependencies
• Fast and lightweight
• Good coding practices
• Excellent support for network programming and concurrency
• Easy to deploy

• Kind tool to run a local Kubernetes cluster in Docker. (I am using: kind v0.14.0 go1.18.2 darwin/arm64)
• Go 1.13+

It’s not deployed anywhere right now.

• deploy - Helm charts to install this application with custom configurations.

• internal - Contains different packages together to build the distributed system.

• cmd - Various utilities and one-time commands live here

• api - Contains gRPC contracts and generated code.

• test - CFSSL for signing, verifying, and bundling TLS certificates.

Deploying your local Kubernetes cluster is very easy. Clone the repo and follow the below steps:

1. Create a Kubernetes cluster

kind create cluster --image=kindest/node:v1.21.2

2. Deploy to local

export PROJECT_ID=distributedlogging

• You can use the prebuilt docker image from gcr.io/$PROJECT_ID/distributedlogging

helm install distributedlogging distributedlogging --set image.repository=gcr.io/$PROJECT_ID/distributedlogging

• Or, you can build the docker image on your local

make build-local-docker docker push helm install distributedlogging --set image.repository=

3. After following the above steps, you would want to forward a pod or Service's pod to a port on your computer so you can request a service running inside Kubernetes without a load balancer:

kubectl port-forward pod/distributedlogging-0 8400 8400

Hurray, now you have deployed the application on your local cluster.

You can deploy the application to the Cloud (we use Google Kubernetes Engine (GKE)) by following the below steps:

1. To deploy to the Cloud, you should have a GKE cluster with version=1.21.* from the stable channel.
2. Connect to your GKE cluster from your local or from Cloud Shell.

gcloud container clusters get-credentials distributedlogging --zone us-central1-c --project distributedlogging

3. Install the Metacontroller chart

cd deploy kubectl create namespace metacontroller helm install metacontroller metacontroller

4. Deploy to the Cloud

export PROJECT_ID=distributedlogging

• You can use the prebuilt docker image from gcr.io/$PROJECT_ID/distributedlogging

helm install distributedlogging distributedlogging --set image.repository=gcr.io/$PROJECT_ID/distributedlogging --set service.lb=true

• Or, you can build the docker image on your local and push it to the gcr.io registry.

make build-docker docker tag :-linux-amd64 gcr.io/$PROJECT_ID/distributedlogging: docker push gcr.io/$PROJECT_ID/distributedlogging: Before running the below command, don’t forget to make your image at gcr.io/$PROJECT_ID/distributedlogging public helm install distributedlogging distributedlogging --set image.repository=gcr.io/$PROJECT_ID/distributedlogging --set service.lb=true

5. After following the above steps, you can get the ip of your External load balancer

kubectl get service -l app=service-per-pod -o go-template='{{range .items}}{{(index .status.loadBalancer.ingress 0).ip}}{{"\n"}}{{end}}'| head -n 1

6. Run following command to verify that our client connects to our service running in the cloud and that our service nodes discovered each other:

ADDR=$(kubectl get service -l app=service-per-pod -o go-template='{{range .items}}{{(index .status.loadBalancer.ingress 0).ip}}{{"\n"}}{{end}}'| head -n 1) go run cmd/getservers/main.go -addr=$ADDR:8400

That's it, you have successfully deployed the application on your Cloud Kubernetes cluster.

Feel free to create a PR, I’m more than happy to review and merge it.

• Onboarding videos and documentation
• Clean code, full test coverage and minimal tech debt

Feel free to create an issue, if you have any problem running this distributed system or any suggestions. I will be happy to help anytime. I have learned so many things from https://github.com/travisjeffery/proglog while developing this project and my skills are now the strongest ever. I hope you will find this repo same or more useful.