Git Product home page Git Product logo

distributed-logging's Introduction

Distributed Logging

A distributed logging service, a largely-scalable, highly available, secure, and millisecond latency system to stream data. Streams data securely!

Why build this? What's new here?

  • Simple reason is that distributed systems are cool because of many reasons and some of them are large-scalable, high availability, secure, and fast.
  • With this project, I attempt to go in-depth on how to build a system that grows in functionality as well as users and the team developing it.
  • It's my attempt to broaden my knowledge and make it strong by developing this real-world end-to-end product.

Why choose Go?

  • Simplicity
  • Strongly typed and compiled
  • Compiles to a single binary with no external dependencies
  • Fast and lightweight
  • Good coding practices
  • Excellent support for network programming and concurrency
  • Easy to deploy

Prerequisites

  • Kind tool to run a local Kubernetes cluster in Docker. (I am using: kind v0.14.0 go1.18.2 darwin/arm64)
  • Go 1.13+

Where is it deployed?

It’s not deployed anywhere right now.

Directory Structure

  • deploy - Helm charts to install this application with custom configurations.

  • internal - Contains different packages together to build the distributed system.

  • cmd - Various utilities and one-time commands live here

  • api - Contains gRPC contracts and generated code.

  • test - CFSSL for signing, verifying, and bundling TLS certificates.

How can I run this locally?

Deploying your local Kubernetes cluster is very easy. Clone the repo and follow the below steps:

  1. Create a Kubernetes cluster

kind create cluster --image=kindest/node:v1.21.2

  1. Deploy to local

export PROJECT_ID=distributedlogging

  • You can use the prebuilt docker image from gcr.io/$PROJECT_ID/distributedlogging

helm install distributedlogging distributedlogging --set image.repository=gcr.io/$PROJECT_ID/distributedlogging

  • Or, you can build the docker image on your local

make build-local-docker docker push helm install distributedlogging --set image.repository=

  1. After following the above steps, you would want to forward a pod or Service's pod to a port on your computer so you can request a service running inside Kubernetes without a load balancer:

kubectl port-forward pod/distributedlogging-0 8400 8400

Hurray, now you have deployed the application on your local cluster.

How can I run this on the Cloud?

You can deploy the application to the Cloud (we use Google Kubernetes Engine (GKE)) by following the below steps:

  1. To deploy to the Cloud, you should have a GKE cluster with version=1.21.* from the stable channel.
  2. Connect to your GKE cluster from your local or from Cloud Shell.

gcloud container clusters get-credentials distributedlogging --zone us-central1-c --project distributedlogging

  1. Install the Metacontroller chart

cd deploy kubectl create namespace metacontroller helm install metacontroller metacontroller

  1. Deploy to the Cloud

export PROJECT_ID=distributedlogging

  • You can use the prebuilt docker image from gcr.io/$PROJECT_ID/distributedlogging

helm install distributedlogging distributedlogging --set image.repository=gcr.io/$PROJECT_ID/distributedlogging --set service.lb=true

  • Or, you can build the docker image on your local and push it to the gcr.io registry.

make build-docker docker tag :-linux-amd64 gcr.io/$PROJECT_ID/distributedlogging: docker push gcr.io/$PROJECT_ID/distributedlogging: Before running the below command, don’t forget to make your image at gcr.io/$PROJECT_ID/distributedlogging public helm install distributedlogging distributedlogging --set image.repository=gcr.io/$PROJECT_ID/distributedlogging --set service.lb=true

  1. After following the above steps, you can get the ip of your External load balancer

kubectl get service -l app=service-per-pod -o go-template='{{range .items}}{{(index .status.loadBalancer.ingress 0).ip}}{{"\n"}}{{end}}'| head -n 1

  1. Run following command to verify that our client connects to our service running in the cloud and that our service nodes discovered each other:

ADDR=$(kubectl get service -l app=service-per-pod -o go-template='{{range .items}}{{(index .status.loadBalancer.ingress 0).ip}}{{"\n"}}{{end}}'| head -n 1) go run cmd/getservers/main.go -addr=$ADDR:8400

That's it, you have successfully deployed the application on your Cloud Kubernetes cluster.

Can I contribute to this project?

Feel free to create a PR, I’m more than happy to review and merge it.

What's the long-term goal?

  • Onboarding videos and documentation
  • Clean code, full test coverage and minimal tech debt

Thank you!

Feel free to create an issue, if you have any problem running this distributed system or any suggestions. I will be happy to help anytime. I have learned so many things from https://github.com/travisjeffery/proglog while developing this project and my skills are now the strongest ever. I hope you will find this repo same or more useful.

distributed-logging's People

Contributors

brijeshlakkad avatar

Stargazers

 avatar

Watchers

 avatar  avatar

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.