This repo contains implementation of a Url Shortner written in Go.

• What is goRubu?
• Why goRubu?
• BenchMarking
• Running Server
  • Docker
  • Local
• Api's
• Monitoring
• Contributing
• Future Todo
• Maintainer
• License

1. A Url Shortner written in Go, with MongoDb based backend.
2. Supports Caching for Hot urls, with Memcached, using a LRU based eviction strategy, and write through type of caching mechanism. Saw 200% decrease in Read Latency for URL redirection, after caching.
3. Used Travis CI for adding a CI/CD pipeline.
4. Dockerized the whole application. Used Docker compose for tying up different containers and multi-stage build for reducing the size of docker image by 900%.
5. Prometheus and Grafana based monitoring, to get an overall picture of the system and application metrics.
6. Handled Concurrent Requests, using a global Mutex, over Request Level. Use Optimistic Concurrency Control and Pessimistic Concurrency Control, to see if Performance can be increased.
7. Contains Api Validation and Logging Middlewares, along with Swagger based documentation

Wanted to Learn Go and system design, by building a project. Hence goRubu.

NOTE - Url Shortner is a read heavy system (read:write = 100:1), and these load tests are done on a single Machine (8Gb Ram, i5 Processor).

Check this out for more info.

1. For Url Redirection, with 1000 Concurrent Users, bombarding the app server for 2mins:

http_req_duration..........: avg=2.39s    min=0s     med=2.27s max=7.51s    p(90)=3.65s p(95)=4.16s
http_reqs..................: 49587   413.224692/s

2. For Url Shortening, for same specs

http_req_duration..........: avg=8.6s     min=0s      med=8.6s  max=19.03s   p(90)=10.33s p(95)=10.69s
http_reqs..................: 13345   111.207074/s

1. You need to have docker and docker-compose installed. After that just do

git clone https://github.com/rv404674/goRubu.git
cd goRubu
make docker

Note - I haven't dockerized prometheus and grafana with it. Containerized goRubu and locally installed prometheus and grafana will work fine as prometheus is only listening to "local:8080/metrics".

Check the Api's Section afterwards.

Ensure you have the following installed. Mongodb Memcached Make

On Macos, simply use thse

$ brew update
$ brew install mongodb/brew/mongodb-community
$ brew install memcached
$ brew install make

Note: After that check whether these have been started, else you will get connection error.

$ brew services list

if any service is not up do

$ brew service start service_name

Then we need to download the tar files for prometheus, grafana and node exporter. We will need to edit their config files, hence we are not using brew install for these. Also it becomes easy to run the server and do changes.

For Grafana

wget https://dl.grafana.com/oss/release/grafana-6.7.3.darwin-amd64.tar.gz
tar -zxvf grafana-6.7.3.darwin-amd64.tar.gz
mv grafana-6.7.3 /usr/local/bin
cd /usr/local/bin/grafana-6.7.3/bin
./grafana-server # this will run the grafana server

Note - By default prometheus server runs on 9090, and grafana on 3000.

Similarly download tar files for prometheus and node exporter and run there servers as well.

1. Go to the dir where prometheus is installed and change the prometheus default .yml file to this one new_yml, run the prometheus server, and node_exporter server.

2. do

$git clone https://github.com/rv404674/goRubu.git

3. cd in goRubu, and do

make deps

It will install all the go dependencies.

4. Then do

make install
make execute

Note: To see what these commands do check out this makefile

1. Hit localhost:8080/all/shorten_url with any url as key.

{
	"Url": "https://www.redditgifts.com/exchanges/manage"
}

The endpoint will return a shortened URL.

2. Hit http://localhost:8080/all/redirect with the shortened url to get the original URL back.

{
	"Url": "https://goRubu/MTAxMTA="
}

Working:

1. Prometheus follows a Pull based Mechanism instead of Push Based.
2. goRubu exposes an HTTP endpoint exposing monitoring metrics.
3. Prometheus then periodically download the metrics. For UI, prometheus is used as a data source for Grafana.

Grafana on Top of Prometheus

Peformance Improvements, bug fixes, better design approaches are welcome. Please Discuss your change by raising an issue, beforehand.

1. Use Promtheus and Grafana as Docker Containers instead of Local Installation.
2. Use Kubernetes instead of Docker Compose.
3. Increase Coverage to more than 90%.
4. Add Monitoring in Prometheus for Number of 2xx, 3xx and 4xx.

Rahul Verma Linkedin Email

MIT © Rahul Verma