ELK + Prometheus + Grafana communication POC

This is POC to understand better how this applications communicate with each other:

• Grafana
• Prometheus
• Spring Boot
  • Actuator
  • Sleuth
  • Logback
• Elasticsearch
• Logstash
• Kibana
• Zipkin
• Graylog

First of all, increase linux virtual allowed space

$ sudo sysctl -w vm.max_map_count=262144

• Open file prometheus.yml, on the last line change the spring-actuator target IP to your local IP (this will be fixed soon :) )
• Start the Spring Boot demo app with cd demo && ./gradlew bootRun
• Start docker-compose with docker-compose -d up)
  • Be patient, wait at least 5 minutes so all the services have plenty of time to comunicate with each other and setup the initial configuration. Grab a cup of coffee or tea or water or something more powerful :)
  • docker ps should list 8 services up
• Check if prometheus target 'spring-actuator' is up
• Open Graylog with user admin, password admin
  • Go to System -> Inputs
  • Select input "GELF UDP" and Launch, fill it with:
    • Check the ckeckbox Global
    • Title: UDP
    • Bind address: 0.0.0.0
    • Port: 12201
    • Save
  • Again select input "GELF UDP" and Launch, fill it with:
    • DONT check the ckeckbox Global
    • Title: ELK Stack UDP
    • Bind address: 0.0.0.0
    • Port: 12202
    • Save
• Call the Spring Boot demo app at least once to generate a log curl http://localhost:8080/info
• Create the index micro-* in Kibana with timefield @timestamp
• Open Grafana dashboard with user admin, password secret
• Go to Configuration->Datasources, "Add data source", choose ElasticSearch and input:
  • URL: http://elasticsearch:9200
  • Index name: micro-*
  • Time field name: @timestamp
  • Click "Save & Test"
• Again go to Configuration->Datasources, "Add data source", choose Prometheus and input:
  • URL: http://prometheus:9090
  • Click "Save & Test"
• Import the Grafana dashboard file grafana-dash.json
• Keep hitting http://localhost:8080/info, http://localhost:8080/warn and http://localhost:8080/error to check logs in Kibana/Graylog and numbers in Grafana. Also checkout Zipkin for the tracing.

Done!

MIT

Free Software, Hell Yeah!