Git Product home page Git Product logo

observability-nomad's Introduction

Adding Observability to Nomad Applications

Recording of the demo is available on the hashitalk 2021 website. (slides)

This repository demonstrates how you can leverage the Grafana Open Source Observability Stack with Nomad workload.

In this demonstration we will deploy an application (TNS) on Nomad along with the Grafana Stack. The TNS application is written in Go and instrumented with:

You can use the instrumentation of your choice such as: OpenTelemetry, Zipkin, json logs...

We'll also deploy backends to store collected signals:

  • Prometheus will scrape Metrics using the scrape endpoint.
  • Loki will receive Logs collected by Promtail.
  • Tempo will directly receives Traces and Spans.

Finally, we'll deploy Grafana and provision it with all our backend datasources and a dashboard to start with.

Getting Started

For simplicity you'll need to install and configure vagrant.

To get started simply run:

vagrant up

In case you want a faster startup not based on Ubuntu but on Flatcar Linux (as CoreOS has been EOLed):

VAGRANT_VAGRANTFILE=Vagrantfile.flatcar vagrant up

IMPORTANT NOTE: Due to the new policies of Docker Hub image pulling, (see https://blog.container-solutions.com/dealing-with-docker-hub-rate-limiting) there may be cases where you will need to docker login to avoid getting error messages like:

Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit

In order to use DockerHub login, you need to provide two additional environment variables as follows:

DOCKERHUBPASSWD=my-dockerhub-password DOCKERHUBID=my-dockerhub-login vagrant up

Then you should be able to access:

You can go to the Nomad UI Jobs page to see all running jobs.

alt text

Nomad Client Configuration

Promtail need to access host logs folder. (alloc/{task_id}/logs) By default the docker driver in nomad doesn't allow mounting volumes. In this example we have enabled it using the plugin stanza:

  plugin "docker" {
    config {
      volumes {
        enabled      = true
      }
    }
  }

However you can also simply run Promtail binary on the host manually too or use nomad host_volume feature.

Promtail also needs to save tail positions in a file, you should make sure this file is always the same between restart. Again in this example we're using a host path mounted in the container to persist this file,

Troubleshooting

Grafana shows nothing or TNS keeps crashing because of it can't connect to Tempo

  • You may have troubles with your dns configuration in the jobs, if your jobs can't talks to each other tries to change the ip to 127.0.0.1 or the internal ip address of your server if using a VPC or just removes the dns stanza. It's recommanded to use Consul Connect to connect every services to each others.

I can't see the logs in Grafana/Loki

  • You may have a different data_dir config in your nomad configuration. Here it's using /opt/nomad/data while we generally sets /opt/nomad. If it's your case, change the volume stanza of your tempo job.

observability-nomad's People

Contributors

cyriltovena avatar jcalonso avatar obourdon avatar scorsi avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar

observability-nomad's Issues

demo image tags appear to be missing from DockerHub

It looks like the :demo image tags are missing from DockerHub now. For example:

docker run --rm -it prom/prometheus:demo 
Unable to find image 'prom/prometheus:demo' locally
docker: Error response from daemon: manifest for prom/prometheus:demo not found: manifest unknown: manifest unknown.
See 'docker run --help'.

These images are referenced in the following nomad jobs:

It seems like these work if you remove the :demo tag. (I did not test tns in my setup though, so it's possible that might still be working.)

Thanks for this repo and the presentation, I learned a lot!

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.