Deploying DataDog for a large scale infrastructure
- Geographic Regions
- Stages
- Applications
Matches the definitions of AWS Regions. It can be used for GCP or on-prem datacenter as well.
Different stages of application deployments, usually: dev, qa, prod.
A service that provides a distinct business functionality.
- having all monitors and dashboards in version control
- having all monitors templated
- being able to address smaller parts of the infrastructure
4 files represent the DataDog configuration for the whole infrastructure.
- infrastructure.yaml
It contains the logical grouping of applications into stages and regions. The relations are always N:M. 1 region can contain many stages and many applications in each stage.
- region.yaml
Defaults for a certain region (region).
- stage.yaml
Defaults for a certain stage (region, stage).
- application.yaml
Configuration that is specific for a certain application (region, stage, application).
I recently discovered Dhall that seems like the perfect fit to write the infrastructure in and than generate the YAML files.
The type safe definitions looks like the following:
let keyValue =
λ(k : Type)
→ λ(v : Type)
→ λ(mapKey : k)
→ λ(mapValue : v)
→ { mapKey = mapKey, mapValue = mapValue }
let ApplicationConfig : Type = { created_at : Text }
let Application = < etcd | postgresql | hadoop >
let Applications = Prelude.Map.Type Application ApplicationConfig
let application = keyValue Application ApplicationConfig
let Stage = < dev | qa | prod >
let Stages = Prelude.Map.Type Stage Applications
let stage = keyValue Stage Applications
let AwsRegion = < us-east-1 | eu-central-1 | eu-west-1 >
let AwsRegions = Prelude.Map.Type AwsRegion Stages
let awsRegion = keyValue AwsRegion Stages
After having these definitions we can create the infrastructure:
in [ awsRegion AwsRegion.us-east-1
[ stage Stage.dev
[ application Application.hadoop { created_at = "2019-11-04T09:00:00Z" }
, application Application.etcd { created_at = "2019-11-04T09:00:00Z" }
]
, stage Stage.qa
[ application Application.hadoop { created_at = "2019-11-04T09:00:00Z" }
, application Application.etcd { created_at = "2019-11-04T09:00:00Z" }
]
]
, awsRegion AwsRegion.eu-west-1
[ stage Stage.dev
[ application Application.hadoop { created_at = "2019-11-04T09:00:00Z" }
, application Application.etcd { created_at = "2019-11-04T09:00:00Z" }
]
]
, awsRegion AwsRegion.eu-central-1
[ stage Stage.dev
[ application Application.hadoop { created_at = "2019-11-04T09:00:00Z" }
, application Application.etcd { created_at = "2019-11-04T09:00:00Z" }
]
]
]
Generating the YAML:
dhall-to-yaml --file infrastructure.dhall > infrastructure.yaml
python3 gen.py
region: eu-central-1, stage: dev
region: eu-central-1, stage: dev, app: etcd
region: eu-central-1, stage: dev, app: hadoop
region: eu-west-1, stage: dev
region: eu-west-1, stage: dev, app: etcd
region: eu-west-1, stage: dev, app: hadoop
region: eu-west-1, stage: prod
region: eu-west-1, stage: prod, app: etcd
region: eu-west-1, stage: prod, app: hadoop
Templates folder has the monitor templates.
Example template:
---
name: High CPU load on application_name:{application_name} stage:{stage} {{{{host.name}}}} / {{{{host.ip}}}}
tags:
- application_name:{application_name}
- stage:{stage}
- region:{region}
type: metric alert
query: avg(last_5m):avg:system.load.norm.5{{application_name:{application_name},stage:{stage}}} by {{host}} > {critical_threshold}
message: >-2
High CPU load on application_name:{application_name} stage:{stage} {{{{host.name}}}} / {{{{host.ip}}}} for 5 consecutive minutes on this node.
Url: https://wd-global-prod.datadoghq.com/monitors/{monitor_id}
{slack_notification_channel}
monitor_options:
notify_audit: False
locked: False
timeout_h: 0
silenced: {{}}
include_tags: True
require_full_window: True
new_host_delay: 300
notify_no_data: False
renotify_interval: 0
escalation_message: >-2
CPU load is still damn high.
thresholds:
critical: {critical_threshold}
warning: {warning_threshold}
This gets rendered using Python format and converted to a dict that used to talk to the DataDog API.
Defaults are stage wide settings specifics are specific to a single application (in a region & stage).
For all of these above to work together nicely there is a dependency on tags being deployed every node, ELB, etc., so that we can reference those in monitors and dashboards.
I gave up on Conda and now just using venv from Python.
/usr/local/opt/python3/bin/python3 -m venv venv
. venv/bin/activate.fish #or the shell you are using
pip install --upgrade pip
pip install --upgrade toml pyyaml
Deploying a whole stage:
./data-cat/data-cat.py deploy-monitors -r eu-west-1 -s qa
Deploying a single application:
./data-cat/data-cat.py deploy-monitors -r eu-west-1 -s qa -a etcd
Deploying a whole stage:
./data-cat/data-cat.py deploy-dashboards -r eu-west-1 -s qa
Deploying a single application:
./data-cat/data-cat.py deploy-dashboards -r eu-west-1 -s qa -a etcd
Following Brendan Gregg's use method and the suggested things to monitor:
- CPUs: sockets, cores, hardware threads (virtual CPUs)
- Memory: capacity
- Network interfaces
- Storage devices: I/O, capacity
- Controllers: storage, network cards
- Interconnects: CPUs, memory, I/O
How to monitor it (examples):
- utilization: as a percent over a time interval. eg, "one disk is running at 90% utilization"
- saturation: as a queue length. eg, "the CPUs have an average run queue length of four"
- errors: scalar counts. eg, "this network interface has had fifty late collisions"
LambdaInsight is a consultancy located in Europe working on large scale infrastructures mostly in the intersection of data and cloud.
Let us know if you are interested in a project with us.