Git Product home page Git Product logo

datadog-formula's Introduction

Datadog Formula

The Datadog SaltStack formula is used to install the Datadog Agent and the Agent-based integrations (checks). For more details on SaltStack formulas, see the Salt formulas installation and usage instructions.

Setup

Requirements

The Datadog SaltStack formula only supports installs on Debian-based and RedHat-based systems.

Installation

The following instructions add the Datadog formula to the base Salt environment. To add it to another Salt environment, change the base references to the name of your Salt environment.

Option 1

Install the Datadog formula in the base environment of your Salt master node, using the gitfs_remotes option in your Salt master configuration file (defaults to /etc/salt/master):

fileserver_backend:
  - roots # Active by default, necessary to be able to use the local salt files we define in the next steps
  - gitfs # Adds gitfs as a fileserver backend to be able to use gitfs_remotes

gitfs_remotes:
  - https://github.com/DataDog/datadog-formula.git:
    - saltenv:
      - base:
        - ref: 3.0 # Pin the version of the formula you want to use

Then restart your Salt Master service to apply the configuration changes:

systemctl restart salt-master
# OR
service salt-master restart

Option 2

Alternatively, clone the Datadog formula on your Salt master node:

mkdir -p /srv/formulas && cd /srv/formulas
git clone https://github.com/DataDog/datadog-formula.git

Then, add it to the base environment under file_roots of your Salt master configuration file (defaults to /etc/salt/master):

file_roots:
  base:
    - /srv/salt/
    - /srv/formulas/datadog-formula/

Deployment

To deploy the Datadog Agent on your hosts:

  1. Add the Datadog formula to your top file (defaults to /srv/salt/top.sls):

    base:
      '*':
        - datadog
    
  2. Create datadog.sls in your pillar directory (defaults to /srv/pillar/). Add the following and update your Datadog API key:

    datadog:
      config:
        api_key: <YOUR_DD_API_KEY>
      install_settings:
        agent_version: <AGENT7_VERSION>
    
  3. Add datadog.sls to the top pillar file (defaults to /srv/pillar/top.sls):

    base:
      '*':
        - datadog
    

Configuration

The formula configuration must be written in the datadog key of the pillar file. It contains three parts: config, install_settings, and checks.

Config

Under config, add the configuration options to write to the minions' Agent configuration file (datadog.yaml for Agent v6 & v7, datadog.conf for Agent v5).

Depending on the Agent version installed, different options can be set:

  • Agent v6 & v7: all options supported by the Agent's configuration file are supported.
  • Agent v5: only the api_key option is supported.

The example below sets your Datadog API key and the Datadog site to datadoghq.eu (available for Agent v6 & v7).

  datadog:
    config:
      api_key: <YOUR_DD_API_KEY>
      site: datadoghq.eu

Install settings

Under install_settings, configure the Agent installation option:

  • agent_version: The version of the Agent to install (defaults to the latest Agent v7).

The example below installs Agent v6.14.1:

  datadog:
    install_settings:
      agent_version: 6.14.1

Checks

To add an Agent integration to your host, use the checks variable with the check's name as the key. Each check has two options:

Option Description
config Add the configuration options to write to the check's configuration file:
Agent v6 & v7: <confd_path>/<check>.d/conf.yaml
Agent v5: <confd_path>/<check>.yaml
version For Agent v6 & v7, the version of the check to install (defaults to the version bundled with the Agent).
third_party For Agent v6 & v7 (versions v6.21.0/v7.21.0 and higher only), boolean to indicate that the integration to install is a third-party integration. Must be paired with the version option.

Below is an example to use v1.4.0 of the Directory integration monitoring the /srv/pillar directory:

datadog:
  config:
    api_key: <YOUR_DD_API_KEY>
  install_settings:
    agent_version: <AGENT7_VERSION>
  checks:
    directory:
      config:
        instances:
          - directory: "/srv/pillar"
            name: "pillars"
      version: 1.4.0

Below is an example to use v1.0.0 of a sample third-party integration named "third-party-integration":

datadog:
  config:
    api_key: <YOUR_DD_API_KEY>
  install_settings:
    agent_version: <AGENT7_VERSION>
  checks:
    third-party-integration:
      config:
        instances:
          - some_config: "some value"
      version: 1.0.0
      third_party: true
Logs

To enable log collection, set logs_enabled to true in the main configuration:

datadog:
  config:
    logs_enabled: true

To send logs to Datadog, use the logs key in a check (either an existing check to setup logs for an integration, or a custom check to setup custom log collection). The following example uses a custom check named system_logs.

The contents of the config: key of this check is written to the /etc/datadog-agent/conf.d/<check_name>.d/conf.yaml file (in this example: /etc/datadog-agent/conf.d/system_logs.d/conf.yaml).

To list the logs you want to collect, fill the config section the same way you'd fill the conf.yaml file of a custom log collection configuration file (see the section on custom log collection in the official docs).

For instance, to collect logs from /var/log/syslog and /var/log/auth.log, the configuration would be:

datadog:
[...]
  checks:
    system_logs:
      config:
        logs:
          - type: file
            path: "/var/log/syslog"
            service: "system"
          - type: file
            path: "/var/log/auth.log"
            service: "system"

States

Salt formulas are pre-written Salt states. The following states are available in the Datadog formula:

State Description
datadog Installs, configures, and starts the Datadog Agent service.
datadog.install Configures the correct repo and installs the Datadog Agent.
datadog.config Configures the Datadog Agent and integrations using pillar data (see pillar.example).
datadog.service Runs the Datadog Agent service, which watches for changes to the config files for the Agent and checks.
datadog.uninstall Stops the service and uninstalls the Datadog Agent.

NOTE: When using datadog.config to configure different check instances on different machines, pillar_merge_lists must be set to True in the Salt master config or the Salt minion config if running masterless.

datadog-formula's People

Contributors

albertvaka avatar alq666 avatar amenasria avatar blacksmith77 avatar chouetz avatar chouquette avatar davedash avatar f4usto avatar irabinovitch avatar kserrania avatar masci avatar mseven avatar mx-psi avatar norrland avatar olivielpeau avatar onlyanegg avatar pythyu avatar remh avatar remicalixte avatar ruthnaebeck avatar sarina-dd avatar sc250024 avatar spencergilbert avatar talwai avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

datadog-formula's Issues

failed to compile

I'm unable to run the formula. Getting the error:

   Data failed to compile:
----------
    Invalid requisite type 'pkgrepo.managed' in state 'datadog-pkg', in SLS 'datadog'. Requisite types must not contain dots, did you mean 'pkgrepo'?
----------
    Invalid requisite type 'pkg.latest' in state 'datadog-conf', in SLS 'datadog'. Requisite types must not contain dots, did you mean 'pkg'?
----------
    Invalid requisite type 'cmd.run' in state 'datadog-conf', in SLS 'datadog'. Requisite types must not contain dots, did you mean 'cmd'?
----------
    Invalid requisite type 'pkg.latest' in state 'datadog-example', in SLS 'datadog'. Requisite types must not contain dots, did you mean 'pkg'?

Unstable agent installed by default

See below. Presumably it would be better to default to the stable version.

datadog-repo:
pkgrepo:
- managed
- humanname: "Datadog Agent"
{% if grains['os'].lower() in ('ubuntu', 'debian') %}
- name: deb http://apt.datadoghq.com/ unstable main
- keyserver: keyserver.ubuntu.com
- keyid: C7A7DA52
- file: /etc/apt/sources.list.d/datadog.list
{% elif grains['os'].lower() == 'redhat' %}
- name: Datadog, Inc.
- baseurl: http://yum.datadoghq.com/rpm/
{% endif %}

add datadog user to groups through pillar config

I'm managing several Ubuntu servers in AWS EC2 and want to ship /var/log/auth.log up to Datadog. However, the dd-agent user does not have permission to open the file since it does not belong to the adm group.

$ ls -al /var/log/auth.log
-rw-r----- 1 syslog adm 9824 Mar 22 17:50 /var/log/auth.log
$ sudo -u dd-agent groups
dd-agent

I can add dd-agent to the group on existing servers manually through Salt:

$ sudo salt 'targets' user.chgroups dd-agent adm append=True
$ sudo salt 'targets' service.restart datadog-agent

But it'd be nice to be able to specify the group membership in the pillar and avoid manual configuration steps (or an extra state across a bunch of minion classes in the topfile), especially as new minions come online.

Repository is not signed, no pubkey 33EE313BAD9589B7

Hi,
I setup some test containers running Ubuntu.
Installed salt-master on one of them, and minions on the others.

I followed the formula setup instructions in the README of this repo.

when I run salt '*' state.apply I get an error

Summary for 77661d052af9
------------
Succeeded: 2
Failed:    3
------------
Total states run:     5
Total run time:  37.522 s
41fc021fb2ca:
----------
          ID: datadog-apt-https
    Function: pkg.installed
        Name: apt-transport-https
      Result: True
     Comment: All specified packages are already installed
     Started: 06:32:35.321031
    Duration: 10.011 ms
     Changes:   
----------
          ID: datadog-repo
    Function: pkgrepo.managed
        Name: deb https://apt.datadoghq.com/ stable 7
      Result: True
     Comment: Attempt 1: Returned a result of "False", with the following comment: "Failed to configure repo 'deb https://apt.datadoghq.com/ stable 7': W: GPG error: https://apt.datadoghq.com stable Release: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 33EE313BAD9589B7
              E: The repository 'https://apt.datadoghq.com stable Release' is not signed."
              Package repo 'deb https://apt.datadoghq.com/ stable 7' already configured
     Started: 06:32:35.332887
    Duration: 35697.684 ms
     Changes:   
----------
          ID: datadog-pkg
    Function: pkg.installed
        Name: datadog-agent
      Result: False
     Comment: An exception occurred in this state: Traceback (most recent call last):
                File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/state.py", line 2385, in call
                  ret = self.states[cdata["full"]](
                File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/loader/lazy.py", line 149, in __call__
                  return self.loader.run(run_func, *args, **kwargs)
                File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/loader/lazy.py", line 1232, in run
                  return self._last_context.run(self._run_as, _func_or_method, *args, **kwargs)
                File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/loader/lazy.py", line 1247, in _run_as
                  return _func_or_method(*args, **kwargs)
                File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/loader/lazy.py", line 1280, in wrapper
                  return f(*args, **kwargs)
                File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/states/pkg.py", line 1703, in installed
                  result = _find_install_targets(
                File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/states/pkg.py", line 714, in _find_install_targets
                  resolved_latest = __salt__["pkg.latest_version"](
                File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/loader/lazy.py", line 149, in __call__
                  return self.loader.run(run_func, *args, **kwargs)
                File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/loader/lazy.py", line 1232, in run
                  return self._last_context.run(self._run_as, _func_or_method, *args, **kwargs)
                File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/loader/lazy.py", line 1247, in _run_as
                  return _func_or_method(*args, **kwargs)
                File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/modules/aptpkg.py", line 470, in latest_version
                  refresh_db(cache_valid_time)
                File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/modules/aptpkg.py", line 595, in refresh_db
                  raise CommandExecutionError(comment)
              salt.exceptions.CommandExecutionError: W: GPG error: https://apt.datadoghq.com stable Release: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 33EE313BAD9589B7
              E: The repository 'https://apt.datadoghq.com stable Release' is not signed.
     Started: 06:33:11.050859
    Duration: 1936.821 ms
     Changes:   
----------
          ID: datadog_yaml_installed
    Function: file.managed
        Name: /etc/datadog-agent/datadog.yaml
      Result: False
     Comment: One or more requisite failed: datadog.install.datadog-pkg
     Started: 06:33:13.005056
    Duration: 0.003 ms
     Changes:   
----------
          ID: datadog-agent-service
    Function: service.running
        Name: datadog-agent
      Result: False
     Comment: One or more requisite failed: datadog.install.datadog-pkg, datadog.config.datadog_yaml_installed
     Started: 06:33:13.007942
    Duration: 0.002 ms
     Changes:   

Contents of /etc/salt/master

# https://github.com/DataDog/datadog-formula
fileserver_backend:
  - roots # Active by default, necessary to be able to use the local salt files we define in the next steps
  - gitfs # Adds gitfs as a fileserver backend to be able to use gitfs_remotes

gitfs_remotes:
  - https://github.com/DataDog/datadog-formula.git:
    - saltenv:
      - base:
        - ref: 3.0 # Pin the version of the formula you want to use

Contents of /srv/salt/top.sls

# https://app.datadoghq.com/account/settings/agent/latest?platform=saltstack
base:
  '*':
    - datadog

Contents of /srv/pillar/datadog.sls

base:
  '*':
    - datadog

Contents of /srv/pillar/top.sls

datadog:
  config:
    api_key: xxx
    site: datadoghq.com
  # install_settings:
  #   agent_version: <AGENT7_VERSION>

TraceFS no permissions, not respecting blacklists

**Output of the info page **

Getting the status from the agent.

===============
Agent (v7.32.1)
===============

  Status date: 2021-12-07 22:28:27.104 UTC (1638916107104)
  Agent start: 2021-12-07 22:25:25.226 UTC (1638915925226)
  Pid: 25010
  Go Version: go1.16.7
  Python Version: 3.8.11
  Build arch: amd64
  Agent flavor: agent
  Check Runners: 4
  Log Level: info

  Paths
  =====
    Config File: /etc/datadog-agent/datadog.yaml
    conf.d: /etc/datadog-agent/conf.d
    checks.d: /etc/datadog-agent/checks.d

  Clocks
  ======
    NTP offset: 18µs
    System time: 2021-12-07 22:28:27.104 UTC (1638916107104)

  Host Info
  =========
    bootTime: 2021-04-29 15:14:30 UTC (1619709270000)
    kernelArch: x86_64
    kernelVersion: 5.4.0-1045-aws
    os: linux
    platform: ubuntu
    platformFamily: debian
    platformVersion: 18.04
    procs: 143
    uptime: 5335h10m56s

  Hostnames
  =========
    ec2-hostname: ip-10-104-11-214.ec2.internal
    hostname: salt-master-1.devops.corp.supercoolplace.net
    instance-id: i-09a3c862a797dd512
    socket-fqdn: salt
    socket-hostname: salt-master-1.devops.corp.supercoolplace.net
    host tags:
      Name:salt-master-1.devops.corp.supercoolplace.net
    hostname provider: os
    unused hostname providers:
      aws: not retrieving hostname from AWS: the host is not an ECS instance and other providers already retrieve non-default hostnames
      azure: cloud provider is disabled by configuration
      configuration/environment: hostname is empty
      gce: cloud provider is disabled by configuration

  Metadata
  ========
    cloud_provider: AWS
    hostname_source: os

=========
Collector
=========

  Running Checks
  ==============
    
    cpu
    ---
      Instance ID: cpu [OK]
      Configuration Source: file:/etc/datadog-agent/conf.d/cpu.d/conf.yaml.default
      Total Runs: 12
      Metric Samples: Last Run: 9, Total: 101
      Events: Last Run: 0, Total: 0
      Service Checks: Last Run: 0, Total: 0
      Average Execution Time : 0s
      Last Execution Date : 2021-12-07 22:28:18 UTC (1638916098000)
      Last Successful Execution Date : 2021-12-07 22:28:18 UTC (1638916098000)
      
    
    disk (4.4.0)
    ------------
      Instance ID: disk:e5dffb8bef24336f [OK]
      Configuration Source: file:/etc/datadog-agent/conf.d/disk.d/conf.yaml.default
      Total Runs: 12
      Metric Samples: Last Run: 144, Total: 1,728
      Events: Last Run: 0, Total: 0
      Service Checks: Last Run: 0, Total: 0
      Average Execution Time : 24ms
      Last Execution Date : 2021-12-07 22:28:25 UTC (1638916105000)
      Last Successful Execution Date : 2021-12-07 22:28:25 UTC (1638916105000)
      
    
    file_handle
    -----------
      Instance ID: file_handle [OK]
      Configuration Source: file:/etc/datadog-agent/conf.d/file_handle.d/conf.yaml.default
      Total Runs: 12
      Metric Samples: Last Run: 5, Total: 60
      Events: Last Run: 0, Total: 0
      Service Checks: Last Run: 0, Total: 0
      Average Execution Time : 0s
      Last Execution Date : 2021-12-07 22:28:17 UTC (1638916097000)
      Last Successful Execution Date : 2021-12-07 22:28:17 UTC (1638916097000)
      
    
    io
    --
      Instance ID: io [OK]
      Configuration Source: file:/etc/datadog-agent/conf.d/io.d/conf.yaml.default
      Total Runs: 12
      Metric Samples: Last Run: 130, Total: 1,470
      Events: Last Run: 0, Total: 0
      Service Checks: Last Run: 0, Total: 0
      Average Execution Time : 0s
      Last Execution Date : 2021-12-07 22:28:24 UTC (1638916104000)
      Last Successful Execution Date : 2021-12-07 22:28:24 UTC (1638916104000)
      
    
    load
    ----
      Instance ID: load [OK]
      Configuration Source: file:/etc/datadog-agent/conf.d/load.d/conf.yaml.default
      Total Runs: 12
      Metric Samples: Last Run: 6, Total: 72
      Events: Last Run: 0, Total: 0
      Service Checks: Last Run: 0, Total: 0
      Average Execution Time : 0s
      Last Execution Date : 2021-12-07 22:28:16 UTC (1638916096000)
      Last Successful Execution Date : 2021-12-07 22:28:16 UTC (1638916096000)
      
    
    memory
    ------
      Instance ID: memory [OK]
      Configuration Source: file:/etc/datadog-agent/conf.d/memory.d/conf.yaml.default
      Total Runs: 12
      Metric Samples: Last Run: 18, Total: 216
      Events: Last Run: 0, Total: 0
      Service Checks: Last Run: 0, Total: 0
      Average Execution Time : 0s
      Last Execution Date : 2021-12-07 22:28:23 UTC (1638916103000)
      Last Successful Execution Date : 2021-12-07 22:28:23 UTC (1638916103000)
      
    
    network (2.4.0)
    ---------------
      Instance ID: network:d884b5186b651429 [OK]
      Configuration Source: file:/etc/datadog-agent/conf.d/network.d/conf.yaml.default
      Total Runs: 12
      Metric Samples: Last Run: 26, Total: 312
      Events: Last Run: 0, Total: 0
      Service Checks: Last Run: 0, Total: 0
      Average Execution Time : 1ms
      Last Execution Date : 2021-12-07 22:28:15 UTC (1638916095000)
      Last Successful Execution Date : 2021-12-07 22:28:15 UTC (1638916095000)
      
    
    ntp
    ---
      Instance ID: ntp:d884b5186b651429 [OK]
      Configuration Source: file:/etc/datadog-agent/conf.d/ntp.d/conf.yaml.default
      Total Runs: 1
      Metric Samples: Last Run: 1, Total: 1
      Events: Last Run: 0, Total: 0
      Service Checks: Last Run: 1, Total: 1
      Average Execution Time : 0s
      Last Execution Date : 2021-12-07 22:25:26 UTC (1638915926000)
      Last Successful Execution Date : 2021-12-07 22:25:26 UTC (1638915926000)
      
    
    process (2.1.0)
    ---------------
      Instance ID: process:ssh:85f05505d53a5fad [WARNING]
      Configuration Source: file:/etc/datadog-agent/conf.d/process.d/conf.yaml
      Total Runs: 13
      Metric Samples: Last Run: 17, Total: 219
      Events: Last Run: 0, Total: 0
      Service Checks: Last Run: 1, Total: 13
      Average Execution Time : 10ms
      Last Execution Date : 2021-12-07 22:28:26 UTC (1638916106000)
      Last Successful Execution Date : 2021-12-07 22:28:26 UTC (1638916106000)
      
      Warning: DEPRECATION NOTICE: Specifying `procfs_path` in process.yaml` is deprecated. Please specify it in `datadog.conf` instead
        
    
    uptime
    ------
      Instance ID: uptime [OK]
      Configuration Source: file:/etc/datadog-agent/conf.d/uptime.d/conf.yaml.default
      Total Runs: 12
      Metric Samples: Last Run: 1, Total: 12
      Events: Last Run: 0, Total: 0
      Service Checks: Last Run: 0, Total: 0
      Average Execution Time : 0s
      Last Execution Date : 2021-12-07 22:28:22 UTC (1638916102000)
      Last Successful Execution Date : 2021-12-07 22:28:22 UTC (1638916102000)
      
========
JMXFetch
========

  Information
  ==================
  Initialized checks
  ==================
    no checks
    
  Failed checks
  =============
    no checks
    
=========
Forwarder
=========

  Transactions
  ============
    Cluster: 0
    ClusterRole: 0
    ClusterRoleBinding: 0
    CronJob: 0
    DaemonSet: 0
    Deployment: 0
    Dropped: 0
    HighPriorityQueueFull: 0
    Job: 0
    Node: 0
    PersistentVolume: 0
    PersistentVolumeClaim: 0
    Pod: 0
    ReplicaSet: 0
    Requeued: 0
    Retried: 0
    RetryQueueSize: 0
    Role: 0
    RoleBinding: 0
    Service: 0
    ServiceAccount: 0
    StatefulSet: 0

  Transaction Successes
  =====================
    Total number: 26
    Successes By Endpoint:
      check_run_v1: 12
      intake: 2
      series_v1: 12

  API Keys status
  ===============
    API key ending with af1ce: API Key valid

==========
Endpoints
==========
  https://app.datadoghq.com - API Key ending with:
      - af1ce

==========
Logs Agent
==========

    Sending compressed logs in HTTPS to agent-http-intake.logs.datadoghq.com on port 443
    BytesSent: 92788
    EncodedBytesSent: 16730
    LogsProcessed: 254
    LogsSent: 250

  system_logs
  -----------
    - Type: file
      Path: /var/log/syslog
      Status: OK
      Inputs:
        /var/log/syslog
      BytesRead: 49639
      Average Latency (ms): 100
      24h Average Latency (ms): 100
      Peak Latency (ms): 2055
      24h Peak Latency (ms): 2055
    - Type: file
      Path: /var/log/auth.log
      Status: OK
      Inputs:
        /var/log/auth.log
      BytesRead: 999
      Average Latency (ms): 544
      24h Average Latency (ms): 544
      Peak Latency (ms): 1631
      24h Peak Latency (ms): 1631

=========
APM Agent
=========
  Status: Running
  Pid: 25012
  Uptime: 181 seconds
  Mem alloc: 8,574,400 bytes
  Hostname: salt-master-1.devops.corp.supercoolplace.net
  Receiver: localhost:8126
  Endpoints:
    https://trace.agent.datadoghq.com

  Receiver (previous minute)
  ==========================
    No traces received in the previous minute.
    Default priority sampling rate: 100.0%

  Writer (previous minute)
  ========================
    Traces: 0 payloads, 0 traces, 0 events, 0 bytes
    Stats: 0 payloads, 0 stats buckets, 0 bytes

=========
Aggregator
=========
  Checks Metric Sample: 4,411
  Dogstatsd Metric Sample: 1,435
  Event: 1
  Events Flushed: 1
  Number Of Flushes: 12
  Series Flushed: 4,380
  Service Check: 124
  Service Checks Flushed: 133
=========
DogStatsD
=========
  Event Packets: 0
  Event Parse Errors: 0
  Metric Packets: 1,434
  Metric Parse Errors: 0
  Service Check Packets: 0
  Service Check Parse Errors: 0
  Udp Bytes: 109,931
  Udp Packet Reading Errors: 0
  Udp Packets: 1,031
  Uds Bytes: 0
  Uds Origin Detection Errors: 0
  Uds Packet Reading Errors: 0
  Uds Packets: 0
  Unterminated Metric Errors: 0


Additional environment details (Operating System, Cloud provider, etc):

Steps to reproduce the issue:

  1. Use the salt-formula
  2. Use the following config:
  checks:
    system_logs:
      config:
        logs:
          - type: file
            path: "/var/log/syslog"
            service: "system"
            source: syslog
            auto_multi_line_detection: true
          - type: file
            path: "/var/log/auth.log"
            service: "system"
            source: authlog
    disk:
      config:
        file_system_blacklist:
          - tracefs
        mount_point_blacklist:
          - /sys/kernel/debug/tracing
  1. Verify the config does indeed exist:

My lovely disk.d/conf.yaml

mwilliams@salt-master-1:/etc/datadog-agent$ sudo cat conf.d/disk.d/conf.yaml
init_config:
file_system_blacklist:
- tracefs
mount_point_blacklist:
- /sys/kernel/debug/tracing

The part of the status command the is relevant:

    disk (4.4.0)
    ------------
      Instance ID: disk:e5dffb8bef24336f [OK]
      Configuration Source: file:/etc/datadog-agent/conf.d/disk.d/conf.yaml.default
      Total Runs: 3
      Metric Samples: Last Run: 144, Total: 432
      Events: Last Run: 0, Total: 0
      Service Checks: Last Run: 0, Total: 0
      Average Execution Time : 21ms
      Last Execution Date : 2021-12-07 22:26:10 UTC (1638915970000)
      Last Successful Execution Date : 2021-12-07 22:26:10 UTC (1638915970000)

Its using the /etc/datadog-agent/conf.d/disk.d/conf.yaml.default instead of the one salt configured at /etc/datadog-agent/conf.d/disk.d/conf.yaml

Describe the results you received:
My lovely agent logs:

2021-12-07 22:14:44 UTC | CORE | INFO | (pkg/collector/worker/check_logger.go:56 in CheckFinished) | check:disk | Done running check, next runs will be logged every 500 runs
2021-12-07 22:14:45 UTC | CORE | INFO | (pkg/collector/worker/check_logger.go:37 in CheckStarted) | check:process | Running check...
2021-12-07 22:14:45 UTC | CORE | WARN | (pkg/collector/python/datadog_agent.go:124 in LogMessage) | process:ssh:85f05505d53a5fad | (process.py:399) | DEPRECATION NOTICE: Specifying `procfs_path` in process.yaml` is deprecated. Please specify it in `datadog.conf` instead
2021-12-07 22:14:49 UTC | CORE | INFO | (pkg/collector/worker/check_logger.go:37 in CheckStarted) | check:network | Running check...
2021-12-07 22:14:50 UTC | CORE | INFO | (pkg/collector/worker/check_logger.go:37 in CheckStarted) | check:load | Running check...
2021-12-07 22:14:51 UTC | CORE | INFO | (pkg/collector/worker/check_logger.go:37 in CheckStarted) | check:file_handle | Running check...
2021-12-07 22:14:52 UTC | CORE | INFO | (pkg/collector/worker/check_logger.go:37 in CheckStarted) | check:cpu | Running check...
2021-12-07 22:14:56 UTC | CORE | INFO | (pkg/collector/worker/check_logger.go:37 in CheckStarted) | check:uptime | Running check...
2021-12-07 22:14:57 UTC | CORE | INFO | (pkg/collector/worker/check_logger.go:37 in CheckStarted) | check:memory | Running check...
2021-12-07 22:14:58 UTC | CORE | INFO | (pkg/collector/worker/check_logger.go:37 in CheckStarted) | check:io | Running check...
2021-12-07 22:14:59 UTC | CORE | INFO | (pkg/collector/worker/check_logger.go:37 in CheckStarted) | check:disk | Running check...
2021-12-07 22:14:59 UTC | CORE | WARN | (pkg/collector/python/datadog_agent.go:124 in LogMessage) | disk:e5dffb8bef24336f | (disk.py:137) | Unable to get disk metrics for /sys/kernel/debug/tracing: [Errno 13] Permission denied: '/sys/kernel/debug/tracing'. You can exclude this mountpoint in the settings if it is invalid.

It does not seem to be honoring the blacklist item that I added per this issue: 2932

Describe the results you expected:
I expected the FS blacklist to be honored and the error message to disappear.

Additional information you deem important (e.g. issue happens only occasionally):

The root issue is likely that the formula seems to have deleted the or not copied over the default file for system_logs and does for disks.conf.d:

mwilliams@salt-master-1:/etc/datadog-agent$ sudo ls -als conf.d/disk.d/
total 24
 4 drwx------   2 dd-agent root     4096 Dec  7 22:13 .
 4 drwxr-xr-x 160 dd-agent dd-agent 4096 Nov 30 22:17 ..
 4 -rw-------   1 dd-agent root       99 Dec  7 22:13 conf.yaml
12 -rw-r--r--   1 dd-agent dd-agent 9165 Nov 18 14:14 conf.yaml.default
mwilliams@salt-master-1:/etc/datadog-agent$ sudo ls -als conf.d/system
system_core.d/ system_logs.d/ system_swap.d/ systemd.d/     
mwilliams@salt-master-1:/etc/datadog-agent$ sudo ls -als conf.d/system_logs.d/
total 12
4 drwx------   2 dd-agent root     4096 Dec  7 21:53 .
4 drwxr-xr-x 160 dd-agent dd-agent 4096 Nov 30 22:17 ..
4 -rw-------   1 dd-agent root      202 Dec  7 21:53 conf.yaml
mwilliams@salt-master-1:/etc/datadog-agent$ 

Add configuration options to datadog.yml

The DD formula has worked great but we are needing to tweak settings within datadog.yml such as tags, collect_ec2_tags, and enable_metadata_collection.

I do see a related PR #24. Looking at how this is designed as the number of options need to get added the amount of if statements will continue to grow (https://github.com/DataDog/datadog-formula/pull/24/files#diff-14b50415a1c3f0d35d35d0e6bd0b322aR40).

Would this repo be open/interest to move datadog.yml to a jinja templated file, so an user could just plug values into pillar for arbitrary config?

Example of the test.pillar updated that would write to datadog.yml only under the dd_config block:

datadog:
  api_key: aaaaaaaabbbbbbbbccccccccdddddddd
  checks:
    process:
      init_config:
        procfs_path: /proc
      instances:
        - name: ssh
          search_string: ['sshd']
    tcp_check:
      instances:
        - host: 127.0.0.1
          name: sshd
          port: 22
  dd_config:
    - collect_ec2_tags: True
    - enable_metadata_collection: True

Adhering to v6 agent conf.d directory structure

Hello,
Trying to get datadog-formula to work with v6 agent directory structure, I had a configuration similar to this:

datadog:
  checks_confd: /etc/datadog-agent/conf.d
  api_key: aabbcc

  checks:
    elasticsearch:
      instances:
        - url: http://localhost:9200
           cluster_stats: false
           pshard_stats: true

    process:
      instances:
        - name: elasticsearch
          search_string: ['elasticsearch']
          exact_match: False

The problem is, it is creating config files for elasticsearch and process checks in conf.d root dir, and not in subdirectories elastic.d and process.d.

Searching through the code, I found the config_folder variable, which sounded like what would be needed to put the files in their right place, but since it was being searched for in datadog_settings.config_folder, it can't be set per check.

The following change in datadog-formula's config.sls seems to do the trick:

diff --git a/datadog/config.sls b/datadog/config.sls
index 97fca75..f76d3a1 100644
--- a/datadog/config.sls
+++ b/datadog/config.sls
@@ -27,7 +27,7 @@ datadog-conf:
 {% for check_name in datadog_settings.checks %}
 datadog_{{ check_name }}_yaml_installed:
   file.managed:
-    - name: {{ datadog_settings.checks_confd }}/{{ check_name }}.yaml
+    - name: {{ datadog_settings.checks_confd }}/{{ datadog_settings.checks[check_name]['config_folder'] }}/{{ check_name }}.yaml
     - source: salt://datadog/files/conf.yaml.jinja
     - user: dd-agent
     - group: root

Now, when you use config_folder per check in the sls file, config files get placed where they should be:

datadog:
  checks_confd: /etc/datadog-agent/conf.d
  api_key: aabbcc

  checks:
    elasticsearch:
      config_folder: elastic.d
      instances:
        -
          url: http://localhost:9200
          cluster_stats: false
          pshard_stats: true

    process:
      config_folder: process.d
      instances:
        - name: elasticsearch
          search_string: ['elasticsearch']
          exact_match: False

Am I doing something wrong, or is this what was missing from the formula?
Thanks,
Nikola

Difficulty upgrading to agent v7

Attempting to upgrade our saltstack instances to run Datadog Agent v7 or latest and having no success. We currently don't define a version on Debian 10 buster, so it has defaulted to

# dpkg-query -l |grep -i datado
ii  datadog-agent                         1:6.22.1-1                        amd64        Datadog Monitoring Agent

I've added the install_settings: agent_version: section to our pillar/init.sls and applying the state does not change anything. The sources.list.d file still specifies version 6, whether I set agent_version to 7.22.1 or 'latest'.

datadog:
  api_key: {{ api_key }}
  install_settings:
    agent_version: 7.22.1

Additionally, I've removed the package and the sources.list.d file and allowed salt to recreate them, and they are still being created and installed as v6.

Expected Behaviour

I'd expect that when changing from a 6.x to 7.x or 'latest' version that the package manager configuration, apt sources in my case, would change to the proper settings for v7, the package manager would run an update to confirm it has the latest information, and then the salt state would install the v7 or 'latest' which the docs say is v7.

Observed Behaviour

Version 6.x is installed no matter what combinations I attempt. I've changed the version to 6.14.1 and removed the 6.latest installation and had the state successfully install the older version, meaning that the formula is apparently reading in the agent_version variable.

Using the pillar, how to monitor logs?

Defining logs in the pillar for the Datadog formula creates a /conf.d/logs.d/conf.yaml that has a single empty line. I have tried moving the logs section around in the pillar with no success.

Pillar:

#!jinja|yaml
datadog:
  config:
    api_key: XXXXXXXXXXXXXXXX
    site: datadoghq.com
    python_version: 2
    logs_enabled: true
    logs_config:
      use_http: true
      use_compression: true
      compression_level: 6
  checks:
    logs:
      - type: file
        path: "/var/log/syslog"
        service: "system"
      - type: file
        path: "/var/log/auth.log"
        service: "system"
    process:
      config:
        instances:
          - name: ssh
            search_string: ['sshd']
    tcp_check:
      config:
        instances:
          - host: 127.0.0.1
            name: sshd
            port: 22
  install_settings:
    agent_version: 6.17.0

Output:

root@saltmaster:/srv/pillars/datadog# salt 'minion1' state.apply datadog
minion1:
----------
          ID: datadog-apt-https
    Function: pkg.installed
        Name: apt-transport-https
      Result: True
     Comment: All specified packages are already installed
     Started: 16:08:57.695942
    Duration: 35.994 ms
     Changes:
----------
          ID: datadog-repo
    Function: pkgrepo.managed
        Name: deb https://apt.datadoghq.com/ stable 6
      Result: True
     Comment: Package repo 'deb https://apt.datadoghq.com/ stable 6' already configured
     Started: 16:08:57.732831
    Duration: 25.258 ms
     Changes:
----------
          ID: datadog-pkg
    Function: pkg.installed
        Name: datadog-agent
      Result: True
     Comment: All specified packages are already installed and are at the desired version
     Started: 16:08:57.758943
    Duration: 22.19 ms
     Changes:
----------
          ID: datadog_yaml_installed
    Function: file.managed
        Name: /etc/datadog-agent/datadog.yaml
      Result: True
     Comment: File /etc/datadog-agent/datadog.yaml is in the correct state
     Started: 16:08:57.783042
    Duration: 28.737 ms
     Changes:
----------
          ID: datadog_process_folder_installed
    Function: file.directory
        Name: /etc/datadog-agent/conf.d/process.d
      Result: True
     Comment: The directory /etc/datadog-agent/conf.d/process.d is in the correct state
     Started: 16:08:57.811937
    Duration: 0.461 ms
     Changes:
----------
          ID: datadog_process_old_yaml_removed
    Function: file.absent
        Name: /etc/datadog-agent/conf.d/process.yaml
      Result: True
     Comment: File /etc/datadog-agent/conf.d/process.yaml is not present
     Started: 16:08:57.812483
    Duration: 0.221 ms
     Changes:
----------
          ID: datadog_process_yaml_installed
    Function: file.managed
        Name: /etc/datadog-agent/conf.d/process.d/conf.yaml
      Result: True
     Comment: File /etc/datadog-agent/conf.d/process.d/conf.yaml is in the correct state
     Started: 16:08:57.812784
    Duration: 20.969 ms
     Changes:
----------
          ID: datadog_tcp_check_folder_installed
    Function: file.directory
        Name: /etc/datadog-agent/conf.d/tcp_check.d
      Result: True
     Comment: The directory /etc/datadog-agent/conf.d/tcp_check.d is in the correct state
     Started: 16:08:57.833873
    Duration: 0.512 ms
     Changes:
----------
          ID: datadog_tcp_check_old_yaml_removed
    Function: file.absent
        Name: /etc/datadog-agent/conf.d/tcp_check.yaml
      Result: True
     Comment: File /etc/datadog-agent/conf.d/tcp_check.yaml is not present
     Started: 16:08:57.834486
    Duration: 0.24 ms
     Changes:
----------
          ID: datadog_tcp_check_yaml_installed
    Function: file.managed
        Name: /etc/datadog-agent/conf.d/tcp_check.d/conf.yaml
      Result: True
     Comment: File /etc/datadog-agent/conf.d/tcp_check.d/conf.yaml is in the correct state
     Started: 16:08:57.834824
    Duration: 20.53 ms
     Changes:
----------
          ID: datadog_logs_folder_installed
    Function: file.directory
        Name: /etc/datadog-agent/conf.d/logs.d
      Result: True
     Comment: Directory /etc/datadog-agent/conf.d/logs.d updated
     Started: 16:08:57.855455
    Duration: 1.127 ms
     Changes:
              ----------
              /etc/datadog-agent/conf.d/logs.d:
                  New Dir
----------
          ID: datadog_logs_old_yaml_removed
    Function: file.absent
        Name: /etc/datadog-agent/conf.d/logs.yaml
      Result: True
     Comment: File /etc/datadog-agent/conf.d/logs.yaml is not present
     Started: 16:08:57.856672
    Duration: 0.215 ms
     Changes:
----------
          ID: datadog_logs_yaml_installed
    Function: file.managed
        Name: /etc/datadog-agent/conf.d/logs.d/conf.yaml
      Result: True
     Comment: File /etc/datadog-agent/conf.d/logs.d/conf.yaml updated
     Started: 16:08:57.856969
    Duration: 19.925 ms
     Changes:
              ----------
              diff:
                  New file
              user:
                  dd-agent
----------
          ID: datadog-agent-service
    Function: service.running
        Name: datadog-agent
      Result: True
     Comment: Service restarted
     Started: 16:08:57.914280
    Duration: 1164.851 ms
     Changes:
              ----------
              datadog-agent:
                  True

Summary for minion1
-------------
Succeeded: 14 (changed=3)
Failed:     0
-------------
Total states run:     14
Total run time:    1.341 s

[Amazon Linux] Bad id for repo: Datadog, Inc., byte = , 7

Ran into the error Bad id for repo: Datadog, Inc., byte = , 7 when trying to install on Amazon Linux.

Modifying the formula to the following removed the error:

datadog-repo:
  pkgrepo:
    - managed
    - humanname: "Datadog Agent"
    {% if grains['os'].lower() in ('ubuntu', 'debian') %}
    - name: deb http://apt.datadoghq.com/ stable main
    - keyserver: keyserver.ubuntu.com
    - keyid: C7A7DA52
    - file: /etc/apt/sources.list.d/datadog.list
    {% elif grains['os'].lower() in ('redhat', 'amazon') %}
    - name: datadog-inc
    - baseurl: http://yum.datadoghq.com/rpm/x86_64
    {% endif %}

Changes amazon added to the elif and name: datadog-inc

Support for enabling network tracking

Hello :)

I can't see any support for enable the network scanning. For manual install this would be configured in the /etc/datadog-agent/system-probe.yaml file. Have I missed something or is this something I'll need to add myself?

Thanks :)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.