Git Product home page Git Product logo

puppet-graylogcollectorsidecar's Introduction

dodevops/graylogcollectorsidecar

Travis

Deprecation notice

This module is deprecated because of the new, incompatible and (in our eyes) unusable new sidecar feature from Graylog 3. If somebody wants to adopt it, please open an issue.

Table of Contents

  1. Description
  2. Setup
  3. Reference
  4. Limitations

Description

This module installs and configures the Graylog collector sidecar.

Setup

Beginning with graylogcollectorsidecar

To install the graylog collector sidecar, simply configure the class:

class { 'graylogcollectorsidecar':
  api_url => 'http://graylog.example.com:9000/api',
  version => '0.1.0',
  tags => [ 'apache.accesslog' ]
}

Or using hiera:

graylogcollectorsidecar::api_url: "http://graylog.example.com:9000/api"
graylogcollectorsidecar::version: "0.1.0"
graylogcollectorsidecar::tags:
    - apache.accesslog

Reference

class graylogcollectorsidecar

Parameter Description
version Select the version of the collector to install. Defaults to 'latest', which selects the latest available release version
api_url Graylog server api url (e.g. http://graylog.example.com:9000/api)
tags An array of tags that the collector should be set up with

Additionally, all other parameters as noted in the collector sidecar documentation can be specified.

The node_id will be set to the local hostname, if not specified.

Limitations

This module uses the githubreleases module to download the graylog distribution package from Github. However, Github imposes a rate limiting on unauthenticated requests, which the module does (currently, this module doesn't allow setting Github credentials).

So if you have a rather large deployment using this module, the rate limit might fail the deployment. If so, you'll have to wait for the Rate limit to be reset.

If this is a constant pain, please open an issue.

OS compatibility:

  • Debian-Family (Ubuntu, Debian)
  • RedHat-Family (RHEL, CentOS)

puppet-graylogcollectorsidecar's People

Contributors

dploeger avatar timdeluxe avatar jarodiv avatar devcfgc avatar

Watchers

 avatar James Cloos avatar Taulant Geci avatar  avatar  avatar Thomas Minor avatar  avatar  avatar  avatar

puppet-graylogcollectorsidecar's Issues

::Fixnum is deprecated

This module is using yaml_settings and that module is droping a deprecation warning.

/var/lib/puppetlabs/puppet/cache/lib/puppet/provider/yaml_setting/mapped.rb:106: warning: constant ::Fixnum is deprecated

version => 'latest' is not working (at least on CentOS)

The default value for graylogcollectorsidecar::version is "latest". Unfortunately, this causes puppet to fail with the message:

Error: Evaluation Error: Error while evaluating a Resource Statement, Evaluation Error: Error while evaluating a Function Call, Can not find a valid download URL for the release. at <...>/githubreleases/manifests/download.pp:69:17 at <...>/graylogcollectorsidecar/manifests/dist/redhat.pp:24 on node <...>

Locating the error, took a little dive into the code:

Including graylogcollectorsidecar with graylogcollectorsidecar::version: 'latest' (api_url and tags are set to meaningful values) instantiates the class graylogcollectorsidecar with the parameters

  $api_url,
  $tags,
  $update_interval   = undef,
  $tls_skip_verify   = undef,
  $send_status       = undef,
  $list_log_files    = undef,
  $node_id           = undef,
  $collector_id      = undef,
  $log_path          = undef,
  $log_rotation_time = undef,
  $log_max_age       = undef,
  $backends          = undef,
  $version           = 'latest'
) {…}




which uses on RedHat systems the class graylogcollectorsidecar::dist::redhat (arguments replaced for better traceability)



    'RedHat': {
      class {
        'graylogcollectorsidecar::dist::redhat':
          version           => ‘latest’,
          api_url           => $api_url,
          tags              => $tags,
          update_interval   => undef,
          tls_skip_verify   => undef,
          send_status       => undef,
          list_log_files    => undef,
          node_id           => undef,
          collector_id      => undef,
          log_path          => undef,
          log_rotation_time => undef,
          log_max_age       => undef,
          backends          => undef
      }
    }




In dist/redhat.pp this brings us to



    githubreleases::download {
      'get_sidecar_package':
        author            => 'Graylog2',
        repository        => 'collector-sidecar',
        release           => ‘latest’,
        is_tag            => true,
        asset             => true,
        asset_filepattern => "${::architecture}\\.rpm",
        target            => '/tmp/collector-sidecar.rpm'
    }




The hardcoded value is_tag => true, will become important later. For the class githubreleases::download this means an instantiation with the parameter


  $author            = 'Graylog2',
  $repository        = 'collector-sidecar',
  $release           = ‘latest’,
  $asset             = true,
  $use_zip           = undef,
  $asset_filepattern = "${::architecture}\\.rpm",
  $asset_contenttype = undef,
  $asset_fallback    = undef,
  $is_tag            = true,
  $use_auth          = undef,
  $username          = undef,
  $password          = undef,
  $target            = '/tmp/collector-sidecar.rpm'




The class itself calls the ruby lib github_release to generate the download URL (as usual, all parameters are already replaced)


 $source_url = github_release({
    author            => 'Graylog2',
    repository        => ‘collector-sidecar',
    release           => ‘latest’,
    asset             => true,
    use_zip           => undef,
    asset_filepattern => "${::architecture}\\.rpm",
    asset_contenttype => undef,
    asset_fallback    => undef,
    is_tag            => true,
    use_auth          => undef,
    username          => undef,
    password          => undef
  })




Taking a look into lib/puppet/parser/functions/github_release.rb brings us to the lines (keep the hardcoded /releases in the sprint function in mind):


    tag_part = ''

    if arguments[:is_tag]
      tag_part = 'tags/'
    end

    url = sprintf(
        '%s/%s/%s/releases/%s%s',
        'https://api.github.com/repos',
        arguments[:author],
        arguments[:repository],
        tag_part,
        arguments[:release]
    )

Since is_tag is true, tag_part is set to 'tags/‘ so that URL is returned as the result of



    sprintf(
        '%s/%s/%s/releases/%s%s',
        'https://api.github.com/repos',
        ‘Graylog2’,
        ‘collector-sidecar',
        'tags/',
        ‘latest’
    )



Which is "https://api.github.com/repos/Graylog2/collector-sidecar/releases/tags/latest" and does not exist:
{ "message": "Not Found", "documentation_url": "https://developer.github.com/v3" } 




What worked for me was:


  • latest release -> https://api.github.com/repos/Graylog2/collector-sidecar/releases/latest
  • specific release -> https://api.github.com/repos/Graylog2/collector-sidecar/releases/tags/0.1.4

    That means, if a release is specified, then is_tag has to be set to true. If “latest” is selected, then is_tag has to be false. Sadly, the value of is_tag is hardcoded and can not be configured, so that the only workaround for me currently is to specify a version instead of choosing “latest”.



(a pull request with a bug fix to this issue will follow)

Should only show puppet output on change

Currently, even if no configuration changed, a puppetrun with outputs 80 lines of graylog collector data. It seems every config option generates four lines of output. The output should be 0 lines if configuration did not change. This severely spams reporting and logging if used on many servers. Also, it makes it siginificantly harder to debug puppet runs.

Furthermore, refresh of collector sidecar should only be triggered on config change, not every puppet run.

Tags should be their own resource

Tags should be their own resource, so it is possible to add the mysql tag in the mysql profile, apache tag in the apache profile..

The current setup is not compatible with roles and profiles design pattern.

Could not autoload puppet/type/yaml_setting

Puppet 5.1.0

Error: Could not retrieve catalog from remote server: Error 500 on SERVER: Server Error: Evaluation Error: Error while evaluating a Resource Statement, Could not autoload puppet/type/yaml_setting: Could not autoload puppet/provider/yaml_setting/mapped: no such file to load -- puppetx/filemapper at

If this is no longer under active development consider updating the README to reflect that it is currently not in a working state.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.