corey-hammerton / puppet-metricbeat Goto Github PK

fields_under_root is not parsed to the metricbeat.yml
And the fields: hash is parsed as , without the fields: into the metricbeat.yml
As a workaround fields:fields: works.

Hiya, metricbeat 8 is out, and when I assign that major_version, puppet:

Server Error: Evaluation Error: Error while evaluating a Resource Statement, Class[Metricbeat]: parameter 'major_version' expects a match for Enum['5', '6', '7'], got '8'

When the module runs under a puppet agent that has the service set to run as a domain user, it fails to unzip silently using the ShellUI method. I've submitted PR #56 in order to give the option to deal with this by getting the archive resource to perform the unzip instead. This is slower but doesn't suffer from the issue mentioned.

As a
Module maintainer

I would like to
Resolve this deprecation warning about puppetlabs_spec_helper: defaults mock_with to :mocha

In order to
Remain up-to-date with Puppet recommendations and address deprecation messages before the features are not supported

Hello Corey.

My modules
/etc/puppetlabs/code/environments/production/modules
├── darin-zypprepo (v1.0.2)
├── eirc-account (v0.2.0)
├── gentoo-portage (v2.3.0)
├── lwf-remote_file (v1.1.3)
├── pcfens-filebeat (v2.1.0)
├── puppet-archive (v1.3.0)
├── puppet-make (v1.1.0)
├── puppet-r10k (v5.0.2)
├── puppetlabs-accounts (v1.1.0)
├── puppetlabs-apache (v1.3.0)
├── puppetlabs-apt (v2.4.0)
├── puppetlabs-concat (v4.0.0)
├── puppetlabs-gcc (v0.3.0)
├── puppetlabs-git (v0.5.0)
├── puppetlabs-inifile (v1.6.0)
├── puppetlabs-powershell (v2.1.2)
├── puppetlabs-ruby (v0.6.0)
├── puppetlabs-stdlib (v4.16.0)
├── puppetlabs-vcsrepo (v1.5.0)
├── rmueller-cron (v0.2.1)
├── saz-ssh (v2.9.1)
├── saz-sudo (v4.1.0)
├── sensu-sensu (v2.2.0)
├── stahnma-epel (v1.3.0)

puppet module install coreyh-metricbeat --version 0.1.0 --modulepath=/etc/puppetlabs/code/configuration-management/modules --debug

Debug: Runtime environment: puppet_version=5.3.3, ruby_version=2.4.2, run_mode=user, default_encoding=UTF-8
Notice: Preparing to install into /etc/puppetlabs/code/configuration-management/modules ...
Debug: Could not find translation files for eirc-account at /etc/puppetlabs/code/configuration-management/modules/account/locales. Skipping i18n initialization.
Debug: Could not find translation files for puppetlabs-accounts at /etc/puppetlabs/code/configuration-management/modules/accounts/locales. Skipping i18n initialization.
Debug: Could not find translation files for puppetlabs-apache at /etc/puppetlabs/code/configuration-management/modules/apache/locales. Skipping i18n initialization.
Debug: Could not find translation files for puppet-archive at /etc/puppetlabs/code/configuration-management/modules/archive/locales. Skipping i18n initialization.
Debug: Could not find translation files for rmueller-cron at /etc/puppetlabs/code/configuration-management/modules/cron/locales. Skipping i18n initialization.
Debug: Could not find translation files for stahnma-epel at /etc/puppetlabs/code/configuration-management/modules/epel/locales. Skipping i18n initialization.
Debug: Could not find translation files for pcfens-filebeat at /etc/puppetlabs/code/configuration-management/modules/filebeat/locales. Skipping i18n initialization.
Debug: Could not find translation files for puppetlabs-gcc at /etc/puppetlabs/code/configuration-management/modules/gcc/locales. Skipping i18n initialization.
Debug: Could not find translation files for puppetlabs-git at /etc/puppetlabs/code/configuration-management/modules/git/locales. Skipping i18n initialization.
Debug: Could not find translation files for puppetlabs-inifile at /etc/puppetlabs/code/configuration-management/modules/inifile/locales. Skipping i18n initialization.
Debug: Could not find translation files for puppet-make at /etc/puppetlabs/code/configuration-management/modules/make/locales. Skipping i18n initialization.
Debug: Could not find translation files for gentoo-portage at /etc/puppetlabs/code/configuration-management/modules/portage/locales. Skipping i18n initialization.
Debug: Could not find translation files for puppet-r10k at /etc/puppetlabs/code/configuration-management/modules/r10k/locales. Skipping i18n initialization.
Debug: Could not find translation files for lwf-remote_file at /etc/puppetlabs/code/configuration-management/modules/remote_file/locales. Skipping i18n initialization.
Debug: Could not find translation files for puppetlabs-ruby at /etc/puppetlabs/code/configuration-management/modules/ruby/locales. Skipping i18n initialization.
Debug: Could not find translation files for sensu-sensu at /etc/puppetlabs/code/configuration-management/modules/sensu/locales. Skipping i18n initialization.
Debug: Could not find translation files for saz-ssh at /etc/puppetlabs/code/configuration-management/modules/ssh/locales. Skipping i18n initialization.
Debug: Could not find translation files for puppetlabs-stdlib at /etc/puppetlabs/code/configuration-management/modules/stdlib/locales. Skipping i18n initialization.
Debug: Could not find translation files for saz-sudo at /etc/puppetlabs/code/configuration-management/modules/sudo/locales. Skipping i18n initialization.
Debug: Could not find translation files for puppetlabs-vcsrepo at /etc/puppetlabs/code/configuration-management/modules/vcsrepo/locales. Skipping i18n initialization.
Debug: Could not find translation files for zebrastuff at /etc/puppetlabs/code/configuration-management/modules/zebrastuff/locales. Skipping i18n initialization.
Debug: Could not find translation files for darin-zypprepo at /etc/puppetlabs/code/configuration-management/modules/zypprepo/locales. Skipping i18n initialization.
Notice: Downloading from https://forgeapi.puppet.com ...
Debug: Evicting cache entry for environment 'production'
Debug: Caching environment 'production' (ttl = 0 sec)
Debug: Failed to load library 'pe_license' for feature 'pe_license': cannot load such file -- pe_license
Debug: HTTP GET https://forgeapi.puppet.com/v3/releases?module=coreyh-metricbeat&sort_by=version
Debug: Failed to load library 'pe_license' for feature 'pe_license': cannot load such file -- pe_license
Debug: HTTP GET https://forgeapi.puppet.com/v3/releases?module=puppetlabs-stdlib&sort_by=version
Debug: Failed to load library 'pe_license' for feature 'pe_license': cannot load such file -- pe_license
Debug: HTTP GET https://forgeapi.puppet.com/v3/releases?module=puppetlabs-stdlib&sort_by=version&limit=20&offset=20
Debug: Failed to load library 'pe_license' for feature 'pe_license': cannot load such file -- pe_license
Debug: HTTP GET https://forgeapi.puppet.com/v3/releases?module=puppetlabs-stdlib&sort_by=version&limit=20&offset=40
Debug: Failed to load library 'pe_license' for feature 'pe_license': cannot load such file -- pe_license
Debug: HTTP GET https://forgeapi.puppet.com/v3/releases?module=puppetlabs-apt&sort_by=version
Debug: Failed to load library 'pe_license' for feature 'pe_license': cannot load such file -- pe_license
Debug: HTTP GET https://forgeapi.puppet.com/v3/releases?module=puppetlabs-apt&sort_by=version&limit=20&offset=20
Debug: Failed to load library 'pe_license' for feature 'pe_license': cannot load such file -- pe_license
Debug: HTTP GET https://forgeapi.puppet.com/v3/releases?module=darin-zypprepo&sort_by=version
Info: Resolving dependencies ...
Error: Could not install module 'coreyh-metricbeat' (???)
No version of 'coreyh-metricbeat' can satisfy all dependencies
Use puppet module install --ignore-dependencies to install only this module

puppetserver version: 5.1.4
puppet --version
5.3.3
CentOS Linux release 7.4.1708 (Core)

Any help appreciated
Thanks
Regards

Hi, thanks for the great module. Really helped me a lot. Just noting that if you use the module on Windows to enable the system module, metricbeat won't start since the load metricset is enabled in the (metricbeat) module config enabled with this (puppet) module. My current workaround is to pass an empty array to the modules param which means that the (enabled by default) system config remains untouched.

In terms of a feature request it'd be really cool if this (puppet) module could use filebeat's capabilities to enable/disable individual modules (i.e. renaming the files) so that we can use the shipped default module files from a given version/platform of metricbeat.

Also in a really ideal world it'd be great to be able to tweak the module configuration in order to enable/disable certain metricsets. I'd particularly like to be able to dynamically configure the number of processes included in a system's metrics depending on how many CPUs it has (and also on the nature of the workload it runs). So exposing a knob to fiddle with the config would be amazing. This may be something I'll have time to work on so I'll just leave this issue here to remind myself.

At the moment I can't find a way to set the metricbeat.config.modules.path option in metricbeat.yml and the recommended way to use modules now it so have the settings in the modules.d directory. Adding this option would be handy.

Hi

I am busy adding this module to our servers but there is a dependency issue with puppetlabs-apt.

We are using the latest version of puppetlabs-apt but this module requires >=4.0.0 <5.0.0.

Is there a chance this can be updated to include the latest puppetlabs-apt (v6.1.1)?

Could you please push the current version (commit 8bc85dc) to Puppet Forge?
This will allow people to resolve upgrade dependency issues.

Thank you!

When defining the modules parameter with more than one hash in the Tuple, I will get the following errror:

Error: Could not retrieve catalog from remote server: Error 500 on SERVER: Server Error: Evaluation Error: Error while evaluating a Function Call, Class[Metricbeat]: parameter 'modules' expects size to be 1, got 2 at /etc/puppetlabs/code/environments/production/manifests/site.pp:46:1 on node xxx

The problem is the type of the modules parameter in the init.pp file which is Tuple[Hash] and only accepts one hash. To correct this error the type of the modules parameter should be Tuple[Hash, default] which accepts any number of hashes.

To be able to set xpack opentions (for example to configure X-Pack Monitoring) please add an xpack config option.

Hi all,
the current version in puppetforge does not support a valid configtest.

    'Linux': {
      $validate_cmd = $metricbeat::disable_configtest ? {
        true    => undef,
        default => $metricbeat::major_version ? {
          '5'     => '/usr/share/metricbeat/bin/metricbeat -configtest -c %',
          default => '/usr/share/metricbeat/bin/metricbeat test config',
        }
      }

      file{'metricbeat.yml':
        ensure       => $metricbeat::ensure,
        path         => $metricbeat::config_file,
        owner        => 'root',
        group        => 'root',
        mode         => $metricbeat::config_mode,
        content      => inline_template('<%= @metricbeat_config.to_yaml() %>'),
        validate_cmd => $validate_cmd,
      }
    }

This command:
default => '/usr/share/metricbeat/bin/metricbeat test config',
is only working if you are currently in /etc/metricbeat folder.

In the current version of the code the error is already fixed.

Can you update the current version of the module in puppetforge?

Kind regards,
JPylypiw

Please release latest to get the fix-export change into the forge release.

I have metricbeat 6.2.2 installed, and the config test failed for another reason, but I noticed in the output that there was the message: Execution of '/usr/share/metricbeat/bin/metricbeat -configtest -c /etc/metricbeat/metricbeat.yml20180309-22203-kxd5wz' returned 1: Flag --configtest has been deprecated, configtest flag has been deprecated, use test config subcommand

It looks like in 6.0 they deprecated the configtest flag.

So the command used to test the configuration should probably be changed for 6.x+

Hi,

the last update on Puppet Forge is from Feb. 2018.

I'm not sure if the push is supposed to happen automatically (if so, it didn't - possibly because the version string hasn't been changed) but the latest changes - the fixes for #7, #10 and #15) - haven't been pushed to Forge, which means they are still causing conflicts to people trying to upgrade.

It would be great if the Forge module could be upgraded too.

Thanks!

The configuration file metricbeat.yml may contain passwords for certain modules (e.g. kafka, mongodb, mysql, postgresql, vsphere). The file should therefore not have mode '0644' but use the file mode '0600' instead.

Hello Corey,

first thank you very much for your effort to develop this nice module. We have just set up our elastic stack and are very happy to have found a good metricbeat module. However, since elastic just released version 6 we would like to update to this version now, before going to production. So would it be possible to add support for version 6?

Best wishes,
Sarah

Module attempts to run the command"
metricbeat -configtest -c /etc/metricbeat/metricbeat.yml.....

whereas metricbeat 7.9.1-1 requires:
metricbeat test config -c /etc/metricbeat/metricbeat.yml...

I am not 100% on which version the config validation test changed but
i believe https://github.com/corey-hammerton/puppet-metricbeat/blob/master/spec/classes/metricbeat_spec.rb

validate_cmd: '/usr/share/metricbeat/bin/metricbeat -configtest -c %',

for 7.6.x for sure should be

validate_cmd: '/usr/share/metricbeat/bin/metricbeat test config -c %',

Because of the updated metadata.json dependencies after the 0.4.1 release we would need a new release to the forge.

README.md says:
'disable_config_test'
Code has:
'disable_configtest'

After updating the module to 0.3.0 I get the following errors:

Error: Execution of '/usr/share/metricbeat/bin/metricbeat test config' returned 1: Exiting: error loading config file: stat metricbeat.yml: no such file or directory
Error: /Stage[main]/Metricbeat::Config/File[metricbeat.yml]/content: change from '{md5}d43aad4e1b67c81b67a882569854df1d' to '{md5}2a926367d58341c322fcef65e4a5733e' failed: Execution of '/usr/share/metricbeat/bin/metricbeat test config' returned 1: Exiting: error loading config file: stat metricbeat.yml: no such file or directory

I can reproduce the error on the commandline:

# /usr/share/metricbeat/bin/metricbeat test config
Exiting: error loading config file: stat metricbeat.yml: no such file or directory

It seems the config file is only found if the validation command is either executed in the correct directory or includes the correct directory.

This works:

# cd /etc/metricbeat
# /usr/share/metricbeat/bin/metricbeat test config
Config OK

This also works:

# /usr/share/metricbeat/bin/metricbeat --path.config /etc/metricbeat test config
Config OK

An exported resource is causing duplicate resource.

Error: Could not retrieve catalog from remote server: Error 500 on SERVER: Server Error: A duplicate resource was found while collecting exported resources, with the type and title Metricbeat::Modulesd[system]

An exported resource declaration specifies a desired state for a resource, and publishes the resource for use by other nodes. It does not manage the resource on the target system. Any node, including the node that exports it, can collect the exported resource and manage its own copy of it.
See

I think you should be using create_resources
Converts a hash into a set of resources and adds them to the catalog.
See

The documentation says, "The metricbeat installs the metricbeat operating system and service collector maintained by elastic."

But, I don't want our hosts to install a package repo and go out to the internet for the metricbeat RPM; is it possible to specify the location of the RPM in a URL?

Module dependencies in metadata.json are outdated and need to be updated.

Would you be interested in a patch to support metricbeat on windows nodes? We have several windows servers that I would like to have send data into logstash.