voxpupuli / puppet-elastic_stack Goto Github PK

License: Apache License 2.0

Ruby 65.22% Puppet 34.78%

hacktoberfest debian-puppet-module linux-puppet-module opensuse-puppet-module puppet redhat-puppet-module sles-puppet-module ubuntu-puppet-module

puppet-elastic_stack's Introduction

elastic_stack

This module contains shared code for various modules to manage Elastic products, like puppet-elasticsearch, puppet-logstash etc.

Version 8 and newer of this module are released by Vox Pupuli. They now follow semantic versioning. Previously the module was maintained by Elastic.

Setting up the Elastic package repository

This module can configure package repositories for Elastic Stack components.

Example:

include elastic_stack::repo

You may wish to specify a major version, since each has its own repository:

class { 'elastic_stack::repo':
  version => 5,
}

To access prerelease versions, such as release candidates, set prerelease to true.

class { 'elastic_stack::repo':
  version    => 6,
  prerelease => true,
}

To access the repository for OSS-only packages, set oss to true.

class { 'elastic_stack::repo':
  oss => true,
}

To use a custom package repository, set base_repo_url, like this:

class { 'elastic_stack::repo':
  base_repo_url => 'https://mymirror.example.org/elastic-artifacts/packages',
}

Transfer Notice

This module was originally authored by Elastic. The maintainer preferred that Vox Pupuli take ownership of the module for future improvement and maintenance. Existing pull requests and issues were transferred over, please fork and continue to contribute here instead of Elastic.

puppet-elastic_stack's People

Contributors

Stargazers

Watchers

puppet-elastic_stack's Issues

What if I want to change apt settings elsewhere?

I have to disable the include apt here since it conflicts with the other settings in my environment.

diff --git a/manifests/repo.pp b/manifests/repo.pp
index 774291c..c035860 100644
--- a/manifests/repo.pp
+++ b/manifests/repo.pp
@@ -66,7 +66,7 @@ class elastic_stack::repo (
 
   case $::osfamily {
     'Debian': {
-      include apt
+      # include apt
 
       apt::source { 'elastic':
         ensure   => 'present',

if ! defined (Class['apt']) {include apt} didn't help either, since this one comes before my own settings.

apt::update ordering creates dependency cycles

This statement Class['apt::update'] -> Package <| |> [1] inserts a dependency in all packages defined not only by this module but for every other .pp file.

In many cases we want to install some packages before running the apt::update but this module makes in impossible. One example would be the package 'apt-transport-https'.

My suggestion is to only add the dependency to the specific packages the modules depends on instead of using the "wildcard".

[1] https://github.com/elastic/puppet-elastic-stack/blob/master/manifests/repo.pp#L34

Can't install elasticsearch, Error autoloading yumrepo Type

Applying class 'elasticsearch' without specifying version couldn't add the RHEL repository giving this error

Warning: /etc/puppetlabs/code/environments/elasticprod/modules/elasticsearch/hiera.yaml: Use of 'hiera.yaml' version 4 is deprecated. It should be converted to version 5
   (file: /etc/puppetlabs/code/environments/elasticprod/modules/elasticsearch/hiera.yaml)
Warning: Defining "data_provider": "hiera" in metadata.json is deprecated.
   (file: /etc/puppetlabs/code/environments/elasticprod/modules/elasticsearch/metadata.json)
Error: Could not autoload puppet/type/yumrepo: undefined method `sensitive' for Puppet::Type::Yumrepo::Proxy_password:Class
Error: Evaluation Error: Error while evaluating a Resource Statement, Could not autoload puppet/type/yumrepo: undefined method `sensitive' for Puppet::Type::Yumrepo::Proxy_password:Class (file: /etc/puppetlabs/code/environments/elasticprod/modules/elastic_stack/manifests/repo.pp, line: 89, column: 7) on node gules.domain.com

Modules Installed:
── elastic-elastic_stack (v6.2.4)
├── elastic-elasticsearch (v6.3.3)
├── puppet-archive (v3.2.0)
├── puppet-yum (v3.1.0)
├── puppetlabs-apt (v6.0.0)
├── puppetlabs-concat (v4.2.1)
├── puppetlabs-java (v3.1.0)
├── puppetlabs-stdlib (v4.25.1)
├── puppetlabs-yumrepo_core (v1.0.1)
└── richardc-datacat (v0.6.2)

release consistency

So,

I noticed on forge you have released 6.1.0 however there is no corresponding 6.1.0 tag/release in git. The elasticsearch module requires 6.1 - can you update one/the other/both for consistency?

Module doesn't install and bugs puppet.

After running this:

exec { 'puppet_module_install':
  command => "puppet module install elastic/elastic_stack",
  unless  => "puppet module list | grep 'elastic/elastic_stack'",
  path    => ['/bin', '/usr/bin', '/opt/puppetlabs/bin']
}

class { 'elastic_stack::repo':
  prerelease => false,
}
include elastic_stack::repo

I get:

Error: Evaluation Error: Error while evaluating a Resource Statement, Could not find declared class elastic_stack::repo (file: /......../Packs/App_ElasticStack.pp, line: 9, column: 1) on node elastistack

I then run another pack (not at the same time, I am running them one at a time) and get this error:

Error: Evaluation Error: Error while evaluating a Function Call, Could not find class ::mtulio::ssh for elastistack (file: /......../Packs/Config_SSH-Server_AllowRoot.pp, line: 7, column: 1) on node elastistack

Note that this file (SSH-Server) makes no reference to elastistack:

exec { 'puppet_module_install':
  command => "puppet module install mtulio/ssh",
  unless  => "puppet module list | grep 'mtulio/ssh'",
  path    => ['/bin', '/usr/bin', '/opt/puppetlabs/bin']
}

include mtulio::ssh
class { '::ssh::sshd_config': 
  permitrootlogin   => 'yes'
}

The puppet module list command does NOT list the elastistack command as being installed...

Manually running puppet module install elastic/elastic_stack command installs the module, but does not resolve the issue with other packs.

Uninstalling it also does not resolve the issue.

Puppet is effectively killed on this box now. Ideas?

This was a clean install (with a few other puppet packs installed) just to build out the elastic stack process.

Allow $version to set an specific release

First of all, thank you for provide this functionality (although it has become now a requirement of the ES Module since version 6.3.0).

Previously in the ES Module we were able pin an specific release we wanted, such as for example:

elasticsearch::repo_version: '1.7.6'

now the suggestion from voxpupuli/puppet-elasticsearch#959, is to do something such as this:

elastic_stack::repo::version: 1.7.6

and you might expect this to work, but since this is not a valid Integer it will fail as you may expect:

Error: Evaluation Error: Error while evaluating a Function Call, Class[Elastic_stack::Repo]: parameter 'version' expects an Integer value, got Float

Can you please modify this parameter so that we can keep pinning specific releases (yes, it's an old release but hey, world is not perfect) without breaking the compatibility?

For testing purposes, this is not specified on the README file, It seems like ES 1.x version is not supported either. On my hiera configuration I have set the following settings:

elasticsearch::version: '1.7.6'
elastic_stack::repo::version: 1

This is what you get:

Notice: /Stage[main]/Apt::Update/Exec[apt_update]/returns: W: The repository 'https://packages.elastic.co/elasticsearch/1.x/debian stable Release' does not have a Release file.
Notice: /Stage[main]/Apt::Update/Exec[apt_update]/returns: E: Failed to fetch https://packages.elastic.co/elasticsearch/1.x/debian/dists/stable/main/binary-amd64/Packages  404  Not Found
Notice: /Stage[main]/Apt::Update/Exec[apt_update]/returns: E: Some index files failed to download. They have been ignored, or old ones used instead.
Error: /Stage[main]/Apt::Update/Exec[apt_update]: Failed to call refresh: '/usr/bin/apt-get update' returned 100 instead of one of [0]
Error: /Stage[main]/Apt::Update/Exec[apt_update]: '/usr/bin/apt-get update' returned 100 instead of one of [0]

This is the resulting file with the repository /etc/apt/sources.list.d/elastic.list:

deb https://packages.elastic.co/elasticsearch/1.x/debian stable main

PS: I'm running this under stack Ubuntu Xenial and Puppet module 6.3.0

Support stdlib 5

Puppet have released stdlib version 5. Ensure this module can use it and bless it in metadata.json.

repo_url parameter

Allow for a repo_url parameter to be set and if defined then override the $_repo_url variable.
This allows for alternative/private mirrors.

Priority parameter should be undef by default

Currently the priority parameter is set to 10 by default, which is one of the lowest options for Debian systems and therefore breaks the installation of elasticsearch installation if it is also available in the main repositories of e.g. Ubuntu 16.04.

The default should be undef.

It will submit a PR.

r10k continuously updating elastic_stack

Hi folk! I have been tracking a bizarre issue with the elastic_stack module for a bit now. I run r10k every 5 minutes currently to update our environments, but I'd been noticing that elastic_stack::repo keeps not being located on catalog runs. Turns out it's because every 5 minutes the module is deleted and reinstalled via r10k. This does not happen with any other modules, but I also have no idea at all why it's happening. I wanted to bounce it off of y'all to see if you have any ideas. I looked around your code and I don't see a single thing that explains it. I have quite a few modules enabled (somewhere between 200 and 300), but here is a snippit with the elastic ones:

mod 'elastic/kibana', '6.3.0'
mod 'elastic/elastic_stack', '6.2.4'
mod 'elastic/elasticsearch', '6.3.3'
mod 'elastic/logstash', '6.1.4'

elastic_stack is the only one that continuously updates despite pointing at a specific version. For reference this is:

Puppet 5.5.4
Puppet Server 5.3.4
R10K 2.6.4

I switched to pointing directly at your git repository and that has resolved the issue for now.

apt weak_ssl paramerer --- workaround for apt key error: "certificate verify failed (unable to get local issuer certificate)"?

Affected Puppet, Ruby, OS and module versions/distributions

Puppet: 6
Ruby: -
Distribution: Debian 11.3 (Bullseye)
Module version: v8.0.0

How to reproduce (e.g Puppet code you use)

using elastic_stack to manage repo at fresh debian bullseye installation:

	class { 'elastic_stack::repo':
  		version => 7,
	}

What are you seeing

Error: certificate verify failed (unable to get local issuer certificate)

What behaviour did you expect instead

like to have "weak_ssl" option for apt::source module as workaround:

Boolean           $apt_weak_ssl  = false,

.. and then:

  case $facts['os']['family'] {
    'Debian': {
      include apt

      apt::source { 'elastic':
        ensure   => 'present',
        comment  => $description,
        location => $base_url,
        release  => 'stable',
        repos    => 'main',
        key      => {
          'id'     => $key_id,
          'source'  => $key_source,
          'weak_ssl'  => $apt_weak_ssl,
        },
        include  => {
          'deb' => true,
          'src' => false,
        },
        pin      => $priority,
      }
    }

Output log

Error: Could not set 'present' on ensure: SSL_connect returned=1 errno=0 state=error: certificate verify failed (unable to get local issuer certificate) (file: /etc/puppetlabs/code/environments/production/modules/apt/manifests/key.pp, line: 55)
Error: Could not set 'present' on ensure: SSL_connect returned=1 errno=0 state=error: certificate verify failed (unable to get local issuer certificate) (file: /etc/puppetlabs/code/environments/production/modules/apt/manifests/key.pp, line: 55)
Wrapped exception:
SSL_connect returned=1 errno=0 state=error: certificate verify failed (unable to get local issuer certificate)
Error: /Stage[first]/Elastic_stack::Repo/Apt::Source[elastic]/Apt::Key[Add key: 46095ACC8548582C1A2699A9D27D666CD88E42B4 from Apt::Source elastic]/Apt_key[Add key: 46095ACC8548582C1A2699A9D27D666CD88E42B4 from Apt::Source elastic]/ensure: change from 'absent' to 'present' failed: Could not set 'present' on ensure: SSL_connect returned=1 errno=0 state=error: certificate verify failed (unable to get local issuer certificate) (file: /etc/puppetlabs/code/environments/production/modules/apt/manifests/key.pp, line: 55)

Any additional information you'd like to impart

Drop yum dependency

This module requires the puppet/yum module however I can't see it used anywhere, unlike the apt module.

update Project URL on Puppet Forge and changelog

Could you please update the Project URL on https://forge.puppet.com/elastic/elastic_stack which currently points to https://github.com/jarpy/puppet-elastic-stack and it doesn't exist?
Could you also fill the changelog as is very useful to see how this project proceeds, please?

kibana packages not found in oss repository

Yum can't find the kibana packages in the https://artifacts.elastic.co/packages/oss-6.x/yum repository.

Yum can find the kibana packages in https://artifacts.elastic.co/packages/6.x/yum repository.

So when oss is set to true, kibana-oss package installation fails.

Current release tarball 6.2.4 erroneously contains local `.git` directory

The release tarball for 6.2.4 contains someone's local .git directory:

$ tar tvf elastic-elastic_stack-6.2.4.tar.gz 
drwxrwxr-x 1000/1000         0 2018-09-17 01:12 elastic-elastic_stack-6.2.4
-rw-rw-r-- 1000/1000       177 2018-05-30 22:10 elastic-elastic_stack-6.2.4/.fixtures.yml
drwxrwxr-x 1000/1000         0 2018-09-17 01:12 elastic-elastic_stack-6.2.4/.git
-rw-rw-r-- 1000/1000       361 2018-09-17 01:09 elastic-elastic_stack-6.2.4/.git/COMMIT_EDITMSG
-rw-rw-r-- 1000/1000       215 2018-09-17 01:08 elastic-elastic_stack-6.2.4/.git/FETCH_HEAD
-rw-rw-r-- 1000/1000        23 2018-07-17 02:37 elastic-elastic_stack-6.2.4/.git/HEAD
-rw-rw-r-- 1000/1000        41 2018-09-17 01:11 elastic-elastic_stack-6.2.4/.git/ORIG_HEAD
drwxrwxr-x 1000/1000         0 2018-09-17 01:12 elastic-elastic_stack-6.2.4/.git/branches
-rw-rw-r-- 1000/1000       474 2018-08-28 01:51 elastic-elastic_stack-6.2.4/.git/config
-rw-rw-r-- 1000/1000        73 2018-05-30 22:10 elastic-elastic_stack-6.2.4/.git/description
drwxrwxr-x 1000/1000         0 2018-09-17 01:12 elastic-elastic_stack-6.2.4/.git/hooks
-rwxrwxr-x 1000/1000       478 2018-05-30 22:10 elastic-elastic_stack-6.2.4/.git/hooks/applypatch-msg.sample
-rwxrwxr-x 1000/1000       896 2018-05-30 22:10 elastic-elastic_stack-6.2.4/.git/hooks/commit-msg.sample
-rwxrwxr-x 1000/1000      3327 2018-05-30 22:10 elastic-elastic_stack-6.2.4/.git/hooks/fsmonitor-watchman.sample
-rwxrwxr-x 1000/1000       189 2018-05-30 22:10 elastic-elastic_stack-6.2.4/.git/hooks/post-update.sample
-rwxrwxr-x 1000/1000       424 2018-05-30 22:10 elastic-elastic_stack-6.2.4/.git/hooks/pre-applypatch.sample
-rwxrwxr-x 1000/1000      1642 2018-05-30 22:10 elastic-elastic_stack-6.2.4/.git/hooks/pre-commit.sample
-rwxrwxr-x 1000/1000      1348 2018-05-30 22:10 elastic-elastic_stack-6.2.4/.git/hooks/pre-push.sample
-rwxrwxr-x 1000/1000      4898 2018-05-30 22:10 elastic-elastic_stack-6.2.4/.git/hooks/pre-rebase.sample
-rwxrwxr-x 1000/1000       544 2018-05-30 22:10 elastic-elastic_stack-6.2.4/.git/hooks/pre-receive.sample
-rwxrwxr-x 1000/1000      1492 2018-05-30 22:10 elastic-elastic_stack-6.2.4/.git/hooks/prepare-commit-msg.sample
-rwxrwxr-x 1000/1000      3610 2018-05-30 22:10 elastic-elastic_stack-6.2.4/.git/hooks/update.sample
-rw-rw-r-- 1000/1000      2001 2018-09-17 01:11 elastic-elastic_stack-6.2.4/.git/index
drwxrwxr-x 1000/1000         0 2018-09-17 01:12 elastic-elastic_stack-6.2.4/.git/info
-rw-rw-r-- 1000/1000       240 2018-05-30 22:10 elastic-elastic_stack-6.2.4/.git/info/exclude
drwxrwxr-x 1000/1000         0 2018-09-17 01:12 elastic-elastic_stack-6.2.4/.git/logs
-rw-rw-r-- 1000/1000      5561 2018-09-17 01:11 elastic-elastic_stack-6.2.4/.git/logs/HEAD

This confuses r10k into endlessly and needlessly redeploying the module, especially when running r10k deploy environment -p. Relates to #19. I believe a new release of elastic_stack should be cut that does not contain a .git directory in order to fix this issue.

Make key_source and key_id configurable

In case of a server not able to access the internet it is required to point the module to an internal mirror of the gpg key file, as it already is possble with the base_repo_url for packages.

Do not use cross module variables

There are way to many mentions of "elasticsearch". As this module should be a provider of the elastic repositories, there should any mentions (or even cross module variable access) to elasticsearch.

I will submit a PR.