Git Product home page Git Product logo

puppet-hiera's Introduction

Hiera module for Puppet

Build Status Code Coverage Puppet Forge Puppet Forge - downloads Puppet Forge - endorsement Puppet Forge - scores

Table of Contents

  1. Overview
  2. Module Description - What the module does and why it is useful
  3. Setup - The basics of getting started with hiera
  4. Usage - Configuration options and additional functionality
  5. Reference - An under-the-hood peek at what the module is doing and how
  6. Limitations - OS compatibility, etc.
  7. Development - Guide for contributing to the module

Module Description

This module configures Hiera for Puppet.

Setup

What hiera affects

  • Hiera yaml file
  • Hiera datadir
  • hiera-eyaml package
  • keys/ directory for eyaml
  • /etc/hiera.yaml for symlink

Setup requirements

To use the eyaml backend with the modern puppetserver, you will need the puppetlabs-puppetserver_gem module.

Beginning with hiera

Declaring the class with a given hierarchy is a pretty good starting point:

This class will write out a hiera.yaml file in either /etc/puppetlabs/puppet/hiera.yaml or /etc/puppet/hiera.yaml (depending on if the node is running Puppet Enterprise or not).

class { 'hiera':
  hierarchy => [
    '%{environment}/%{calling_class}',
    '%{environment}',
    'common',
  ],
}

For Hiera version 5

class { 'hiera':
          hiera_version   =>  '5',
          hiera5_defaults =>  {"datadir" => "data", "data_hash" => "yaml_data"},
          hierarchy       =>  [
                                {"name" =>  "Virtual yaml", "path"  =>  "virtual/%{virtual}.yaml"},
                                {"name" =>  "Nodes yaml", "paths" =>  ['nodes/%{trusted.certname}.yaml', 'nodes/%{osfamily}.yaml']},
                                {"name" =>  "Default yaml file", "path" =>  "common.yaml"},
                              ],
}

** Note: For Hiera version 5 when calling the class, please remember to pass '5' to 'hiera_version' as in the example above. ** ** Also please note that 'hierarchy' is an array of hash in version 5. **

The resulting output in /etc/puppet/hiera.yaml:

---
:backends:
  - yaml
:logger: console
:hierarchy:
  - "%{environment}/%{calling_class}"
  - "%{environment}"
  - common

:yaml:
   :datadir: /etc/puppet/hieradata

Resulting output for Hiera 5

# hiera.yaml Managed by Puppet
version: 5
defaults:
  datadir: data
  data_hash: yaml_data
hierarchy:

  - name: "Virtual yaml"
    path: "virtual/%{virtual}.yaml"

  - name: "Nodes yaml"
    paths:
      - "nodes/%{trusted.certname}.yaml"
      - "nodes/%{osfamily}.yaml"

  - name: "Default yaml file"
    path: "common.yaml"

Usage

Reference

This module will also allow you to configure different options for logger and merge_behavior. The default behavior is to set logger to console and merge behavior to native.

For details and valid options see Configuring Hiera.

class { 'hiera':
  hierarchy      => [
    '%{environment}/%{calling_class}',
    '%{environment}',
    'common',
  ],
  logger         => 'console',
  merge_behavior => 'deeper'
}

For Hiera version 5 please see the example above in beginning with Hiera.

The resulting output in /etc/puppet/hiera.yaml:

---
:backends:
  - yaml
:logger: console
:hierarchy:
  - "%{environment}/%{calling_class}"
  - "%{environment}"
  - common

:yaml:
   :datadir: /etc/puppet/hieradata

:merge_behavior: deeper

Hiera-Eyaml-GPG

The default PKCS#7 encryption scheme used by hiera-eyaml is perfect if only simple encryption and decryption is needed.

However, if you are in a sizable team it helps to encrypt and decrypt data with multiple keys. This means that each team member can hold their own private key and so can the puppetmaster. Equally, each puppet master can have their own key if desired and when you need to rotate keys for either users or puppet masters, re-encrypting your files and changing the key everywhere does not need to be done in lockstep.

Requirements

Note: This module will create a /gpg sub-directory in the $keysdir.

  1. The GPG keyring must be passphraseless on the on the PuppetServer(Master).
  2. The GPG keyring must live in the /gpg sub-directory in the $keysdir.
  3. The GPG keyring must be owned by the Puppet user. ex: pe-puppet

GPG Keyring Creation Tips

RNG-TOOLS

When generating a GPG keyring the system requires a good amount of entropy. To help generate entropy to speed up the process then rng-tools package on RHEL based systems or equivilent can be used. Note: Update the /etc/sysconfig/rngd or equivilent file to set the EXTRAOPTIONS to EXTRAOPTIONS="-r /dev/urandom -o /dev/random -t 5"

Keyring Generation

Below is a sample GPG answers file that will assist in generating a passphraseless key

cat << EOF >> /tmp/gpg_answers
%echo Generating a Puppet Hiera GPG Key
Key-Type: RSA
Key-Length: 4096
Subkey-Type: ELG-E
Subkey-Length: 4096
Name-Real: Hiera Data
Name-Comment: Hiera Data Encryption
Name-Email: puppet@$(hostname -d)
Expire-Date: 0
%no-ask-passphrase
# Do a commit here, so that we can later print "done" :-)
# %commit
# %echo done
EOF

You can then use the GPG answer file to generate your keyring within the /gpg sub-directory in the $keysdir

gpg --batch --homedir /etc/puppetlabs/code-staging/keys/gpg --gen-key /tmp/gpg_answers

Usage

class { 'hiera':
  hierarchy            => [
    'nodes/%{clientcert}',
    'locations/%{location}',
    'environments/%{applicationtier}',
    'common',
  ],
  eyaml                => true,
  eyaml_gpg            => true,
  eyaml_gpg_recipients => '[email protected],[email protected],[email protected]',
}

The resulting output in /etc/puppet/hiera.yaml:

---
:backends:
  - eyaml
  - yaml
:logger: console
:hierarchy:
  - "nodes/%{clientcert}"
  - "locations/%{location}"
  - "environments/%{applicationtier}"
  - common

:yaml:
   :datadir: /etc/puppet/hieradata


:eyaml:
   :datadir: /etc/puppet/hieradata
   :pkcs7_private_key: /etc/puppet/keys/private_key.pkcs7.pem
   :pkcs7_public_key:  /etc/puppet/keys/public_key.pkcs7.pem
   :encrypt_method: "gpg"
   :gpg_gnupghome: "/etc/puppet/keys/gpg"
   :gpg_recipients: "[email protected],[email protected],[email protected]"

Classes

Public Classes

  • hiera: Main class to configure hiera

Private Classes

  • hiera::params: Handles variable conditionals
  • hiera::eyaml: Handles eyaml configuration

Limitations

The eyaml_version parameter does not currently modify the eyaml version of the command-line gem on pe-puppetserver.

Development

Pull requests on github! If someone wrote spec tests, that would be awesome.

puppet-hiera's People

Contributors

alexjfisher avatar bastelfreak avatar dan33l avatar dgoodlad avatar dhoppe avatar ekohl avatar elyscape avatar emning avatar ghoneycutt avatar hunner avatar igalic avatar juniorsysadmin avatar jyaworski avatar kenyon avatar logicminds avatar m4rx0 avatar maxadamo avatar mcmartins avatar naeem98 avatar pjfoley avatar razorsedge avatar reidmv avatar rnelson0 avatar sharpie avatar smbambling avatar smortex avatar tampakrap avatar vchepkov avatar vstone avatar zilchms avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

puppet-hiera's Issues

Hiera::params class attempts to set file ownership to `puppet` under Puppet Enterprise 2015.2

Overview

When running under Puppet Enterprise 2015.2 the hiera::params class attempts to change ownership to the puppet user which does not exist. This causes a cascading failure and results in a failed puppet agent run.

Root cause

The hiera::params module is attempting to reference the is_pe fact but on Puppet Enterprise 2015.2 this always returns false (see https://tickets.puppetlabs.com/browse/MODULES-2279).

Suggested fix

The defaults computation in the hiera::params class needs to be reworked

Workaround

Override the owner parameter of the hiera class until a proper workaround can be developed.

Error output

[root@puppet01 vagrant]# puppet agent -t 
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Loading facts
Info: Caching catalog for puppet01.ops.rms.nsw.gov.au
Info: Applying configuration version '1438159437'
Error: Could not find user puppet
Error: /Stage[main]/Hiera::Eyaml/File[/etc/puppetlabs/code/keys]/owner: change from root to puppet failed: Could not find user puppet
Error: Could not find group puppet
Error: /Stage[main]/Hiera::Eyaml/File[/etc/puppetlabs/code/keys]/group: change from root to puppet failed: Could not find group puppet
Notice: /Stage[main]/Hiera::Eyaml/Exec[createkeys]: Dependency File[/etc/puppetlabs/code/keys] has failures: true
Warning: /Stage[main]/Hiera::Eyaml/Exec[createkeys]: Skipping because of failed dependencies
Notice: /Stage[main]/Hiera::Eyaml/File[/etc/puppetlabs/code/keys/private_key.pkcs7.pem]: Dependency File[/etc/puppetlabs/code/keys] has failures: true
Warning: /Stage[main]/Hiera::Eyaml/File[/etc/puppetlabs/code/keys/private_key.pkcs7.pem]: Skipping because of failed dependencies
Notice: /Stage[main]/Hiera::Eyaml/File[/etc/puppetlabs/code/keys/public_key.pkcs7.pem]: Dependency File[/etc/puppetlabs/code/keys] has failures: true
Warning: /Stage[main]/Hiera::Eyaml/File[/etc/puppetlabs/code/keys/public_key.pkcs7.pem]: Skipping because of failed dependencies
Notice: /Stage[main]/Hiera/File[/etc/puppetlabs/code/hiera.yaml]: Dependency File[/etc/puppetlabs/code/keys] has failures: true
Warning: /Stage[main]/Hiera/File[/etc/puppetlabs/code/hiera.yaml]: Skipping because of failed dependencies
Notice: /Stage[main]/Hiera/File[/etc/hiera.yaml]: Dependency File[/etc/puppetlabs/code/keys] has failures: true
Warning: /Stage[main]/Hiera/File[/etc/hiera.yaml]: Skipping because of failed dependencies
Notice: /Stage[main]/Puppet_enterprise::Master::Puppetserver/Service[pe-puppetserver]: Dependency File[/etc/puppetlabs/code/keys] has failures: true
Warning: /Stage[main]/Puppet_enterprise::Master::Puppetserver/Service[pe-puppetserver]: Skipping because of failed dependencies
Notice: Applied catalog in 21.13 seconds

ReadMe incorrect with regards master_service

Current readme indicates that the one needs to set the hiera::parameters::masterservice parameter in order to enable restart of puppetserver.

Based on the code this is incorrect. The parameter is master_service (note underscore) and the settings is hiera::master_service

Generate a new release

Hi. Can you cut a new release? I could really use the optional key creation feature merged in recently.

hiera-eyaml-gpg and other gems never installed to puppetserver

Right now a fresh install of RedHat 6 Puppetserver open source puppet master using this module results in Puppet thinking hiera-eyaml-gpg, ruby_gpg and deep_merge are all installed to puppetserver when they are not. Somehow hiera-eyaml managed to get installed though. This is the reason I think:

# puppetserver gem list

*** LOCAL GEMS ***

hiera-eyaml (2.1.0)
highline (1.6.21)
jar-dependencies (0.1.13)
jruby-openssl (0.9.7 java)
json (1.8.0 java)
rake (10.1.0)
rdoc (4.1.2)
trollop (2.1.2)

# find /var/lib/puppet/jruby-gems/gems/ -maxdepth 1 -type d -regextype posix-egrep -regex '.*/hiera-eyaml-gpg-([0-9].?)+' ; echo $?
0
# find /var/lib/puppet/jruby-gems/gems/ -maxdepth 1 -type d -regextype posix-egrep -regex '.*/hiera-eyaml-([0-9].?)+' ; echo $?
/var/lib/puppet/jruby-gems/gems/hiera-eyaml-2.1.0
0

The exit code for find is always 0, regardless of the regex finding a match. So far the only solution seems to be piping the output to egrep and letting egrep's exit code be used by the Exec resource.

Problems configuring eyaml-gpg without eyaml

Affected Puppet, Ruby, OS and module versions/distributions

  • Puppet: 4.7.x
  • Ruby: 2.1.9 (AIO)
  • Distribution: CentOS
  • Module version: 2.2.0

How to reproduce (e.g Puppet code you use)

class { '::hiera':
  backends             => ['yaml'],
  hierarchy            => ['fqdn/%{::trusted.certname}',
                           'operatingsystem/%{::operatingsystem}/%{::operatingsystemmajrelease}',
                           'operatingsystem/%{::operatingsystem}/common',
                           'common' ],
  hiera_yaml           => "${::settings::confdir}/hiera.yaml",
  eyaml                => false,
  eyaml_gpg            => true,
}

What are you seeing

The hiera.yaml file isn't configured with eyaml as a backend. I think this code in init.pp isn't getting triggered?

  if ( $eyaml_gpg ) or ( $eyaml ) {

    $eyaml_real_datadir = empty($eyaml_datadir) ? {
      false => $eyaml_datadir,
      true  => $datadir,
    }
    # the requested_backends has a side affect in that the eyaml will always be
    # last for backend lookups.  This can be fixed by specifing the order in
    # the backends parameter ie. ['yaml', 'eyaml', 'redis']
    $requested_backends = unique(concat($backends, 'eyaml'))
  } 

What behaviour did you expect instead

eyaml correctly configured in hiera.yaml as a backend by specifying only eyaml_gpg => true.

Output log

N/A

Any additional information you'd like to impart

N/A

hiera::eyaml breaks ACLs on Ext4 file systems

The eyamp.pp class defined the private key permissions to be 0600. This value is hard coded.

However when working with a team we use ACL on ext4 using a mask of r--. While this doesn't open up the permissions any more than needed it unfortunately means that via "ls -l" the file looks like it has 0640 permissions when actually is has -rw-r-----+ and when looking at the ACL 'group' and 'other' are both set to ---, viz:

0 codo:~# getfacl /etc/puppet/keys/private_key.pkcs7.pem
getfacl: Removing leading '/' from absolute path names
\# file: etc/puppet/keys/private_key.pkcs7.pem
\# owner: puppet
\# group: puppet
user::rw-
user:someuser:r-x #effective:r--
group::---
mask::r--
other::---

0 codo:~# ls -l /etc/puppet/keys/private_key.pkcs7.pem
-rw-r-----+ 1 puppet puppet 1675 Aug 4 14:57 /etc/puppet/keys/private_key.pkcs7.pem
0 codo:~#

Of course eyamp.pp "fixes" this by setting it back to "0600" and damages the ACL while doing so, viz:

0 codo:~# puppet_run.sh
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Loading facts
Info: Loading facts
Info: Caching catalog for codo.xxx.xxx
Info: Applying configuration version '[test] 6df1c6c Change Phoenix dnsmasq servers. Both are now link local.'
Notice: /Stage[main]/Hiera::Eyaml/File[/etc/puppet/keys/private_key.pkcs7.pem]/mode: mode changed '0640' to '0600'
Notice: Finished catalog run in 58.87 seconds
0 codo:~# getfacl /etc/puppet/keys/private_key.pkcs7.pem
getfacl: Removing leading '/' from absolute path names
\# file: etc/puppet/keys/private_key.pkcs7.pem
\# owner: puppet
\# group: puppet
user::rw-
user:someuser:r-x #effective:---
group::---
mask::---
other::---

0 codo:~#

Please can the permissions of the private key be made optional so we can pass a value of "0640" into it?
We already have puppet classes that can handle the ACL part.

Affected Puppet, Ruby, OS and module versions/distributions

  • Puppet: 3.8.x
  • Ruby: ruby 1.8.7 (2011-06-30 patchlevel 352) [x86_64-linux]
  • Distribution: Red Hat Enterprise Linux Server release 6.5
  • Module version: 2.1.0

eyaml datadir

I used the datadir => '/var/lib/hiera' parameter, and the eyaml => true parameter. I was expecting all the datadirs to be the same, but in the hiera.yaml it has /etc/puppet/hieradata as the eyaml datadir:

:yaml:
:datadir: /var/lib/hiera

:eyaml:
:datadir: /etc/puppet/hieradata

Is there a way to specify the eyaml datadir, and/or is keeping eyaml intermixed with normal yaml good/bad?

restart required when changing hiera.yaml

According to the hiera documentation the puppet master needs to be restarted for the changes to take effect. I think it's somewhat useless to manage hiera.yaml without taking care that the changes actually take effect.

Well, currently I don't see an obvious way to achieve this without interferring with other modules (i.e. theforeman-puppet, a module to manage puppet master/agent).

If someone comes up with an idea I'd be glad to implement it (either in puppet-hiera or theforeman-puppet).

Permissions issue on hieradata directory w/ vagrant

When hieradata is located within the puppet-control repo permissions cannot be set on the shared vagrant folder within the VM.

[root@master modules]# puppet agent -t
Info: Using configured environment 'production'
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Loading facts
Info: Caching catalog for master.vagrant.vm
Info: Applying configuration version '1454621749'
Error: Failed to set owner to '494': Operation not permitted @ lchown_internal - /vagrant/hieradata
Error: /Stage[main]/Hiera/File[/vagrant/hieradata]/owner: change from vagrant to pe-puppet failed: Failed to set owner to '494': Operation not permitted @ lchown_internal - /vagrant/hieradata
Error: Failed to set group to '495': Operation not permitted @ lchown_internal - /vagrant/hieradata
Error: /Stage[main]/Hiera/File[/vagrant/hieradata]/group: change from vagrant to pe-puppet failed: Failed to set group to '495': Operation not permitted @ lchown_internal - /vagrant/hieradata
Notice: /Stage[main]/Puppet_enterprise::Master::Puppetserver/Service[pe-puppetserver]: Dependency File[/vagrant/hieradata] has failures: true
Warning: /Stage[main]/Puppet_enterprise::Master::Puppetserver/Service[pe-puppetserver]: Skipping because of failed dependencies
Notice: Applied catalog in 9.35 seconds

ws-marsegm:puppet-control marsegm$ tree -L 2
.
โ”œโ”€โ”€ Puppetfile
โ”œโ”€โ”€ Readme.md
โ”œโ”€โ”€ Vagrantfile
โ”œโ”€โ”€ environment.conf
โ”œโ”€โ”€ hieradata
โ”‚ย ย  โ”œโ”€โ”€ domain
โ”‚ย ย  โ”œโ”€โ”€ global.yaml
โ”‚ย ย  โ”œโ”€โ”€ nodes
โ”‚ย ย  โ””โ”€โ”€ tier
โ”œโ”€โ”€ hooks
โ”‚ย ย  โ”œโ”€โ”€ commit_hooks
โ”‚ย ย  โ”œโ”€โ”€ pre-commit
โ”‚ย ย  โ””โ”€โ”€ pre-receive
โ”œโ”€โ”€ manifests
โ”‚ย ย  โ””โ”€โ”€ site.pp
โ”œโ”€โ”€ pe-install.txt
โ”œโ”€โ”€ provision
โ”‚ย ย  โ”œโ”€โ”€ config
โ”‚ย ย  โ””โ”€โ”€ ssh
โ”œโ”€โ”€ reference
โ”‚ย ย  โ””โ”€โ”€ Puppet\ Code\ Promotion\ Workflow.jpeg
โ””โ”€โ”€ site
โ”œโ”€โ”€ profiles
โ””โ”€โ”€ roles

hieradata directory in PE 2015.2 multi-environment

It appears you can't leave datadir blank, and you can't ignore that setting, it always wants to write something to hiera.yaml. On PE 2015.2 it's blank by default so multi-environment works, but it breaks after this because hiera is no longer looking in the environment, only in one location. There should be an option to leave it blank like it is by default.

Using a hash for the template

Hi,

I am looking to extend this module to setup hiera to use hiera-file.

Firstly I am new to ruby programming, which is causing me to stumble on trying to make this module a little bit more generic. Essentially I am trying to work out a way to take a hash of values which can than be used within the template file to setup a range of hiera backends.

Here is how I have things setup so far:

I have a yaml file with this hash in it:

---
hiera::backends:
   yaml:
     datadir: '/root/dev/test'
   file:
     datadir: '/root/dev/foo'
   eyaml:
     datadir: '/root/dev/bar'
     pkcs7_private_key: '<%= @confdir %>/keys/private_key.pkcs7.pem'
     pkcs7_public_key: '<%= @confdir %>/keys/public_key.pkcs7.pem'

I than use hiera within puppet to pull the hash into a variable which the hiera.yaml.erg template will use:

 ---
:backends:
 <%= @testing.keys.to_yaml.split("\n")[1..-1].join("\n") %>

 <%= @testing.to_yaml.split("\n")[1..-1].join("\n") %>

It almost seems to work

---
:backends:
   - yaml
   - file
   - eyaml

   yaml:.
     datadir: /root/dev/test
   file:.
     datadir: /root/dev/foo
   eyaml:.
     datadir: /root/dev/bar
     pkcs7_private_key: "<%= @confdir %>/keys/private_key.pkcs7.pem"
     pkcs7_public_key: "<%= @confdir %>/keys/public_key.pkcs7.pem"

As you can see from the generated yaml file, ruby does not prepend a ":" (colon) in front of the keys and it also has indented the first key's.

Does anyone have any ideas on how I can modify the behaviour of the .to_yaml function to prepend a colon and not have it indent?

Thanks,

Peter

Unknown variable: '::pe_server_version'

Affected Puppet, Ruby, OS and module versions/distributions

  • Puppet: puppet-agent-1.8.0-1.el7.x86_64, puppetserver-2.6.0-1.el7.noarch
  • Ruby: ruby-2.0.0.598-25.el7_1.x86_64
  • Distribution: CentOS 7
  • Module version: v2.3.0

How to reproduce (e.g Puppet code you use)

     class { '::hiera':
         hierarchy  => [
             'nodes/%{::trusted.certname}',
             'virtual_%{::is_virtual}',
             'common',
         ],
     }

What are you seeing

Error: Could not retrieve catalog from remote server: Error 500 on SERVER: Server Error: Evaluation Error: Unknown variable: '::pe_server_version'. at /etc/puppetlabs/code/environments/production/forge_modules/hiera/manifests/params.pp:39:10 on node ...

What behaviour did you expect instead

To use it as is?

Output log

Any additional information you'd like to impart

I strongly suspect this is because I have strict_variables = true in my
puppet.conf though I've not studied this code that deeply yet.

hiera-eyaml failing to install with puppetserver_gem as a provider

I tried to install hiera-eyaml on puppetserver using puppetserver_gem, and got this error:

Error: Could not retrieve catalog from remote server: Error 400 on SERVER: Duplicate declaration: Package[hiera-eyaml] is already declared in file /etc/puppet/environments/enable_eyaml/modules/hiera/manifests/install.pp:60; cannot redeclare at /etc/puppet/environments/enable_eyaml/modules/hiera/manifests/install.pp:65 on node puppet1.domain.com

The relevant code is here: https://github.com/hunner/puppet-hiera/blob/master/manifests/install.pp#L54-L69

I suspect there's a name collision.

Allow user to specify version of hiera-eyaml

The current module lets you install eyaml but, once it's installed, you're stuck on whatever version happened to be current at the time. Please add a version property so that updates can be applied with this module. Thank you!

incompatible changes in version 2.2.0

After upgrading module to version 2.2.0 order of the configured backends has changed

version 2.1.2:

:backends:
  - eyaml
  - yaml

version 2.2.0:

:backends:
- yaml
- eyaml

Also, notice lack of spaces before the array

Changing order of the backends can lead to unexpected results (as it was in my case)

eyaml backend config doesn't quite work with 2015.2.3

Hi there,

I've just been setting up hiera eyaml at my site. using the following config in my site.pp

 class { 'hiera':
    hierarchy => [
      'secure',
      'node/%{::clientcert}',
      '%{environment}',
      'common',
    ],
    datadir   => '/etc/puppetlabs/code/environments/%{environment}/hieradata',
    eyaml     => true,
  }

this was basically all I did but puppet runs on my dev box started giving this

Error: Could not retrieve catalog from remote server: Error 400 on SERVER: Evaluation Error: Error while evaluating a Function Call, Must pass hieratestsecure to Class[Helloworld] at /etc/puppetlabs/code/environments/dev/manifests/site.pp:29:3 on node testy.mctester.test
Warning: Not using cache on failed catalog
Error: Could not retrieve catalog; skipping run

which confused me a bit because running a test from the puppet master would generate

hieratestsecure environment=dev --debug
DEBUG: 2015-12-09 13:40:08 +0100: [eyaml_backend]: Hiera eYAML backend starting
DEBUG: 2015-12-09 13:40:08 +0100: Using Hiera 1.x backend API to access instance of class Hiera::Backend::Eyaml_backend. Lookup recursion will not be detected
DEBUG: 2015-12-09 13:40:08 +0100: [eyaml_backend]: Set option: datadir = /etc/puppetlabs/code/environments/dev/hieradata
DEBUG: 2015-12-09 13:40:08 +0100: [eyaml_backend]: Set option: pkcs7_private_key = /etc/puppetlabs/code/keys/private_key.pkcs7.pem
DEBUG: 2015-12-09 13:40:08 +0100: [eyaml_backend]: Set option: pkcs7_public_key = /etc/puppetlabs/code/keys/public_key.pkcs7.pem
DEBUG: 2015-12-09 13:40:08 +0100: [eyaml_backend]: Looking up helloworld::hieratestsecure in eYAML backend
DEBUG: 2015-12-09 13:40:08 +0100: [eyaml_backend]: Looking for data source secure
DEBUG: 2015-12-09 13:40:08 +0100: [eyaml_backend]: Found helloworld::hieratestsecure in secure
DEBUG: 2015-12-09 13:40:08 +0100: [eyaml_backend]: Attempting to decrypt
this is soo secure!

and resolve correctly. eventually I found that running

$ sudo /opt/puppetlabs/bin/puppetserver gem install hiera-eyaml

then restart the pe-puppetmaster and I was up and running. I'm not sure that this is an actual issue with this module but it's a very confusing problem on an almost brand new server and there seems to be some confusion surrounding it.

prior to doing the above I had 2 outputs of 'local gems'

sudo /opt/puppetlabs/puppet/bin/gem list

*** LOCAL GEMS ***
(truncated list.....
faraday_middleware-multi_json (0.0.6)
hiera-eyaml (2.0.8)
..
..
test-unit (2.1.7.0)
trollop (2.1.2)

and then

/$ sudo /opt/puppetlabs/bin/puppetserver gem list

*** LOCAL GEMS ***

jar-dependencies (0.1.13)
jruby-openssl (0.9.7 java)
json (1.8.0 java)
rake (10.1.0)
rdoc (4.1.2)

and puppetserver seems to require the gem or it doesn't work.

Invalid package provider puppetserver_gem

Hi,

I am having the following error, when trying to configure eyaml, I think my version of PE is the issue (PE2016.1.2).

Error: Failed to apply catalog: Parameter provider failed on Package[puppetserver hiera-eyaml]: Invalid package provider 'puppetserver_gem' at /etc/puppetlabs/code/environments/production/modules/hiera/manifests/eyaml.pp:66

The class declaration is available here: custom_profile_puppetmaster/manifests/init.pp

manage hiera gem|package?

I use puppetlabs-operations to install puppet-agent|master, but I use this module (as per their recommendation) to manage hiera.

However, this module does not install hiera, that is, of course, because hiera has been a puppet dependency since 3.0.

Now the problem comes with trying to keep the infrastructure puppet-wise current: puppetlabs-operations puppet module has ensure=>latest, but only for puppet agent/master, this does not cascade down to their dependencies, and so hiera (and facter) are stuck at super old versions.

What's the best solution to that "problem"?

versioncmp expects a string value

In a new installation of PE 2015.3 I ran into the following error:

Error: Could not retrieve catalog from remote server: Error 400 on SERVER: Evaluation Error: Error while evaluating a Resource Statement, Evaluation Error: Error while evaluating a Function Call, 'versioncmp' parameter 'a' expects a String value, got Undef at /etc/puppetlabs/code/environments/production/modules/hiera/manifests/params.pp:43:8 on node

I created a pull request for the fix I'm using from my local copy.

pe_gem is not listed as a module dependency

Error: Parameter provider failed on Package[hiera-eyaml]: Invalid package provider 'pe_gem' at /etc/puppetlabs/puppet/modules/hiera/manifests/eyaml.pp:24
Wrapped exception:
Invalid package provider 'pe_gem'

Don't notify pe-puppetserver during eyaml installation

Notifying pe-puppetserver during eyaml installation chains the hiera::eyaml class to the puppet_enterprise module. This makes it difficult to bootstrap eyaml via puppet apply without dragging in the puppet_enterprise module and associated classification

json backend datadir can not be set

Affected Puppet, Ruby, OS and module versions/distributions

  • Puppet: 2016.2
  • Ruby: 2.1.9p490
  • Distribution: Debian
  • Module version: 2.1.3-rc0

How to reproduce (e.g Puppet code you use)

class { 'hiera': backends => [yaml , json], datadir => '/path/to/hieradata' }

What are you seeing

datadir only gets set for :yaml:


:backends:

  • yaml
  • json

:yaml:
:datadir: /path/to/hieradata

What behaviour did you expect instead

datadir is customizable for json

:backends:

  • yaml
  • json

:yaml:
:datadir: /path/to/hieradata

:json:
:datadir: /path/to/hieradata


:backends:

  • yaml
  • json

:yaml:
:datadir: /path/to/hieradata

:json:
:datadir: /path/to/hieradata_json

Output log

Any additional information you'd like to impart

In addition to this :yaml: will allways be set even if its not chosen as a backend

deep_merge_options are not set when merge_behavior => deeper

Affected Puppet, Ruby, OS and module versions/distributions

  • Puppet: 4.8.1
  • Ruby: 2.1.9p490
  • Distribution: Ubuntu 16.04
  • Module version: 2.3.0

How to reproduce (e.g Puppet code you use)

  class { '::hiera':
    create_keys        => false,
    master_service     => 'puppetserver',
    manage_package     => true,
    provider           => 'puppet_gem',
    eyaml              => true,
    eyaml_version      => 'latest',
    eyaml_datadir      => '/etc/puppetlabs/code/environments/%{::environment}/hieradata',
    eyaml_extension    => 'yaml',
    eyaml_gpg          => true,
    ruby_gpg_version   => 'latest',
    merge_behavior     => 'deeper',
    deep_merge_version => 'latest',
    backends           => [
      'eyaml',
    ],
    backend_options    => {
      'eyaml'      => {
        'gpg_gnupghome'    => '/etc/puppetlabs/puppet/keys/gpg',
      },
    },
    hierarchy          => [
      'nodes/%{::trusted.certname}',
      'common',
    ],
    deep_merge_options => {
      'knockout_prefix' => '--',
    },
  }

What are you seeing

hiera.yaml does not contain

:deep_merge_options:
  :knockout_prefix: --

What behaviour did you expect instead

hiera.yaml should contain

:deep_merge_options:
  :knockout_prefix: --

Any additional information you'd like to impart

I have submitted PR #172 to fix this.

Variables in a single-quote is changed to value - should not be (according to the README)

Affected Puppet, Ruby, OS and module versions/distributions

Puppet v3.8.4

What are you seeing

In the /etc/puppet/hiera.yaml the values in a hierarchy are completed strings but should be with some variables. According to the README it should work in other way.

What behavior did you expect instead

After the puppet executes the code my hiera.yaml file looks as below:

:hierarchy:
  - hieradata/host_type/dev/foreman

But should...:

:hierarchy:
  - "hieradata/host_type/%{::deployment_group}/%{::host_type}"

If I "escape" the special characters I have a mess in my hiera.yaml:
Configuration:

classes:
  - hiera
hiera::hierarchy:
  - 'hieradata/host_type/\%\{\:\:deployment_group\}/\%\{\:\:host_type\}'

Output:

:hierarchy:
  - 'hieradata/host_type/\x5c%\x5c{\x5c:\x5c:deployment_group\x5c}/\x5c%\x5c{\x5c:\x5c:host_type\}'

How did this behavior get triggered

My configuration for THIS host in hieradata is:

classes:
  - hiera
hiera::hierarchy:
  - 'hieradata/host_type/%{::deployment_group}/%{::host_type}'

Output log

Any additional information you'd like to impart

masterservice in the Documentation

The documentation states that one has to use add masterservice => 'puppetserver' to the configuration of the hiera module when using the FOSS Puppet verison. In the init.pp this parameter is named master_service.

Installing hiera-eyaml on a system that uses Ruby 1.8 results in failure

As per voxpupuli/hiera-eyaml#142, installation of hiera-eyaml on a system that uses Ruby 1.8 causes hiera-eyaml to be broken due to the release of HighLine 1.7.0 breaking things for Ruby 1.8. In fact, HighLine 1.7.0 has been yanked and HighLine 1.7.1 explicitly requires Ruby >=1.9.3, attempting to install hiera-eyaml fails now:

[root@puppet-eli ~]# puppet apply -e 'package{"hiera-eyaml":provider=>gem}'
Notice: Compiled catalog for puppet-eli.localdomain in environment production in 0.49 seconds
Error: Execution of '/usr/bin/gem install --no-rdoc --no-ri hiera-eyaml' returned 1: ERROR:  Error installing hiera-eyaml:
        highline requires Ruby version >= 1.9.3.
Error: /Stage[main]/Main/Package[hiera-eyaml]/ensure: change from absent to present failed: Execution of '/usr/bin/gem install --no-rdoc --no-ri hiera-eyaml' returned 1: ERROR:  Error installing hiera-eyaml:
        highline requires Ruby version >= 1.9.3.
Notice: Finished catalog run in 8.22 seconds

The solution for this is to detect old versions of Ruby and install a compatible version of HighLine first:

[root@puppet-eli ~]# puppet apply -e 'package{"highline":ensure=>"~> 1.6.19",provider=>gem}->package{"hiera-eyaml":provider=>gem}'
Notice: Compiled catalog for puppet-eli.localdomain in environment production in 0.51 seconds
Notice: /Stage[main]/Main/Package[highline]/ensure: created
Notice: /Stage[main]/Main/Package[hiera-eyaml]/ensure: created
Notice: Finished catalog run in 22.66 seconds

I'll submit a PR shortly.

Issue when installing Puppet Enterprise Compile Only Master (LEI)

When installing a Puppet Enterprise Compile Only Master (LEI) with the Puppet-Agent AIO package from the Master of Masters (MoM).

The current logic in the params.pp sees the node as a standalone instance of Puppet without a PuppetServer version. When the eyaml.pp manifest is run and tries to created the it is trying to set the owner and group to the user puppet when it should be pe-puppet.

The current solution that I am using is to set the owner and group in the profile that is used for managing my compile only masters.

@hunner : Not sure if its worth the complexity to add logic into the params.pp or just update the README to list this fix when installing compile only masters. IMO updating the README sounds like a good option

Update Puppet Forge

Hey, can you release a new module version with the eyaml changes? Thanks!

manage gems for backends

right now it's possible to declare a backends array, but if any of those backends need extra packages, those won't be installed.

(how?) should we fix that?

Enable Travis CI

@hunner if you can enable PR/Push testing in Travis, I'll be glad to work up the PR to get it working.

Add support for puppet 3.7 (puppetserver)

This module doesn't install the hiera-eyaml gem correctly on the new-ish puppet 3.7. This requires us to manually run the following command or work around it with an exec resource.

puppetserver gem install hiera-eyaml

Merge configuration, gem not installed by hiera module

Any chance the merge_behavior => 'deeper' parameter can manage the gem install as well? It's already being done on the eyaml-hiera gem...

/var/log/httpd/error_log
App 552 stderr: WARN: Thu Apr 09 10:39:33 -0400 2015: Ignoring configured merge_behavior
App 552 stderr: WARN: Thu Apr 09 10:39:33 -0400 2015: Must have 'deep_merge' gem installed.

Add support for 3rd party backends

I am currently using this module to manage hiera yaml backend and now need to support a different backend that is not currently supported by this module. Additionally, others also want support for adding backends of their own types especially like #152. In order to accommodate all the types of backends we need to manage them in a generic way instead of static parameters for each type.

Quoting of hierarchy entries only if string starts with percent sign

Affected Puppet, Ruby, OS and module versions/distributions

  • Puppet: Puppet Enterprise 2016.2
  • Ruby: Included in puppet-agent 1.5.2-1xenial
  • Distribution: Ubuntu 16.04.1 LTS
  • Module version: v2.1.0

How to reproduce (e.g Puppet code you use)

I experienced a similar behaviour as documented in issues/141 with PE 2016.2. I added the hiera class in the PE console as follows:

hierarchy

["nodes/%{::trusted.certname}", "common"]

to match PE defaults

What are you seeing

I got the result

 :hierarchy:
-  - "nodes/%{::trusted.certname}"
+  - nodes/%{::trusted.certname}
   - common

What behaviour did you expect instead

 :hierarchy:
  - "nodes/%{::trusted.certname}"
   - common

Output log

Notice: /Stage[main]/Hiera/File[/etc/puppetlabs/puppet/hiera.yaml]/content: 
--- /etc/puppetlabs/puppet/hiera.yaml   2016-06-13 20:41:34.000000000 +0200
+++ /tmp/puppet-file20160805-22530-rg1hp7   2016-08-05 16:11:26.562772693 +0200
@@ -1,13 +1,12 @@
+# managed by puppet
 ---
 :backends:
   - yaml
+
+:logger: console
+
 :hierarchy:
-  - "nodes/%{::trusted.certname}"
-  - common
+  - n/%{environment}

 :yaml:
-# datadir is empty here, so hiera uses its defaults:
-# - /etc/puppetlabs/code/environments/%{environment}/hieradata on *nix
-# - %CommonAppData%\PuppetLabs\code\environments\%{environment}\hieradata on Windows
-# When specifying a datadir, make sure the directory exists.
-  :datadir:
+  :datadir: /etc/puppetlabs/code/environments/%{::environment}/hieradata

Notice: /Stage[main]/Hiera/File[/etc/puppetlabs/puppet/hiera.yaml]/content: current_value {md5}38be16eebd05a89c2837135ab8dcec8b, should be {md5}d0b33fa37f02339ce04858cef5064035 (noop)
Notice: /Stage[main]/Hiera/File[/etc/puppetlabs/puppet/hiera.yaml]/owner: current_value root, should be pe-puppet (noop)
Notice: /Stage[main]/Hiera/File[/etc/puppetlabs/puppet/hiera.yaml]/group: current_value root, should be pe-puppet (noop)
Info: /Stage[main]/Hiera/File[/etc/puppetlabs/puppet/hiera.yaml]: Scheduling refresh of Service[pe-puppetserver]
Info: /Stage[main]/Hiera/File[/etc/puppetlabs/puppet/hiera.yaml]: Scheduling refresh of Service[pe-puppetserver]
Info: /Stage[main]/Hiera/File[/etc/puppetlabs/puppet/hiera.yaml]: Scheduling refresh of Service[pe-puppetserver]
Notice: /Stage[main]/Hiera/File[/etc/hiera.yaml]/ensure: current_value absent, should be link (noop)
Notice: /Stage[main]/Hiera/Ini_setting[puppet.conf hiera_config main section]/ensure: current_value absent, should be present (noop)
Info: /Stage[main]/Hiera/Ini_setting[puppet.conf hiera_config main section]: Scheduling refresh of Service[pe-puppetserver]

Any additional information you'd like to impart

In the template hiera.yaml.erb it is stated that the variable has to start with % to add quotes:

<%= @hierarchy.map{|x| x  =~ /%/ ? "  - \"#{x}\"" : "  - #{x}"}.join("\n") %>

Hierarchy interpolation seems to have changed.

Using this yaml file and including the class ::hiera...


---
classes:
  role::puppet
hiera::hierarchy:
    - 'clientcert/%%{}{clientcert}'
    - 'puppet_role/%%{}{puppet_role}'
    - 'global'
hiera::datadir: '/etc/puppetlabs/code/environments/%%{}{environment}/hiera'

Used to result in this file with Puppet 3.7 FOSS and Passenger or PE3.7/puppetserver:

# managed by puppet

---
:backends:
  - yaml

:logger: console

:hierarchy:
  - "clientcert/%{clientcert}"
  - "puppet_role/%{puppet_role}"
  - global

:yaml:
   :datadir: /etc/puppetlabs/code/environments/%{environment}/hiera

Using puppet4/puppetserver results in this file instead:

# managed by puppet

---
:backends:
  - yaml

:logger: console

:hierarchy:
  - clientcert/%#<Hiera::Scope:0x8668515>{clientcert}
  - puppet_role/%#<Hiera::Scope:0x584e63d>{puppet_role}
  - global

:yaml:
  :datadir: /etc/puppetlabs/code/environments/%#<Hiera::Scope:0x17a1410d>{environment}/hiera

Is there any known reason for why this behavior has changed? I was able to remedy this with the following changes, but it does not seem like the right solution:

#in site.pp
$percent = '%'

# in puppet.yaml

---
classes:
  role::puppet
hiera::hierarchy:
    - clientcert/%{percent}{clientcert}
    - 'puppet_role/%{percent}{puppet_role}'
    - 'global'

In Puppet 3.8.0, pe_puppetserver_gem provider is not working

The declared package resource in eyaml.pp line 28 - 32 does not seem to work in Puppet Enterprise 3.8.0. Only the hiera-eyaml gem shows up in the output of "/opt/puppet/bin/gem list", but NOT in "/opt/puppet/bin/puppetserver gem list" after applying the module.

Note: both providers (pe_gem and pe_puppetserver_gem) work in Puppet Enterprise 3.7.2 and hiera-eyaml shows up in gem list, and puppetserver gem list.

Release 1.3.3....

We currently have r10k deploying this off my forked branch because we need #88 .... any chance you could get a fresh forge module out soon?

Thanks :)

Path to eyaml gem binary is wrong

When not using PE, the gem gets installed by default (in CentOS 6) to /usr/bin/eyaml, but the eyaml.pp looks for the gem in /usr/bin/puppet/eyaml.

I'd suggest one or both of the following options:

  1. Change the default path in the params.pp to /usr/bin for $cmdpath
  2. Allow for setting the module's $cmdpath as a class parameter in init.pp (the public entry point)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.