Git Product home page Git Product logo

puppet-puppetmaster's Introduction

puppet-puppetmaster

This is a Puppet module and Kafo installer for setting up:

  • Puppetserver
  • Puppetserver with PuppetDB
  • Puppetserver with PuppetDB and Puppetboard

Each of the above is a separate Kafo installer scenario. This installer should work on CentOS 7, Debian 9, Ubuntu 18.04 and Ubuntu 20.04.

Note that files related to Kafo installer and Vagrant are only available in the GitHub project, not in the version published on Puppet Forge.

Setup outside of Vagrant

You can run the installer outside of Vagrant quite easily. First remove any old Puppet packages you might have installed. For example:

$ apt-get remove puppet-agent puppet5-release

Then do what Vagrant does to prepare the installer:

$ apt-get update && apt-get install git
$ cd /usr/share
$ git clone https://github.com/Puppet-Finland/puppet-puppetmaster puppetmaster-installer
$ cd puppetmaster-installer
$ vagrant/prepare.sh -b /usr/share/puppetmaster-installer -m

If you are going to use r10k and/or eyaml create a directory for their keys:

$ mkdir /usr/share/puppetmaster-installer/keys

Then copy your eyaml keys and your r10k private key there, naming them private_key.pkcs7.pem, public_key.pkcs7.pem and r10k_key. You need to configure control repo settings within the installer as well.

As the final preparatory step set the hostname, e.g.

$ hostnamectl set-hostname puppet.example.org

Now restart your shell session to refresh your PATH and Ruby environment. After this you can run the installer.

Interactive usage

To run the installer just

$ sudo -i
$ puppetmaster-installer -i

The "-i" switch to sudo ensures that the environment is root's environment, which is particularly important on Ubuntu and Debian. The -i switch to the installer makes it run in interactive mode, which is probably what you want to do.

Automatic installs

You can run the installer automatically like this:

$ sudo -i
$ /usr/share/puppetmaster-installer/bin/puppetmaster-installer\
 --scenario puppetserver-with-puppetboard\
 --puppetmaster-puppetboard-puppetdb-database-password='pass'\
 --puppetmaster-puppetboard-timezone='Europe/Helsinki'\
 --puppetmaster-puppetboard-puppetboard-password=‘pass’

When using Vagrant you can automate puppetserver setup during provisioning as well. To do this you need to modify two config files:

  • config/automated_install.conf (basic settings)
  • config/installer-scenarios.d/automated_install_answers.yaml (installer settings)

Make sure that you change the default passwords in the answer file.

If you change the scenario from the default (puppetserver-with-puppetboard) make sure that your answer file matches the scenario. You can create a matching answer file by removing config/installer-scenarios.d/last_scenario.yaml (if present), launching the installer interactively in a Vagrant VM, selecting the desired scenario, setting the parameters and launching the installer. Once the installer is running, the contents of that scenario's answer file will match what you selected. You can then copy that answer file to automated_install_answers.yaml. Alternatively you can use "Display current config" output as the content of the answer file. However, you then need to replace the first underscore ("_") with "::" because the kafo installer does module mapping. In either case make sure you have defined the passwords in the answer file or installation will fail immediately.

Then, in the root of the repository, run

$ sudo -i
$ bin/puppetmaster-install.sh

Vagrant will automatically copy your r10k deployment key and eyaml keys to correct locations if they are placed under keys directory in the repository root:

  • keys/private_key.pkcs7.pem (eyaml private key)
  • keys/public_key.pkcs7.pem (eyaml public key)
  • keys/r10k_key (r10k deployment key)

Development

Testing with PDK

This module has basic rspec tests that help ensure that catalog compilation works across all supported platforms. To run the unit tests do

$ pdk test unit

To validate code run

$ ./pdk-validate.sh

You cannot run "pdk validate" directly as it would scan through all the dependency modules multiple times (r10k modules, fixtures, build directories) and give tons of false positives and be really slow in general. If you interrupt that script you can just run it again to restore the offending directories to their original places.

Testing with Vagrant

This repository makes heavy use of Vagrant and Virtualbox for testing. You will need to use a fairly up-to-date Vagrant or you will run into networking issue with the Ubuntu boxes. We recommend using Vagrant 2.1.5 or later.

It is possible to run installer at the end of provisioning. This feature is primarily designed for Vagrant-based testing. To automatically setup a puppetserver with your desired answers first copy your answer file to config/installer-scenarios.d and run the installer like this:

RUN_INSTALLER=true SCENARIO=puppetserver vagrant up puppetserver-bionic

The answer file needs to match the scenario you chose. See vagrant/run_vagrant_tests.sh to see how this feature is utilized to automate regression testing.

Alternatively you can use installer's command-line parameters to define your answers.

Testing AWS AMI images created with packer

We use vagrant-aws Vagrant plugin to ease testing of packer-generated puppetmaster AMI images. First you need to setup vagrant-aws as per documentation:

$ vagrant plugin install vagrant-aws
$ vagrant box add dummy https://github.com/mitchellh/vagrant-aws/raw/master/dummy.box

Then make sure that the following standard AWS environment variables are set:

  • AWS_SECRET_ACCESS_KEY
  • AWS_ACCESS_KEY_ID

Optionally you can also set

  • AWS_DEFAULT_REGION: define which region Vagrant creates the instance to

There are a few non-standard environment variables you need to set as well:

  • AWS_AMI: the AMI ID that has puppetmaster-installer preconfigured
  • AWS_KEY_PAIR_NAME: the name of the SSH keypair at the AWS end
  • SSH_PRIVATE_KEY_PATH: path to the SSH private key matching the SSH keypair name, above

Once all these are set, you can use create, connect to and destroy the AWS instances as needed:

$ vagrant up puppetserver-bionic-aws
$ vagrant ssh puppetserver-bionic-aws
$ vagrant destroy puppetserver-bionic-aws

To run automated tests against an AWS instance use:

$ cd vagrant
$ AWS_AMI=<ami-id> ./run_aws_tests.sh

The log files are written to vagrant/test/logs.

Creating deb/rpm packages

Creating Debian and RPM packages is straightforward with the Debian 9 -based packager VM:

$ vagrant up packager
$ vagrant ssh packager
$ cd /home/ubuntu/puppetmaster-installer/packaging
$ ./package.sh

When upgrading package to new version the following Git spell will show which files have been added, modified or deleted since the last release:

$ git show --pretty="" --name-status <start-commit>...HEAD|sort|uniq

This helps avoid leaving critical files out of the packages.

Living with changes Kafo makes to versioned answer files

Kafo installers have a nasty habit of modifying answer files which are versioned by Git. To prevent these local answer file modifications from getting committed by accident you can use a command like this:

$ find config -name "*-answers.yaml"|xargs git update-index --assume-unchanged

Known issues

  • The installer will fail miserably if something else is locking dpkg/apt. This happens easily with Ubuntu Cloud images as they are configured (with systemd timers) to upgrade system packages on every boot. You should let the package upgrade process finish before trying to run the installer.

LICENSE

This project uses the two-clause BSD license. See LICENSE for details.

puppet-puppetmaster's People

Contributors

kibahop avatar mattock avatar msvilp avatar

Stargazers

 avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar

puppet-puppetmaster's Issues

Running installer adds .yardoc directory

When the installer is uninstalled with apt/dpkg, the following thing happens:

Removing puppetmaster-installer (1.1.2-1) ...
dpkg: warning: while removing puppetmaster-installer, directory '/usr/share/puppetmaster-installer' not empty so not removed

See if the directory could be configured to be elsewhere.

Firewalld blocks Puppetboard on CentOS 7

By default firewalld is enabled on CentOS 7. This means that Puppetboard won't be accessible until firewalld is disabled and iptables rules purged on CentOS 7 unless $manage_packetfilter = true.

Parameter storeconfigs goes into wrong section

The installer puts the "storeconfigs" settings into [master] section, which seems wrong nowadays as the setting does not have any effect according to "puppet config print storeconfigs".

In Puppet 5.x the correct section seems to "main":
puppet config set --section main storeconfigs true

This is probably resolvable by update of Foreman's puppet modules.

Wobbliness in host_entries management

In certain so far unknown conditions an empty $host_entries manages to slip into the hosts template, causing a ERB error. This is possibly related to answer files.

Remove or integrate reboot

It's again a separate scenario. Originally for selinux. If we don't need it, then let's get rid of it.

Lock modules in Puppetfile to particular versions

We don't librarian-puppet to fetch the latest version of the modules but a particular version of a module. Upgrades should be done in controlled fashion, and the results of the librarian-puppet run should be predictable and repeatable.

Add in documentation / README how to generate new Hiera eyaml keys

Hello,
Please add in documentation / README how to generate new Hiera eyaml keys for new Puppet installation. It may seems as unnessecary, but it will save time to beginers (who deploy Puppet first time) and doesn't know how to do it.

$ sudo mkdir /etc/puppetlabs/puppet/eyaml
$ sudo /opt/puppetlabs/puppet/bin/eyaml createkeys \
   --pkcs7-private-key /etc/puppetlabs/puppet/eyaml/private_key.pkcs7.pem \
   --pkcs7-public-key /etc/puppetlabs/puppet/eyaml/public_key.pkcs7.pem

$ sudo chown -R puppet:puppet
$ sudo chmod 500 /etc/puppetlabs/puppet/eyaml
$ sudo chmod 400 /etc/puppetlabs/puppet/eyaml/*.pem

Option --skip-puppet-version-check is required

The Kafo installer has to be launched with --skip-puppet-version-check because one of the modules claims not to support anything above Puppet 3, even though it works just fine on Puppet 5.

Add proper packerfilter support for Foreman scenarios

Currently the packetfilter class is turned off by default in Foreman and Smart Proxy scenarios. This is because the standard Puppetserver packetfilter configuration is insufficient. Write the proper support and enable packetfilter in these scenarios by default.

Remove possibly useless (yaml) files from installer

It seems that some (yaml) files in the installer are not read at all and as such are completely useless and confusing. For example config/kafo.yaml seems to be one of these. Remove such files and ensure that nothing breaks because of that.

Remove or integrate control-repo

Do we want to set up the control-repo alongside puppet master? Do most people want this? If not, we should remove it. It's really a separate scenario anyway.

Puppetboard scenario fails on Debian 9

It fails because the Puppetboard module makes faulty assumptions in its params.pp:

 case $facts['os']['family'] {
    'Debian': {
      if ($facts['os']['name'] == 'ubuntu') {
        $apache_confd = '/etc/apache2/conf-enabled'
      } else {
        $apache_confd   = '/etc/apache2/conf.d'
      }
      $apache_service = 'apache2'
}

In Debian 9 the correct directory is /etc/apache2/conf-enabled.

Add back host_entries management

Right now it is not possible to pass proper hash entries in Kafo wizard if the key is an IP address. In that case Kafo mangles the key by taking the first substring ending to an ".". For example, '10.10.50.1' becomes '10'. This can prevents adding host entries using the Kafo installer as a hash. The workaround might be to take an array of array of strings and convert that to a hash in a Ruby function.

The host entries are needed for two-way TLS communication when DNS entries are not present. This is in particular related to Foreman and Smart Proxy scenarios.

Fix puppetboard key issues

Puppet-lint reports a couple of problems:

./puppetboard.pp - WARNING: top-scope variable being used without an explicit namespace on line 42
./puppetboard.pp - WARNING: top-scope variable being used without an explicit namespace on line 44

The lines in question are:

$puppetboard_puppetdb_key  = "${::settings::ssldir}/private_keys/${puppetboard_certname}.pem"
$puppetboard_puppetdb_cert = "${::settings::ssldir}/certs/${puppetboard_certname}.pem"

The variable $puppetboard_certname is not defined.

Add support for automatically deploying a control repo

Introduction

Currently the installer installs r10k but does not configure it. This can be fixed, but there is some effort.

Add the config file

This is what the config file, /etc/puppetlabs/r10k/r10k.yaml, looks like:

---
:cachedir: /opt/puppetlabs/puppet/cache/r10k
:sources:
  control:
    basedir: /etc/puppetlabs/code/environments
    prefix: false
    remote: <remote-git-url>

Install deployment key

Most control repositories are not public and a SSH deployment key is usually required. We don't have the key, so the user has to place it in the correct location. We have traditionally used /etc/puppetlabs/r10k/ssh/id_rsa.

Add SSH host entry

If a non-default SSH key is used then an SSH host entry is needed in /root/.ssh/config:

Host bitbucket.org
StrictHostKeyChecking no
RSAAuthentication yes
IdentityFile /etc/puppetlabs/r10k/ssh/id_rsa
User git

Add known_hosts entry

r10k will fail miserably if there is no known_host entry for the Git repository server hosting the control repository. We can add them for most of the common hosted ones (GitHub, GitLab, Bitbucket), but for others the user has to provide entries for.

Fetch the control repository

Control repository should be fetched automatically.

Deploy production environment

The production environment should be deployed automatically.

Foreman scenarios are lacking sane defaults

The Foreman scenarios (foreman_proxy and lcm) depended heavily on sane defaults stored in answer files. Now the answer files are empty, so the default values are missing. Add sane defaults which can be used reasonably in any environment.

A related problem is defaults for Vagrant, which can be considered a special environment with additional sane defaults. My proposed solution is this:

  • Add generic, sane defaults to foreman_proxy.pp and lcm.pp
  • Create an answer file suitable for the Vagrant environment
  • Store and version the answer file
  • Ensure that the default (almost empty) answer file is overwritten during Vagrant provisioning with the answer file tailored for Vagrant

This way Vagrant-specifisms don't accidentally end up in production environments.

Deploying environments fails with recent r10k versions

Latest r10k does not seem to work properly in our context:

root@puppet:~# r10k deploy environment production -vp -c /etc/puppetlabs/r10k/r10k.yaml

r10k: Runtime error: #<NoMethodError: undefined method `[]=' for nil:NilClass> 

This is with r10k version 3.2.0. I suspect that the new version cannot parse the old-format config file, r10k.yaml, and hence fails with this very unhelpful error message.

The workaround is to downgrade to r10k 2.6.5:

/opt/puppetlabs/puppet/bin/gem uninstall r10k
/opt/puppetlabs/puppet/bin/gem install r10k -v 2.6.5

CentOS 7 box version 1804.02 complains about locales

When launching the installer the following error is displayed:

2018-07-17 15:09:44.434737 WARN puppetlabs.facter - locale environment variables were bad; continuing with LANG=C LC_ALL=C

This only occurs with box version 1804.02 which we need to update to to fix other issues.

Installer complains about deprecated settings

Warning: Setting autosign is deprecated. 
   (location: /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/settings.rb:1169:in `issue_deprecation_warning')
Warning: Setting ca is deprecated. 
   (location: /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/settings.rb:1169:in `issue_deprecation_warning')

Invalid Kafo-generated hiera.conf

Recently the installer has started failing with these errors:

[ERROR 2019-03-20T05:45:35 main] Errors encountered during run:
[ERROR 2019-03-20T05:45:35 main]  Evaluation Error: Error while evaluating a Function Call, Lookup of key 'lookup_options' failed:  The Lookup Configuration a
t '/tmp/kafo_hiera20190320-5670-iw2i8y/hiera.conf' has wrong type, unrecognized key 'backends'

The offending hiera.conf looks like this:

---
:backends:
- yaml
:hierarchy:
- "%{::osfamily}"
- common
:yaml:
  :datadir: "/usr/share/puppetmaster-installer/config/data"
version: 5
defaults:
  datadir: "/tmp/kafo_hiera20190320-6062-di2vrb/data"
  data_hash: yaml_data
hierarchy:
- name: Kafo Answers
  path: kafo_answers.yaml
  datadir: "/tmp/kafo_hiera20190320-6062-di2vrb/data"
  data_hash: yaml_data

Pre-filled answer files may end up in the package

The package.sh script does not "git checkout" the answer files. So, if the answer files have been modified, for example during Vagrant testing, these modified files end up in the package. This should be prevented to ensure consistency of the packages.

Puppetboard scenario does not work on CentOS 7

As we have no immediate need for CentOS 7 Puppetmaster with Puppetboard I will just note down the issues encountered with it.

The httpd service does not start because /etc/httpd/conf.modules.d/00-mpm.conf tries to load a conflicting MPM module:

LoadModule mpm_prefork_module modules/mod_mpm_prefork.so

This is probably the reason why we originally set the MPM to "prefork" across all platforms. But now, as we don't purge_configs with puppetlabs-apache this causes a issue. Simply commenting out or removing that line makes httpd start correctly.

Another issue is that the Python module attempts to install python3-virtualenv package which is not present on CentOS 7. This would have to be fixed.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.