Git Product home page Git Product logo

puppetlabs-peadm's Introduction

Puppet Enterprise Administration Module (PEADM)

The Puppet Enterprise Administration Module (PEADM) contains a set of Bolt plans designed for deploying and managing Puppet Enterprise (PE) infrastructure. These plans automate key PE lifecycle activities to accelerate deployment and reduce the risk of human error.

You can use PEADM to deploy and manage PE installations for standard, large, and extra-large architectures.

Important: PEADM is compatible with PE 2019.8.1 and later versions. If your PE version is older than 2019.8.1 and you want to use PEADM, you must upgrade PE before converting your installation to a PEADM-managed installation.

Table of contents

Support

PEADM is a supported PE module. If you are a PE customer with the standard or premium support service, you can contact Support or your Technical Account Manager for assistance.

Overview

This is the standard workflow for installing PEADM.

  1. Install Bolt on a jump host.
  2. To create a Bolt project that includes the PEADM module, run:
    mkdir <YOUR_PROJECT> && cd <YOUR_PROJECT> && bolt project init <YOUR_PROJECT> --modules puppetlabs-peadm
  3. In the Bolt project directory, update the inventory.yaml file with connection information for the servers you plan to use for hosting PE infrastructure. For example:
---
groups:
  - name: puppet-enterprise-nodes
    config:
      transport: ssh
      ssh:
        host-key-check: false
        user: centos
        run-as: root
    targets:
      - pe-xl-core-0.lab1.puppet.vm
      - pe-xl-core-1.lab1.puppet.vm
      - pe-xl-core-2.lab1.puppet.vm
      - pe-xl-core-3.lab1.puppet.vm
      - name: pe-xl-compiler-0.lab1.puppet.vm
        uri: 10.234.6.45
      - name: pe-xl-compiler-1.lab1.puppet.vm
        uri: 10.234.14.131

What PEADM affects

  • The peadm::install plan adds a number of custom original identifier (OID) trusted facts to the certificates of deployed PE infrastructure nodes. These trusted facts are used by PEADM plans to identify nodes that host PE infrastructure components.
  • Depending on the scale of your architecture, up to four node groups may be created to configure puppet_enterprise class parameters for the following PE infrastructure components:
    • The primary server
    • The primary server replica
    • PostgreSQL nodes (database servers)
    • Compilers (compiler hosts are designated as belonging to availability group A or B)

What PEADM does not affect

  • PEADM does not impact regular PE operations. After using it to deploy a new PE installation or upgrade an existing one, PEADM is not required until you want to use it to upgrade PE or expand your installation.
  • Using PEADM to install PE or upgrade PE does not prevent you from using documented PE procedures such as setting up disaster recovery or performing a manual upgrade.

Requirements

  • PEADM is compatible with Puppet Enterprise 2019.8.1 or newer versions.

  • To use PEADM, you must first install Bolt version 3.17.0 or newer.

  • PEADM supports PE installations on the following operating systems:

    • Amazon Linux 2
    • AlmaLinux 8
    • CentOS 7 and 8
    • RHEL 7, 8 and 9
    • SLES 12
    • Ubuntu 18.04 and 20.04
  • To successfully convert your current PE installation to a PEADM-managed installation, ensure that the PE setting for editing classifier configuration data is enabled. This setting is enabled by default on new PE installations, but it could be disabled if the relevant configuration was removed from your global hiera.yaml file. See the PE docs for more information.

Usage

For instructions on using PEADM plans, see the following PEADM docs:

Reference

To understand which architecture is right for you, see the following information on the Puppet documentation site:

To learn more about the PEADM module and its uses, see the following PEADM docs:

Getting help

License

This codebase is licensed under Apache 2.0. However, the open source dependencies included in this codebase might be subject to other software licenses such as AGPL, GPL2.0, and MIT.

puppetlabs-peadm's People

Contributors

aaronshannon avatar abuxton avatar bastelfreak avatar billclaytor avatar bryanjbelanger avatar bwilcox avatar cathal41 avatar cdenneen avatar comfucios avatar davidsandilands avatar github-actions[bot] avatar golflimaechoecho avatar h0tw1r3 avatar j-hunniford avatar jo-lillie avatar johnduarte avatar jortencio avatar jpartlow avatar logicminds avatar martyewings avatar mcka1n avatar miharp avatar ody avatar ragingra avatar reidmv avatar sharpie avatar spidersddd avatar timidri avatar tkishel avatar vchepkov avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

puppetlabs-peadm's Issues

convert fails on newer installs

Describe the Bug

The convert step fails if the pe_version is new and convert has never been run

Expected Behavior

that the error message would not occur.

Steps to Reproduce

Steps to reproduce the behavior:

  1. have pe_version = 2019.8.3
  2. run convert plan

Additional Context

{
  "msg" : "PE deployment cannot be converted! PE deployment must be a deployment created by pe_xl, by an older version of peadm, or be PE version 2019.7.0 or newer. Detected PE version 2019.8.3, and did not detect signs that the deployment was previously created by peadm/pe_xl.\n",
  "kind" : "bolt/plan-failure",
  "details" : {
    "class" : "Bolt::PlanFailure"
  }
}

Soltion

Use the following solution instead
(!$previously_configured_by_peadm and ($pe_version =~ SemVerRange(‘< 2019.7.0’)))

optional data incorrectly parsed by function or task file_or_content

Describe the Bug

https://github.com/puppetlabs/puppetlabs-peadm/blob/master/functions/file_or_content.pp
the variables are strongly typed as String, yet the input is Optional

both license and r10k value are opotional but last code refactor includes chanegs that expect the values to be set! this is undesirable in usage while demonstrating the module and or in mono builds as the supporting infrastructure is not always available these were previously captured in if statments!

https://github.com/puppetlabs/puppetlabs-peadm/blob/master/plans/action/install.pp#L106
https://github.com/puppetlabs/puppetlabs-peadm/blob/master/plans/action/install.pp#L109
https://github.com/puppetlabs/puppetlabs-peadm/blob/master/plans/action/install.pp#L133

Expected Behavior

it should work without optional data

Steps to Reproduce

Steps to reproduce the behavior:
run the task with out values for license or r10k key

Environment

Additional Context

2 possible fixes
update the function to allow undef Optional[String]
update the plan to re introduce if case statement

task metadata lint error

Describe the Bug

When I run pdk validate I receive the following error

error: task-metadata-lint: tasks/agent_install.json: The property '#/parameters/install_flags/default' of type array did not match the following type: string %

Expected Behavior

no errors.

Steps to Reproduce

Steps to reproduce the behavior:

  1. pdk validate

peadm::convert doesn't add Compiler groups to the correct parent

Describe the Bug

When running peadm::convert in a monolithic configuration with compilers and external postgres (and replica master + postgres), the plan can fail to apply the Node_group[PE Compiler Group A] and Node_group[PE Compiler Group B] resources from the /manifests/setup/node_manager.pp manifest.

Expected Behavior

A clear and concise description of what you expected to happen.

/manifests/setup/node_manager.pp is expected to configure its own group requirements according to the assumptions both peadm::convert and peadm::install make about which groups exist, what their names are, and what information and nodes they contain.

Steps to Reproduce

Steps to reproduce the behavior:

  1. Create a monolithic deployment with compilers
  2. Create all standard node groups EXCEPT "PE Compiler"
  3. Run the peadm::convert plan

Environment

  • Version 2019.2.2
  • Platform Ubuntu 18.04

Additional Context

We believe this is caused by these two pieces being misaligned:

https://github.com/puppetlabs/puppetlabs-peadm/blob/main/manifests/setup/node_manager.pp#L52

  node_group { 'PE Master':
    parent    => 'PE Infrastructure',
    data      => $compiler_pool_address_data,
    variables => { 'pe_master' => true },
    rule      => ['or',
      ['and', ['=', ['trusted', 'extensions', 'pp_auth_role'], 'pe_compiler']],
      ['=', 'name', $master_host],
    ],
  }

The above appears to create the correct expected Node group for compilers. (peadm::install also expects there to be specific class information here)

However, the below resource tries to put the compiler A pool into a group called "PE Compiler". A group named "PE Compiler" is not declared in node_manager.pp and appears (at least on the surface) to be a mistaken assumption when coming from PE 2019.2.2. The same is repeated for compiler pool B node_group resource.

https://github.com/puppetlabs/puppetlabs-peadm/blob/main/manifests/setup/node_manager.pp#L90

  # Configure the A pool for compilers. There are up to two pools for HA, each
  # having an affinity for one "availability zone" or the other.
  node_group { 'PE Compiler Group A':
    ensure  => 'present',
    parent  => 'PE Compiler',
    data    => { },
    rule    => ['and',
      ['=', ['trusted', 'extensions', 'pp_auth_role'], 'pe_compiler'],
      ['=', ['trusted', 'extensions', peadm::oid('peadm_availability_group')], 'A'],
    ],
    classes => {
      'puppet_enterprise::profile::puppetdb' => {
        'database_host' => $puppetdb_database_host,
      },
      'puppet_enterprise::profile::master'   => {
        'puppetdb_host' => ['${trusted[\'certname\']}', $master_replica_host].filter |$_| { $_ }, # lint:ignore:single_quote_string_with_variables
        'puppetdb_port' => [8081],
      }
    },
  }

We believe that either:
A) node_manager.pp should declare "PE Compiler" or
B) node_manager.pp should associate both compiler pools with the "PE Master" node group.

upgrade fails during puppet_runonce

Describe the Bug

During a upgrade there is a bash script that fails and causes the plan to fail with an unfinished upgrade.

# Plan Step: upgrade-primary-compilers
Starting: task peadm::puppet_infra_upgrade on pe-std.puppet.vm
Finished: task peadm::puppet_infra_upgrade with 0 failures in 1.22 sec
# Plan Step: upgrade-replica
Starting: task peadm::puppet_runonce on pe-std.puppet.vm
Finished: task peadm::puppet_runonce with 0 failures in 29.89 sec
Starting: command 'if [ -e /opt/puppetlabs/server/apps/puppetdb/cli/apps/delete-reports -a ! -h /opt/puppetlabs/server/apps/puppetdb/cli/apps/delete-reports ]
then
  mv /opt/puppetlabs/server/apps/puppetdb/cli/apps/delete-reports /opt/puppetlabs/server/apps/puppetdb/cli/apps/delete-reports.original
  ln -s $(which true) /opt/puppetlabs/server/apps/puppetdb/cli/apps/delete-reports
fi
' on pe-std-replica.puppet.vm
Finished: command 'if [ -e /opt/puppetlabs/server/apps/puppetdb/cli/apps/delete-reports -a ! -h /opt/puppetlabs/server/apps/puppetdb/cli/apps/delete-reports ]
then
  mv /opt/puppetlabs/server/apps/puppetdb/cli/apps/delete-reports /opt/puppetlabs/server/apps/puppetdb/cli/apps/delete-reports.original
  ln -s $(which true) /opt/puppetlabs/server/apps/puppetdb/cli/apps/delete-reports
fi
' with 1 failure in 0.91 sec
Finished: plan peadm::upgrade in 2 min, 21 sec
Failed on pe-std-replica.puppet.vm:
  The command failed with exit code 126
  OCI runtime exec failed: exec failed: container_linux.go:380: starting container process caused: exec: "if": executable file not found in $PATH: unknown


  This might be caused by the default tmpdir being mounted using 'noexec'. See http://pup.pt/task-failure for details and workarounds.
Failed on 1 target: pe-std-replica.puppet.vm

Expected Behavior

The upgrade plan with finish without errors. I don't really know what other steps are required to complete the replica upgrade.

Steps to Reproduce

Steps to reproduce the behavior:

  1. From peadm codebase, cd spec/docker
  2. bash install.sh (should install 2019.8.5)
  3. Choose standard-ha
  4. Once complete, will need to upgrade next
  5. change the version in standard-ha/upgrade_params.json to 2019.8.8
  6. From spec/docker, bash upgrade.sh, select standard-ha
  7. Wait for upgrade and watch for error to occur towards end of upgrade.
{
  "primary_host": "pe-std.puppet.vm",
  "replica_host": "pe-std-replica.puppet.vm",
  "version": "2019.8.8"
}

Environment

CentOS:8
PE 2019.8.5, 2019.8.8

Additional Context

The current dockerfile doesn't have centos:8 since I have not created a PR for it yet. So this might just be a centos:8 problem as I haven't tested with centos:7. The steps above will use centos:7 which might have different results.

peadm default pe.conf settings conflict with pe_tune

pe.conf java args settings added by peadm::generate_pe_conf function are not compatible with pe_tune module.

# cat /etc/puppetlabs/enterprise/conf.d/pe.conf 
{
  "puppet_enterprise::profile::master::java_args": {
    "Xmx": "2048m",
    "Xms": "512m"
  },
  "puppet_enterprise::profile::console::java_args": {
    "Xmx": "768m",
    "Xms": "256m"
  },
  "puppet_enterprise::profile::orchestrator::java_args": {
    "Xmx": "768m",
    "Xms": "256m"
  },
  "puppet_enterprise::profile::puppetdb::java_args": {
    "Xmx": "768m",
    "Xms": "256m"
  }
}
# puppet infrastructure tune --local --pe_conf
# Unable to output optimized settings to: /etc/puppetlabs/enterprise/conf.d/pe.conf: conflicting settings found.

Unable to deploy LTS (2018.1.11)

Describe the Bug

Master fails to sign certificates with dns alternate names.

CSR 'pe-compiler-1e5a2a-1.us-west1-b.c.slice-cody.internal' contains subject alternative names (DNS:pe-compiler-1e5a2a-1.us-west1-b.c.slice-cody.internal, DNS:puppet, DNS:puppet.pe-compiler-lb-1e5a2a.il4.us-west1.lb.slice-cody.internal), which are disallowed. To allow subject alternative names, set allow-subject-alt-names to true in your puppetserver.conf file, restart the puppetserver, and try signing this certificate again.

Expected Behavior

That the certificates are appropriately signed like other, newer versions of PE

Additional Context

PE 2019.2.2 ships with this configuration set by default

DNS Alt Names in signing

Use Case

Compile masters with dns alt names

Describe the Solution You Would Like

Ability to have compile masters with DNS alt names signed with the peadm::sign_csr

Describe Alternatives You've Considered

Forking module and wrapping it or creating an autosign and attempting to inject before the task

direct download of installer does not respect staging directory path

Describe the Bug

When I am using the direct download mode and a custom staging directory, the peadm module does not download to the staging direct. Instead it goes to `/tmp/${pe_tarball_name}"

Expected Behavior

If my staging director is /stagingdir I expect the tarball to be download there.

Steps to Reproduce

Run the installer with these params

{
  "primary_host": "pe-lg.puppet.vm",
  "compiler_hosts": ["pe-lg-compiler-0.puppet.vm"],
  "console_password": "puppetlabs",
  "dns_alt_names": [ "puppet", "pe-lg.puppet.vm" ],
  "version": "2019.8.8",
  "stagingdir": "/stagingdir",
  "download_mode": "direct"
}

Environment

centos:7

Additional Context

https://github.com/puppetlabs/puppetlabs-peadm/blob/main/plans/subplans/install.pp#L208

The path should be "${stagingdir}/${pe_tarball_name}", instead of $upload_tarball_path

task peadm::pe_install doesn't handle commands exit codes properly [SOLARCH-545]

Seems I have started upgrading too early, file hasn't been properly uploaded on yet, so I got this content (consistently)

$ cat /opt/staging/puppet-enterprise-2019.5.0-el-8-x86_64.tar.gz 
<?xml version="1.0" encoding="UTF-8"?>
<Error><Code>NoSuchKey</Code><Message>The specified key does not exist.</Message><Key>released/2019.5.0/puppet-enterprise-2019.5.0-el-8-x86_64.tar.gz</Key><RequestId>281FD5D76966A802</RequestId><HostId>30Us5WIkg+ITt5JFSwxlX184ncmHJH/vWG5lltbLM7NP89F/VSEk2ZwaA/hrb9mfBcikeJbstBE=</HostId></Error>

upgrade plan proceeded all the way to task peadm::pe_install and it executed successfully as well, without doing an upgrade.

debugging of the task showed that results of the tar command are ignored:

$ bolt task run peadm::pe_install --targets puppet.chepkov.lan tarball=/tmp/puppet-enterprise-2019.5.0-el-8-x86_64.tar.gz puppet_service_ensure=stopped --debug
Loaded configuration from: '/Users/vvc/Bolt/home/bolt.yaml'
Loading modules from /opt/puppetlabs/bolt/lib/ruby/gems/2.5.0/gems/bolt-2.2.0/bolt-modules:/Users/vvc/Bolt/home/modules:/Users/vvc/Bolt/home/site-modules:/Users/vvc/Bolt/home/site:/opt/puppetlabs/bolt/lib/ruby/gems/2.5.0/gems/bolt-2.2.0/modules
Analytics opt-out is set, analytics will be disabled
Loaded inventory from /Users/vvc/Bolt/home/inventory.yaml
Skipping submission of 'task_run' screenview because analytics is disabled
Started with 100 max thread(s)
Starting: task peadm::pe_install on puppet.chepkov.lan
Skipping submission of 'Transport initialize' event because analytics is disabled
Running task peadm::pe_install with '{"tarball":"/tmp/puppet-enterprise-2019.5.0-el-8-x86_64.tar.gz","puppet_service_ensure":"stopped","_task":"peadm::pe_install"}' on ["puppet.chepkov.lan"]
Running task run '#<Bolt::Task:0x00007fb040c06618>' on puppet.chepkov.lan
Started on puppet.chepkov.lan...
Initializing ssh connection to puppet.chepkov.lan
Opened session
Running '/Users/vvc/Bolt/home/modules/peadm/tasks/pe_install.sh' with {"tarball":"/tmp/puppet-enterprise-2019.5.0-el-8-x86_64.tar.gz","puppet_service_ensure":"stopped","_task":"peadm::pe_install"}
Executing: mkdir -m 700 /tmp/645c830d-bc50-44bc-af85-71df61a05759
Command returned successfully
Uploading /Users/vvc/Bolt/home/modules/peadm/tasks/pe_install.sh, to /tmp/645c830d-bc50-44bc-af85-71df61a05759/pe_install.sh
Executing: chmod u\+x /tmp/645c830d-bc50-44bc-af85-71df61a05759/pe_install.sh
Command returned successfully
Executing: id -g root
stdout: 0
Command returned successfully
Executing: sudo -S -H -u root -p \[sudo\]\ Bolt\ needs\ to\ run\ as\ another\ user,\ password:\  sh -c 'cd && chown -R root:0 /tmp/645c830d-bc50-44bc-af85-71df61a05759'
Command returned successfully
Executing: PT_tarball=/tmp/puppet-enterprise-2019.5.0-el-8-x86_64.tar.gz PT_puppet_service_ensure=stopped PT__task=peadm::pe_install sudo -S -H -u root -p \[sudo\]\ Bolt\ needs\ to\ run\ as\ another\ user,\ password:\  -E sh -c 'cd && /tmp/645c830d-bc50-44bc-af85-71df61a05759/pe_install.sh'
stderr: tar: This does not look like a tar archive
stderr: gzip: stdin: not in gzip format
stderr: tar: Child returned status 1
tar: Error is not recoverable: exiting now
stderr: dirname: missing operand
Try 'dirname --help' for more information.
stderr: gzip: stdin: not in gzip format
stderr: tar: Child returned status 1
tar: Error is not recoverable: exiting now
stderr: 
stderr: /bin/bash: /tmp//puppet-enterprise-installer: No such file or directory
Command returned successfully
Executing: sudo -S -H -u root -p \[sudo\]\ Bolt\ needs\ to\ run\ as\ another\ user,\ password:\  sh -c 'cd && rm -rf /tmp/645c830d-bc50-44bc-af85-71df61a05759'
Command returned successfully
Closed session
{"target":"puppet.chepkov.lan","action":"task","object":"peadm::pe_install","status":"success","value":{"_output":""}}
Finished on puppet.chepkov.lan:
  Task completed successfully with no result
Finished: task peadm::pe_install with 0 failures in 1.3 sec
Successful on 1 target: puppet.chepkov.lan
Ran on 1 target in 1.37 sec

add_compiler plan not configuring puppedb allow list on secondary

Describe the Bug

When the add_compiler plan is ran in a deployment with a replica configured, the compiler's puppetserver is configured to do command broadcast to its colocated puppetdb and the instance which resided on the primary or replica housed in the alternate availability group. The rule which allows that compiler's puppetserver to connect to the additional puppetdb instance is configured through a puppet run but right now that puppet run it not occuring.

Expected Behavior

That a new compiler is authorized to command broadcast to all configured puppetdb instances.

license.key is installed with restrictive permissions

While installing PE2019.8.0 with parameter license_key_file, task peadm::pe_install completes successfully, but task peadm::puppet_runonce fails due to restrictive permissions set on the license file:

# ls -l /etc/puppetlabs/license.key 
-r-------- 1 root root 367 Jul  6 20:51 /etc/puppetlabs/license.key
Jul  6 20:52:32 puppet puppet-agent[10085]: Could not retrieve catalog from remote server: Error 500 on SERVER: Server Error: Evaluation Error: Error while evaluating a Resource Statement, Evaluation Error: Error while evaluating a Function Call, Permission denied - /etc/puppetlabs/license.key (file: /opt/puppetlabs/puppet/modules/puppet_enterprise/manifests/license.pp, line: 14, column: 22) on node puppet.example.com

puppetdb node classification group map [SOLARCH-543]

Use Case

it would be really useful to have a default inventory with groups for the pe infrastructure classification groups.

Describe the Solution You Would Like

Work out a PuppetDB query such as https://puppet.com/docs/bolt/latest/bolt_connect_puppetdb.html#practical-usage
that will query PuppetDB based on the assigned classes and create a suitable inventory.yaml that can be used with bolt directly or as the query for Bolt with the orchestrator.

Describe Alternatives You've Considered

parse puppet resource pe_node_group and render the inventory.yaml from the pe infrastructure node groups tree parent and children.

Additional Context

we can handball it, but it's a pain it's useful to have for customers to prefill in as a template before setup, it is valuable to have post-deploy for reference and can be used for Plan target_nodes

puppetdb.conf for "PE Compiler Group B" contains 3 entries when HA is enabled

Describe the Bug

Using the XL with HA architecture, the puppetdb.conf on the compilers in PE Compiler Group B has the certname, master, and replica in the server_list. Since command_broadcast is enabled, the commands will be sent to two PuppetDB instances using the same PostgreSQL instance, increasing load and potentially causing contention without any benefit.

Example:

[main]
server_urls = https://pe-201980-compile-02.puppetdebug.vlan:8081,https://pe-201980-master.puppetdebug.vlan:8081,https://pe-201980-replica.puppetdebug.vlan:8081
command_broadcast = true

Expected Behavior

The puppetdb.conf should only have the certname, and master in the server_list for compilers in PE Compiler Group B.

Steps to Reproduce

  1. Provision PE 2019.2.0+ PE XL with HA
  2. Log into a compiler in PE Compiler Group B
  3. Observe the /etc/puppetlabs/puppet/puppetdb.conf has three entries in the server_list.

Environment

  • Version: 2.1.1
  • Platform: Any. Tested in EL 7

Additional Context

This happens because $puppet_enterprise::ha_enabled_replicas is automatically merged with the puppetdb_host in puppet_enterprise::profile::master::puppetdb due to PE-27165.

The proposed workaround is to set the following in the enterprise/conf.d/nodes/*.conf, hiera, or console data only for the compilers.

puppet_enterprise::profile::master::puppetdb::ha_enabled_replicas: []

add puppetlabs/metrics for supports use case as is standard deployment practice

Use Case

Support have created https://forge.puppet.com/puppetlabs/puppet_metrics_collector this module is deployed with every Puppet Enterprise deployment to append to the support script.
it should be deployed as part of the standard platform.

Describe the Solution You Would Like

include deployment of puppet_metrics_collector in each pattern (it needs different information providing to it depending upon platform deployment)

Describe Alternatives You've Considered

NA

Additional Context

we may want to tack the https://forge.puppet.com/puppetlabs/puppet_metrics_dashboard onto a pe_client_tools bastion host (or workstation)

submit_csr hangs in existing puppet agent was not configured with a server

Describe the Bug

If the system has a previously installed puppet-agent with a configuration that has no server configured, or points to something other than the CA used to run peadm::install. This causes the agent-install task to not configure the puppet.conf with the correct server information. The ca certificate will not be present on a compiler host as it will be pointing to either puppet:8140 or some other pre-configured servername. On new installs the CA cert would not be present either. Upon executation of the submit_csr the task fails since the servername never times out and appear to hang from the user's perspective.

Expected Behavior

Submit CSR task would finish within a few seconds.

Steps to Reproduce

Steps to reproduce the behavior:
You can use the docker containers under spec/docker and start a large instance. The Dockerfile in the spec/docker will need to install the puppet_agent or use a image that already has the agent installed.

  1. cd spec/docker
  2. vi Docker and replace FROM centos:7 to an image with the puppet_agent pre-installed (centos based)
  3. run bash install.sh
  4. select large
  5. wait a few minutes
  6. Should see the submit_csr task hang
  7. You can docker exec -ti pe-lg-compiler-1.puppet.vm /bin/bash and run puppet agent -t for issue

Note: I have since patched the Dockerfile to remove the puppet-agent rpm so you may need to change this line.

PEADM Incompatible with FIPS Image

Describe the Bug

PEADM isn't able to install the FIPS Image of PE.

The logic in peadm that determines the version in https://github.com/puppetlabs/puppetlabs-peadm/blob/main/tasks/precheck.sh doesn't account for FIPS

The PE Installer script uses different logic to determine which version is valid and they don't match

Expected Behavior

PEADM accurately detects the host is FIPS and deploys the FIPS image of PE.

Steps to Reproduce

  1. Attempt to install PE on a RHEL 7 FIPS Image
  2. PEADM Downloads the non-FIPS RHEL Image
  3. PE Installer refuses to install a non-FIPS Image on a FIPS system

Environment

  • RHEL 7 FIPS

Add a Github Action to run validation and unit testing on PRs

Example workflow:

# This is a basic workflow to help you get started with Actions

name: CI

# Controls when the action will run. 
on:
  # Triggers the workflow on push or pull request events but only for the main branch
  push:
    branches: [ main ]
  pull_request:
    branches: [ main ]

  # Allows you to run this workflow manually from the Actions tab
  workflow_dispatch:

# A workflow run is made up of one or more jobs that can run sequentially or in parallel
jobs:
  spec:
    name: "Spec Tests"
    runs-on: ubuntu-20.04
    strategy:
      fail-fast: false
      matrix:
        include:
          - puppet_ver: '~> 6'
            ruby_ver: 2.5
          - puppet_ver: '~> 7'
            ruby_ver: 2.7
    steps:
      - name: Checkout Source
        uses: actions/checkout@v2

      - name: Activate Ruby ${{ matrix.ruby_ver }}
        uses: ruby/setup-ruby@v1
        env:
          PUPPET_GEM_VERSION: ${{ matrix.puppet_ver }}
        with:
          ruby-version: ${{ matrix.ruby_ver }}
          bundler-cache: true

      - name: Print Bundler Environment
        env:
          PUPPET_GEM_VERSION: ${{ matrix.puppet_ver }}
        run: |
          bundle env

      - name: Code Validation
        env:
          PUPPET_GEM_VERSION: ${{ matrix.puppet_ver }}
        run: |
          bundle exec rake check:symlinks check:git_ignore check:dot_underscore check:test_file rubocop syntax lint metadata_lint

      - name: Parallel Spec Tests
        env:
          PUPPET_GEM_VERSION: ${{ matrix.puppet_ver }}
        run: |
          bundle exec rake parallel_spec

During upgrade plan do not stop the pe-postgresql service on the Primary Master

Describe the Bug

Currently the peadm::upgrade plan stops the 'pe-postgresql' service on the Primary Master before executing the puppet-enterprise-installer. However, during the upgrade one of the first steps is to backup the Postgresql instance which requires that the service is running.

# Shut down pe-* services on the master. Only shutting down the ones
# that have failover pairs on the master replica.
['pe-console-services', 'pe-nginx', 'pe-puppetserver', 'pe-puppetdb', 'pe-postgresql'].each |$service| {
  run_task('service', $master_target,
    action => 'stop',
    name   => $service,
  )
}

Expected Behavior

Remove 'pe-postgresql' from the list of services that are being stopped before the upgrade is initiated.

Steps to Reproduce

Steps to reproduce the behavior:

  1. Start with a deployed instance of PE, running at >=2019.0
  2. Execute upgrade plan
  3. Failure during puppet-enterprise-installer task, due to pe-postgresql service being stopped
run_task('peadm::pe_install', $master_target,
  tarball => $upload_tarball_path,
)

Environment

  • Version = 2019.0
  • Platform Ubuntu 18.04

Additional Context

plan fails to install agent on agent_installer_targets

Describe the Bug

When I use plan in Vagrant environment, standard DR with primary/replica is deployed without issues.
Same plan, but targets in VMware environment, fail during agent install on the replica:

 The task failed with exit code 1 and no stdout, but stderr contained:
    % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                   Dload  Upload   Total   Spent    Left  Speed
100 26193  100 26193    0     0   752k      0 --:--:-- --:--:-- --:--:--  752k
  E: 'curl --cacert /etc/puppetlabs/puppet/ssl/certs/ca.pem -o /tmp/tmp.7BTJ4LYjrj -L --write-out %{http_code} -s https://primary.example.com:8140/packages/2021.5.0/el-8-x86_64.bash ' failed (exit_code: 60)
  Error code 60 attempting to reach https://primary.example.com:8140/packages/2021.5.0/el-8-x86_64.bash

The only 'moving' part here is /etc/puppetlabs/puppet/ssl/certs/ca.pem, so for an experiment, I stubbed agent with ca.pem before the install step:

diff --git a/plans/subplans/install.pp b/plans/subplans/install.pp
index 41354d2..0bee25c 100644
--- a/plans/subplans/install.pp
+++ b/plans/subplans/install.pp
@@ -350,6 +350,10 @@ plan peadm::subplans::install (
     run_task('peadm::puppet_runonce', $database_targets)
   }
 
+  $ca_pem = run_task('peadm::read_file', $primary_target,
+    path => '/etc/puppetlabs/puppet/ssl/certs/ca.pem',
+  ).first['content']
+
   parallelize($agent_installer_targets) |$target| {
     $common_install_flags = [
       '--puppet-service-ensure', 'stopped',
@@ -357,6 +361,13 @@ plan peadm::subplans::install (
       "main:certname=${target.peadm::certname()}",
     ]
 
+    # Stub agent with CA
+    run_task('peadm::mkdir_p_file', $target,
+      path    => '/etc/puppetlabs/puppet/ssl/certs/ca.pem',
+      mode    => '0644',
+      content => $ca_pem,
+    )
+
     # Get an agent installed and cert signed
     run_task('peadm::agent_install', $target,
       server        => $primary_target.peadm::certname(),

agent then installs successfully

I would be happy to provide more information. I tried PE2019.8.9 installer with the same results

Environment

  • Version "3.7.0
  • Platform RHEL8

Ability to provision using an existing cert for recovery [SOLARCH-544]

Use Case

Ability to take an SSL tar.gz directory and import during provision phase for the sake of DR.

Describe the Solution You Would Like

Pass a file or location of a file to download and expand into the ssl directory prior to the installation process to ensure the generated certs for the cluster are from the same CA

Describe Alternatives You've Considered

Outside of the plan do the same above task using something like a parent plan or terraform remote-exec to expand into the directory prior to installation.

Additional Context

Geo located cluster for redundancy without depending on existing cluster. Quick fail over in a true DR scenario.

no documentation for the configuration post-promote

Describe the Change You Would Like

That the configuration created by provision is invalid after promoting:

root@pe-201901-replica-xl:~# puppet node purge pe-201901-master-xl.platform9.puppet.net
Error: Failed to open TCP connection to pe-201901-master-xl.platform9.puppet.net:8140 (Connection refused - connect(2) for "pe-201901-master-xl.platform9.puppet.net" port 8140)
Error: Try 'puppet help node purge' for usage

Resolved via documentation, or plans to reconfigure following a promotion and provisioning a new Replica.

GPG signature verification fails on download for older PE versions

Describe the Bug

Puppet changed their signing key in 2020. Therefore, if you try to install a version of PE that predates that signing key being used, the default behavior of peadm::download won't be sufficient to verify it. Because the older signing key won't be installed, which is the key that was used to sign the older package.

Finished: plan peadm::install in 42.17 sec
Failed on pe-lg.puppet.vm:
  The task failed with exit code 1
  Downloading: https://s3.amazonaws.com/pe-builds/released/2019.8.1/puppet-enterprise-2019.8.1-el-7-x86_64.tar.gz.asc
  Temporary file created at: /tmp/tmp.gHYmklkIwi
  Verifying /tmp/puppet-enterprise-2019.8.1-el-7-x86_64.tar.gz...
  Signature verification failed
  Downloading: https://s3.amazonaws.com/pe-builds/released/2019.8.1/puppet-enterprise-2019.8.1-el-7-x86_64.tar.gz
  Temporary file created at: /tmp/tmp.IZxjQVsW9T
  Verifying /tmp/puppet-enterprise-2019.8.1-el-7-x86_64.tar.gz...
  pub   4096R/9E61EF26 2019-04-08 [expires: 2025-04-06]
  uid                  Puppet, Inc. Release Key (Puppet, Inc. Release Key) <[email protected]>
  sub   4096R/9F057A83 2019-04-08 [expires: 2025-04-06]
  
  Signature verification failed
Failed on 1 target: pe-lg.puppet.vm

It works if you pre-install the old key.

gpg --keyserver hkp://keyserver.ubuntu.com:11371 --recv-key 7F438280EF8D349F

Expected Behavior

The peadm::install plan should run, out of the box, for all supported PE versions. It shouldn't be necessary to perform any special pre-install steps.

Steps to Reproduce

Try to install PE 2019.8.1 on a target that has gpg already installed.

Additional Context

We don't actually care about signature validation by default, we really just wanted digest validation. Suggested solution is to make signature validation opt-in, with options to supply the signing key. By default, don't do gpg validation. It's overkill. See if simple digest validation can be achieved, skip it if not.

Testing new Github alert (hook)

Use Case

A clear and concise description of what the problem you're trying to solve is. Ex. I'm always frustrated when [...]

Describe the Solution You Would Like

A clear and concise description of what you want to happen.

Describe Alternatives You've Considered

A clear and concise description of any alternative solutions or features you've considered.

Additional Context

Add any other context or screenshots about the feature request here.

When PE installer fails on upgrade, upgrade plan should halt

Describe the Bug

When the upgrade plan runs and the installer fails, the upgrade does not stop.

Expected Behavior

The upgrade should stop when the PE installer emits a failed exit code.

Additional Context

Because in the Extra Large architecture there are circumstances in which the installer is expected to fail, be careful in fixing this that for expected installer failures, plans still continue.

The pe_install task should be updated with failure condition parameters, probably.

if [ ! -z "$PT_peconf" ]; then
/bin/bash "${tgzdir}/${pedir}/puppet-enterprise-installer" -y -c "$PT_peconf"
else
/bin/bash "${tgzdir}/${pedir}/puppet-enterprise-installer" -y
fi

The peadm::upgrade plan can time out when applied to PE XL installs that need to migrate PuppetDB reports [SOLARCH-548]

Describe the Bug

When upgrading a PE XL install older than 2019.7.0 the peadm::upgrade plan has to migrate PuppetDB report data on both the primary and replica nodes. The second migration runs under the control of puppet infrastructure upgrade replica which assumes that the reports and resource_events tables have been emptied and that a copy of the freshly migrated data can be obtained from the primary node using pg_basebackup. These assumptions are not met by peadm:upgrade:

  • As a result of PR #129, the upgrade replica step that empties the reporting tables is turned into a no-op. This results in the the upgrade process on the replica hitting timeouts when pe-puppetdb is started with fully loaded tables:
    • The service start occurs during a puppet agent run, which now has a default runtimeout = 1h in Puppet 6.
    • The service start timeout for pe-puppetdb has been lowered back to 900 minutes under the assumption that long-running migrations are done by the puppetdb upgrade command.
  • Running pg_basebackup on the replica node and pointing it at the primary node does not have the intended result as the migrated PuppetDB database lives on the primary puppetdb postgres node and needs to be copied to the replica puppetdb postgres node.

Expected Behavior

peadm::upgrade is able to upgrade installations older than PE 2019.7.0 without timing out.

Steps to Reproduce

Steps to reproduce the behavior:

  1. Install PE 2018.1 in an XL configuration.
  2. Run peadm::convert on the installation.
  3. Add a reporting history for about 10,000 nodes generating about 30 million events to the installation.
  4. Use peadm::upgrade to upgrade the installation to the latest 2019.8 version.

Environment

  • Version: PE 2018.1 or newer.
  • Platform: Any OS supported as a primary master platform by PE 2019.8.

Preflight check: LANG environment variable

Use Case

When LANG is set to an unsupported value, such as C.utf8, various parts of PEAdm will cryptically fail.

Describe the Solution You Would Like

The PEAdm preflight checks should check what LANG is set to, and print a warning if it is set to an unexpected/unsupported value. This will help users debug strange LANG-related issues that would otherwise be very difficult to identify.

add Litmus support and spec tests

Use Case

Litmus support would be a valuable addition to PEADM and Litmus tasks could then be used for preflight and post-flight sanity which could form the bases of health checks and pe services

Describe the Solution You Would Like

first step update to module to impliment litmus and litmus config requirements.
Additional but not necessery more to record for future tickets

  • litmus serverspec for preflight check
  • limus serverspec for post flight checks
  • litmus tests for all tasks/classification

Describe Alternatives You've Considered

A clear and concise description of any alternative solutions or features you've considered.

Additional Context

Add any other context or screenshots about the feature request here.

peadm::restore fails when puppet-db export times out

Describe the Bug

When restoring a backup made with peadm::backup that includes a dump of pe-puppetdb, the restore plan uses puppet-db export to save more recent data. In even a modestly-sized installation this export fails consistently for me, causing the plan to fail.

Expected Behavior

I expected the restore plan to disregard the contents of PuppetDB on the target system and simply restore the pg_dump dump made by peadm::backup. If I remove the run_command calls to puppet-db then this works just as I expect.

Steps to Reproduce

  1. Make a backup with peadm::backup::backup['puppetdb'] set to true.
  2. Restore the backup onto a target with a working and running PuppetDB.

Environment

RHEL 7, XL with replica, two compilers, three agents.

Additional Context

I can see the benefit of exporting PDB, but I think it's undesirable for it to be necessarily tied to whether the plan restores the PDB database dump.

add autosign configuration support

Use Case

configure autosign

Describe the Solution You Would Like

  1. Add https://forge.puppet.com/danieldreier/autosign to the dependencies
  2. Add an apply plan to add the autosign module to CA node group and provide configuration data optional flag to print config data in hiera format with instructions to update hiera in Control_repo
  3. Add a pl

an to run autosign:generate and agent install with created token.

Describe Alternatives You've Considered

NA

Additional Context

NA

Code sync undefined method

Describe the Bug

While running code sync on a test cluster it threw an undeclared method statement

Expected Behavior

It should run cleanly

Steps to Reproduce

Steps to reproduce the behavior:

  1. Go to '...'
  2. Click on '....'

Environment

  • 2019.8.9
  • to be confirmed
  • gcp

Additional Context

(gcp-3.9.1) ➜ pe_large_201989 git:(gavindidrichsen/pr/add_sudo_root_to_sshcondfig) ✗ pbis

  • bolt inventory show
    Targets
    pe-server-7c3d32-0.us-west1-c.c.customer-support-scratchpad.internal
    pe-compiler-7c3d32-0.us-west1-c.c.customer-support-scratchpad.internal
    agnt89-beaad0-0.us-west1-c.c.customer-support-scratchpad.internal
    agnt89-beaad0-1.us-west1-b.c.customer-support-scratchpad.internal
    agnt89-beaad0-2.us-west1-a.c.customer-support-scratchpad.internal
    dashboard89-beaad0-0.us-west1-c.c.customer-support-scratchpad.internal

Inventory source
/Users/gavin.didrichsen/Documents/@REFERENCE/github/app/tools/puppet/lab_environments/gcp/deployments/pe_large_201989/inventory.yaml

Target count
6 total, 6 from inventory, 0 adhoc

Additional information
Use the '--targets', '--query', or '--rerun' option to view specific targets
Use the '--detail' option to view target configuration and data

  • set +x
    (gcp-3.9.1) ➜ pe_large_201989 git:(gavindidrichsen/pr/add_sudo_root_to_sshcondfig) ✗ pbtr peadm::code_sync_status --targets=pe-server-7c3d32-0.us-west1-c.c.customer-support-scratchpad.internal,pe-compiler-7c3d32-0.us-west1-c.c.customer-support-scratchpad.internal
  • bolt task run peadm::code_sync_status --targets=pe-server-7c3d32-0.us-west1-c.c.customer-support-scratchpad.internal,pe-compiler-7c3d32-0.us-west1-c.c.customer-support-scratchpad.internal
    Started on pe-server-7c3d32-0.us-west1-c.c.customer-support-scratchpad.internal...
    Started on pe-compiler-7c3d32-0.us-west1-c.c.customer-support-scratchpad.internal...
    Finished on pe-server-7c3d32-0.us-west1-c.c.customer-support-scratchpad.internal:
    {
    "development": {
    "latest_commit": "0d86318140145a1d669679a35eb6a56ec9407556",
    "servers": {
    "pe-server-7c3d32-0.us-west1-c.c.customer-support-scratchpad.internal": {
    "commit": "0d86318140145a1d669679a35eb6a56ec9407556",
    "sync": true
    },
    "pe-compiler-7c3d32-0.us-west1-c.c.customer-support-scratchpad.internal": {
    "commit": "0d86318140145a1d669679a35eb6a56ec9407556",
    "sync": true
    },
    "pe-server-7c3d32-0.us-west1-c.c.customer-support-scratchpad.internal-orchestrator": {
    "commit": "0d86318140145a1d669679a35eb6a56ec9407556",
    "sync": true
    },
    "agnt89-beaad0-0.us-west1-c.c.customer-support-scratchpad.internal": {
    "commit": "0d86318140145a1d669679a35eb6a56ec9407556",
    "sync": true
    }
    },
    "sync": true
    },
    "production": {
    "latest_commit": "0d86318140145a1d669679a35eb6a56ec9407556",
    "servers": {
    "pe-server-7c3d32-0.us-west1-c.c.customer-support-scratchpad.internal": {
    "commit": "0d86318140145a1d669679a35eb6a56ec9407556",
    "sync": true
    },
    "pe-compiler-7c3d32-0.us-west1-c.c.customer-support-scratchpad.internal": {
    "commit": "0d86318140145a1d669679a35eb6a56ec9407556",
    "sync": true
    },
    "pe-server-7c3d32-0.us-west1-c.c.customer-support-scratchpad.internal-orchestrator": {
    "commit": "0d86318140145a1d669679a35eb6a56ec9407556",
    "sync": true
    },
    "agnt89-beaad0-0.us-west1-c.c.customer-support-scratchpad.internal": {
    "commit": "0d86318140145a1d669679a35eb6a56ec9407556",
    "sync": true
    }
    },
    "sync": true
    },
    "sync": true
    }
    Failed on pe-compiler-7c3d32-0.us-west1-c.c.customer-support-scratchpad.internal:
    The task failed with exit code 1 and no stdout, but stderr contained:
    /tmp/a5a8a6ce-8b98-41fb-9956-3128d2ef388b/code_sync_status.rb:91:in sync_status': undefined method []' for nil:NilClass (NoMethodError)
    from /tmp/a5a8a6ce-8b98-41fb-9956-3128d2ef388b/code_sync_status.rb:16:in execute!' from /tmp/a5a8a6ce-8b98-41fb-9956-3128d2ef388b/code_sync_status.rb:112:in '
    Successful on 1 target: pe-server-7c3d32-0.us-west1-c.c.customer-support-scratchpad.internal
    Failed on 1 target: pe-compiler-7c3d32-0.us-west1-c.c.customer-support-scratchpad.internal
    Ran on 2 targets in 8.84 sec
  • set +x
    (gcp-3.9.1) ➜ pe_large_201989 git:(gavindidrichsen/pr/add_sudo_root_to_sshcondfig) ✗

HA upgrade sequence

Currently, while upgrading HA installation with only master/replica, plan uses installer script to upgrade the master and then proceeded upgrading replica using curl/upgrade.sh

Unfortunately, installer always unconfigures replica's access and while upgrading replica it fails to connect.

2020-03-14T14:32:13.262-04:00 - [Notice]: /Stage[main]/Pe_postgresql::Server::Config/Pe_concat[/opt/puppetlabs/server/data/postgresql/9.6/data/pg_hba.conf]/File[/opt/puppetlabs/server/data/postgresql/9.6/data/pg_hba.conf]/content:
--- /opt/puppetlabs/server/data/postgresql/9.6/data/pg_hba.conf 2020-02-08 06:24:19.240036914 -0500
+++ /tmp/puppet-file20200314-7669-kgq9h2        2020-03-14 14:32:13.252528149 -0400
@@ -20,11 +20,6 @@
 # Order: 0
 hostssl        pe-orchestrator pe-ha-replication       0.0.0.0/0       cert    map=pe-orchestrator-pe-ha-replication-map clientcert=1

-# Rule Name: Allow infprdx-puppet101.localdomain to connect to pe-puppetdb as pe-puppetdb (ipv4)
-# Description: none
-# Order: 0
-hostssl        pe-puppetdb     pe-puppetdb     0.0.0.0/0       cert    map=pe-puppetdb-pe-puppetdb-map clientcert=1
-
 # Rule Name: Allow infprdx-puppet101.localdomain to connect to pe-rbac as pe-ha-replication (ipv4)
 # Description: none
 # Order: 0
@@ -135,11 +130,6 @@
 # Order: 1
 hostssl        pe-orchestrator pe-ha-replication       ::/0    cert    map=pe-orchestrator-pe-ha-replication-map clientcert=1

-# Rule Name: Allow infprdx-puppet101.localdomain to connect to pe-puppetdb as pe-puppetdb (ipv6)
-# Description: none
-# Order: 1
-hostssl        pe-puppetdb     pe-puppetdb     ::/0    cert    map=pe-puppetdb-pe-puppetdb-map clientcert=1
-
 # Rule Name: Allow infprdx-puppet101.localdomain to connect to pe-rbac as pe-ha-replication (ipv6)
 # Description: none
 # Order: 1
2020-03-14T14:32:13.268-04:00 - [Notice]: /Stage[main]/Pe_postgresql::Server::Config/Pe_concat[/opt/puppetlabs/server/data/postgresql/9.6/data/pg_hba.conf]/File[/opt/puppetlabs/server/data/postgresql/9.6/data/pg_hba.conf]/content: content changed '{md5}812273660d0a44470f64fe637123fbb9' to '{md5}c73330422a1ae9cc6665167a6333190d'
2020-03-14T14:32:13.381-04:00 - [Notice]: /Stage[main]/Puppet_enterprise::Profile::Database/Pe_concat[/opt/puppetlabs/server/data/postgresql/9.6/data/pg_ident.conf]/Exec[pe_concat_/opt/puppetlabs/server/data/postgresql/9.6/data/pg_ident.conf]/returns: executed successfully
2020-03-14T14:32:13.442-04:00 - [Notice]: /Stage[main]/Puppet_enterprise::Profile::Database/Pe_concat[/opt/puppetlabs/server/data/postgresql/9.6/data/pg_ident.conf]/Exec[pe_concat_/opt/puppetlabs/server/data/postgresql/9.6/data/pg_ident.conf]: Triggered 'refresh' from 1 event
2020-03-14T14:32:13.455-04:00 - [Notice]: /Stage[main]/Puppet_enterprise::Profile::Database/Pe_concat[/opt/puppetlabs/server/data/postgresql/9.6/data/pg_ident.conf]/File[/opt/puppetlabs/server/data/postgresql/9.6/data/pg_ident.conf]/content:
--- /opt/puppetlabs/server/data/postgresql/9.6/data/pg_ident.conf       2020-02-08 06:24:19.393051257 -0500
+++ /tmp/puppet-file20200314-7669-1i60s7        2020-03-14 14:32:13.446546362 -0400
@@ -2,7 +2,6 @@
 pe-activity-pe-ha-replication-map infprdx-puppet101.localdomain pe-ha-replication
 pe-classifier-pe-ha-replication-map infprdx-puppet101.localdomain pe-ha-replication
 pe-orchestrator-pe-ha-replication-map infprdx-puppet101.localdomain pe-ha-replication
-pe-puppetdb-pe-puppetdb-map infprdx-puppet101.localdomain pe-puppetdb
 pe-rbac-pe-ha-replication-map infprdx-puppet101.localdomain pe-ha-replication
 pe-activity-pe-activity-map infprdx-puppet201.localdomain pe-activity

IMHO, plan should run puppet agent on master before it attempts to upgrade the replica, otherwise after upgrade I always end up with puppet infrastructure status in faulty state.
I can attach the whole installer log if it helps

peadm should print a summary of plan parameters to the standard log stream [SOLARCH-546]

Use Case

The peadm::install and peadm::upgrade plans co-ordinate tasks across many nodes using a variety of inputs. When one of these plans fails, the output is often the first thing presented for debugging. Currently, this output lacks important information such as:

  • The peadm version being used

  • The nodes involved in the plan and their assigned roles

  • The PE version being installed.

  • The PE version being upgraded.

This lack of information leads to several follow-up questions before debugging can proceed.

Describe the Solution You Would Like

peadm plans that perform significant actions, such as install or upgrade, should log relevant parameters and version numbers using out::message so that most relevant information is captured in one log stream.

Describe Alternatives You've Considered

The current alternative is to manually gather plan inputs. This is tedious, often results in incomplete data, and can miss inconsistencies (i.e. the parameters the user thought they passed to the plan are not the parameters the plan ended up using due to inventory settings, peadm plans being called by other user-written plans, etc.).

test github slack integration again

Use Case

A clear and concise description of what the problem you're trying to solve is. Ex. I'm always frustrated when [...]

Describe the Solution You Would Like

A clear and concise description of what you want to happen.

Describe Alternatives You've Considered

A clear and concise description of any alternative solutions or features you've considered.

Additional Context

Add any other context or screenshots about the feature request here.

hostname task fails if FQDN provided in inventory/params

Given that a FQDN is provided in the inventory and the params.json file for a role, the pe_xl::hostname task will throw an error because the hostname returned does not include the domain.

nodes.yaml snippet

...
    nodes:
      - l44n1csrf0zp9s4.delivery.puppetlabs.net
...

params.json snippet

{
...
  "master_host": "l44n1csrf0zp9s4.delivery.puppetlabs.net",
...
}

Plan fails with

...
{"node":"l44n1csrf0zp9s4.delivery.puppetlabs.net","target":"l44n1csrf0zp9s4.delivery.puppetlabs.net","action":"task","object":"pe_xl::hostname","status":"success","result":{"hostname":"l44n1csrf0zp9s4"}}
Command returned successfully
...
{
  "kind": "bolt/plan-failure",
  "msg": "Hostname / DNS name mismatch: target l44n1csrf0zp9s4.delivery.puppetlabs.net reports 'l44n1csrf0zp9s4'",
  "details": {
  }
}

I presume that the hostname task should be producing the fqdn instead of just the base hostname.

Possible update

diff --git a/tasks/hostname.sh b/tasks/hostname.sh
index bdabaf7..f64a4d8 100755
--- a/tasks/hostname.sh
+++ b/tasks/hostname.sh
@@ -1,6 +1,6 @@
 #!/bin/bash

-hostname=$(/usr/bin/hostnamectl --transient)
+hostname=$(/usr/bin/hostname -f)

 # Output a JSON result for ease of Task usage in Puppet Task Plans
 echo '{ "hostname": "'$hostname'" }'

puppet ssl submit command fails on PE LTS release

Describe the Bug

A clear and concise description of what the bug is.

bolt plan run peadm::provision fails on client verification run_command:

./modules/peadm/plans/action/install.pp: /opt/puppetlabs/bin/puppet ssl submit_request

The "ssl" option does not exist as part of the puppet command on the agent system

error is

The command failed with exit code 1
STDOUT:
Error: Unknown Puppet subcommand 'ssl'
See 'puppet help' for help on available puppet subcommands

Expected Behavior

A clear and concise description of what you expected to happen.

/opt/puppetlabs/bin/puppet ssl submit_request should execute successfully as a test that the certificate has been setup

Steps to Reproduce

Steps to reproduce the behavior:

  1. when the agent is installed on the client from the bolt server execute

bolt command run '/opt/puppetlabs/bin/puppet ssl submit_request' --targets

Environment

  • PE 2018.1 (LTZ)
  • centos 7

Additional Context

Convert needs to restart pe-puppetserver [SOLARCH-547]

Bug Description

When running peadm::convert as a pre-step to upgrade from PE 2019.2 to 2019.8, it has been observed that it is necessary to restart pe-puppetserver after running peadm::convert, or else the upgrade will not succeed. This restart of pe-puppetserver should be part of the peadm::convert plan.

Expected Behavior

Immediately after running peadm::convert, the PE deployment should be ready to run peadm::upgrade on without any manual intervention or fix steps required.

add agent switch over plan/task [SOLARCH-542]

Use Case

Module focuses on clean fresh install it would be great to also provide replatform support starting with agent switch over as 80% of the time we just build new masters anyway.

Describe the Solution You Would Like

task and simple ssh script to apply to an agent that follows cert regen and repoints at new dns entry for master/ compiler LB pool.
associated plan to handle signing new cert with flag to negate when autosign is in use.

Describe Alternatives You've Considered

puppet plan with apply based task

Additional Context

error provisioning 2019.8 with containers

Describe the Bug

When using the docker-compose containers for peadm under spec/docker a new provision of 2019.8.0 fails to complete and dumps the catalog to the STDOUT.

Expected Behavior

Successfully install

Steps to Reproduce

Steps to reproduce the behavior:

  1. go to spec/docker/standard
  2. edit params.json and update version to 2019.8.0
  3. cd spec/docker
  4. run provision.sh
  5. select standard
  6. error should occur after 5-10 minutes

Environment

  • Docker
  • CentOS7
  • Bolt 2.19

Additional Context

...
Starting: apply catalog on pe-lg.puppet.vm
Finished: apply catalog with 1 failure in 6.34 sec
Finished: plan peadm::action::configure in 7.06 sec
Finished: plan peadm::provision in 8 min, 16 sec
Failed on pe-lg.puppet.vm:
  Report result contains an '_output' key. Catalog application may have printed extraneous output to stdout: {"catalog":{"tags":["settings","peadm::setup::node_manager_yaml","peadm","setup","node_manager_yaml","peadm::setup::node_manager","node_manager","class"],"name":"pe-lg.puppet.vm","version":1596645286,"code_id":null,"catalog_uuid":"eb1e06d9-460e-4861-8707-a7f5a5fcab6f","catalog_format":1,"environment":"bolt_catalog","resources":[{"type":"Stage","title":"main","tags":["stage","class"],"exported":false,"parameters":{"name":"main"}},{"type":"Class","title":"Settings","tags":["class","settings"],"exported":false},{"type":"Class","title":"main","tags":["class"],"exported":false,"parameters":{"name":"main"}},{"type":"Class","title":"Peadm::Setup::Node_manager_yaml","tags":["class","peadm::setup::node_manager_yaml","peadm","setup

document haproxy load balancer classes

Use Case

The existing classes are hidden :(

Describe the Solution You Would Like

The existing classes are documented.

Describe Alternatives You've Considered

Reinventing the wheel and manually setting up a load balancer.

Upgrade deletes class parameters from PE Master (other groups too?)

Bug Description

When running peadm::upgrade, any class parameter configuration added by the user to PE Master, such as code_manager_auto_configure=true, will be deleted.

Expected Behavior

User configuration of class parameters in PE Master and other built-in groups should be preserved by peadm::upgrade.

Allow overriding CSR attribute settings

Use Case

peadm is opinionated in the use of CSR attributes. Some customers have existing use for these attributes and the ability to override and add attributes would be beneficial.

Describe the Solution You Would Like

The ability to specify attributes and the values used for each host in a deployment.

Describe Alternatives You've Considered

Forking the project and changing the current hard-coded values for another set. But this creates something that requires on-going maintenance effort to keep current with changes.

peadm::status plan fails

peadm::status plan fails in PE2019.8.0

$ bolt plan run peadm::status -t puppet.chepkov.lan
Starting: plan peadm::status
Starting: task peadm::infrastatus on puppet.chepkov.lan
Finished: task peadm::infrastatus with 0 failures in 10.5 sec
Finished: plan peadm::status in 10.51 sec
{
  "kind": "bolt/pal-error",
  "msg": "Evaluation Error: Operator '[]' is not applicable to an Undef Value. (file: /Bolt/modules/peadm/plans/status.pp, line: 20, column: 13)",
  "details": {
    "file": "/Bolt/modules/peadm/plans/status.pp",
    "line": 20,
    "column": 13
  }
}

After examining the line in question, I found that if I replace this code

$data = $item[result][output]

with

$data = $item[value][output]

plan works as expected

$ bolt plan run peadm::status -t puppet.chepkov.lan
Starting: plan peadm::status
Starting: task peadm::infrastatus on puppet.chepkov.lan
Finished: task peadm::infrastatus with 0 failures in 10.5 sec
+---------------------------------------+--------------------------------------+
|                         Overall Status: operational                          |
+---------------------------------------+--------------------------------------+
| Stack                                 | Status                               |
+---------------------------------------+--------------------------------------+
| puppet.chepkov.lan                    | operational                          |
+---------------------------------------+--------------------------------------+
Finished: plan peadm::status in 10.61 sec
Plan completed successfully with no result

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.