bodgit / puppet-zfs Goto Github PK

View Code? Open in Web Editor NEW

6.0 3.0 10.0 299 KB

Puppet Module for managing ZFS

Home Page: https://forge.puppet.com/bodgit/zfs

License: Apache License 2.0

Ruby 57.28% Puppet 39.79% Shell 0.07% HTML 2.86%

puppet zfs zpool zpool-scrub

puppet-zfs's Introduction

zfs

Description
Setup - The basics of getting started with zfs
Usage - Configuration options and additional functionality
Reference - An under-the-hood peek at what the module is doing and how
Limitations - OS compatibility, etc.
Development - Guide for contributing to the module

Description

This module currently ensures that the ZFS packages available from zfsonlinux.org are installed and configured.

RHEL/CentOS, Ubuntu and Debian are supported using Puppet 5 or later. On RHEL/CentOS platforms there is support for installing the kABI-tracking kernel modules as opposed to the default of DKMS-style kernel modules.

Setup

What zfs affects

This module will install kernel modules which utilises the DKMS framework to accomplish this. This means kernel headers, toolchains, etc. will be installed.

Setup Requirements

You will need pluginsync enabled. On RHEL/CentOS platforms you will need to have access to the EPEL repository by using puppet/epel or by other means to use the DKMS-style kernel modules. On Debian you will need to enable backports using puppetlabs/apt with something like:

class { 'apt::backports':
  repos  => 'main contrib',
  pin    => 990,
  before => Class['zfs'],
}

Beginning with zfs

In the very simplest case, you can just include the following:

include zfs

Usage

For example on RHEL/CentOS to instead install the kABI-tracking kernel modules and tune the ARC, you can do:

class { 'zfs':
  kmod_type   => 'kabi',
  zfs_arc_max => to_bytes('256 M'),
  zfs_arc_min => to_bytes('128 M'),
}

To also install the ZFS Event Daemon (zed):

include zfs
include zfs::zed

Reference

The reference documentation is generated with puppet-strings and the latest version of the documentation is hosted at https://bodgit.github.io/puppet-zfs/ and available also in the REFERENCE.md.

Limitations

This module has been built on and tested against Puppet 5 and higher.

The module has been tested on:

Red Hat/CentOS Enterprise Linux 6/7/8
Ubuntu 16.04/18.04/20.04
Debian 9/10

Development

The module relies on PDK and has both rspec-puppet and Litmus tests. Run them with:

$ bundle exec rake spec
$ bundle exec rake litmus:*

Please log issues or pull requests at github.

puppet-zfs's People

Contributors

Stargazers

Watchers

Forkers

russell-io dynamikemiller neatnerdprime john-saxton optiz0r kscherer sergiik runejuhl wmoore28 pfhonore

puppet-zfs's Issues

Convert to Litmus

Add support for Ubuntu 18.04

Add puppetlabs/zfs_core as a dependency

Now that all the non-core resources have been split out, does it make sense to helpfully pull in https://github.com/puppetlabs/puppetlabs-zfs_core as a dependency on this module?

Use service_provider fact from stdlib

Fix APT module dependency

It's pinned at '>=1.4.2 <2.0.0' which is from the initial release, it probably works fine as '>=1.4.2 <3.0.0' as the backwards-incompatible changes shouldn't affect this modules usage. Test on Debian-ish platforms.

Drop old Puppet versions

A lot of Puppetlabs modules are using 5.5.10 as the oldest version supported. Save this for a major version bump.

Does not install the kernel modules on Debian

Hi and thanks for your work on this module.

I notice that it doesn't actually install the zfs-dkms package on Debian 9, which would be really useful.

Add support for Debian 10

Convert to PDK

Set up zpool scrub cron job

It's generally a good idea to run zpool scrub <pool> periodically, it should be straightforward to manage a cron job to do this.

Don't configure PPA on Ubuntu 16.04+

The PPA is only needed on 12.04 (see #17 though) and 14.04. There are native ZFS packages in the main repositories for 16.04+.

Debian 9

Conditional on finding a usable Vagrant box for acceptance tests.

Drop support for old OS versions

I don't even think the packages for precise/12.04 are available any longer. I can only see packages as far back as trusty/14.04. When 14.04 goes away, there's actually no need to depend directly on the apt module, 16.04+ shouldn't install the PPA.

Add ability to tune ARC size

This is simply a case of defining suitable module options:

options zfs zfs_arc_min=XXX zfs_arc_max=YYY

ZFS init fails on Centos due to not loaded module

I would like to check with you on following issue.
When the Puppet ZFS initializes on CentOS 7.7 (fresh system), it fails with zfs-import-scan due to not loaded zfs kernel module.

Do you have an idea how I can do the ZFS initial install in a way to workaround this issue? I see on Ubuntu youre explicitly loading the kernel module. I guess ZFS module will only be loaded automatically if a ZFS volume already exists on the server?

What I wonder is, if this might be a new issue related to Centos 7.7 update, new ZFS 0.8.3 behaviour or new version of zfs puppet module. Because previously, this has worked automatically.

So this part already fails:
class { '::zfs': kmod_type => 'dkms', zfs_arc_max => to_bytes('16 G'), zfs_arc_min => to_bytes('1 G'), }

With following error:
2020-02-03 15:38:13 +0100 /Stage[main]/Zfs::Install/Package[kernel-devel]/ensure (notice): created 2020-02-03 15:38:14 +0100 /Stage[main]/Fail2ban::Service/Service[fail2ban]/ensure (notice): ensure changed 'stopped' to 'running' (corrective) 2020-02-03 15:39:41 +0100 /Stage[main]/Zfs::Install/Package[zfs]/ensure (notice): created 2020-02-03 15:39:41 +0100 Puppet (err): Systemd start for zfs-import-scan failed! journalctl log for zfs-import-scan: -- Logs begin at Mon 2020-02-03 15:37:44 CET, end at Mon 2020-02-03 15:39:41 CET. -- Feb 03 15:39:41 servername systemd[1]: Starting Import ZFS pools by device scanning... Feb 03 15:39:41 servername zpool[28269]: The ZFS modules are not loaded. Feb 03 15:39:41 servername zpool[28269]: Try running '/sbin/modprobe zfs' as root to load them. Feb 03 15:39:41 servername systemd[1]: zfs-import-scan.service: main process exited, code=exited, status=1/FAILURE Feb 03 15:39:41 servername systemd[1]: Failed to start Import ZFS pools by device scanning. Feb 03 15:39:41 servername systemd[1]: Unit zfs-import-scan.service entered failed state. Feb 03 15:39:41 servername systemd[1]: zfs-import-scan.service failed. 2020-02-03 15:39:41 +0100 /Stage[main]/Zfs::Service/Service[zfs-import-scan]/ensure (err): change from 'stopped' to 'running' failed: Systemd start for zfs-import-scan failed! journalctl log for zfs-import-scan: -- Logs begin at Mon 2020-02-03 15:37:44 CET, end at Mon 2020-02-03 15:39:41 CET. -- Feb 03 15:39:41 servername systemd[1]: Starting Import ZFS pools by device scanning... Feb 03 15:39:41 servername zpool[28269]: The ZFS modules are not loaded. Feb 03 15:39:41 servername zpool[28269]: Try running '/sbin/modprobe zfs' as root to load them. Feb 03 15:39:41 servername systemd[1]: zfs-import-scan.service: main process exited, code=exited, status=1/FAILURE Feb 03 15:39:41 servername systemd[1]: Failed to start Import ZFS pools by device scanning. Feb 03 15:39:41 servername systemd[1]: Unit zfs-import-scan.service entered failed state. Feb 03 15:39:41 servername systemd[1]: zfs-import-scan.service failed. 2020-02-03 15:39:41 +0100 /Stage[main]/Zfs::Service/Service[zfs-mount] (notice): Dependency Service[zfs-import-scan] has failures: true 2020-02-03 15:39:41 +0100 /Stage[main]/Zfs::Service/Service[zfs-mount] (warning): Skipping because of failed dependencies 2020-02-03 15:39:41 +0100 /Stage[main]/Zfs::Service/Service[zfs-share] (warning): Skipping because of failed dependencies 2020-02-03 15:39:41 +0100 /Stage[main]/Profile::Redhat/Exec[zfs-bootstrap.sh] (warning): Skipping because of failed dependencies 2020-02-03 15:39:41 +0100 /Stage[main]/Profile::Redhat/Exec[zfs-swap.sh] (warning): Skipping because of failed dependencies

Centos 7.7 wrong ZFS repository configuration

The repository name is wrong (/etc/yum.repos.d/zfs.repo):

http://download.zfsonlinux.org/epel/7_7/x86_64/libnvpair1-0.8.2-1.el7.x86_64.rpm: [Errno 14] HTTP Error 404 - Not Found
Trying other mirror.
libuutil1-0.8.2-1.el7.x86_64.r FAILED

Should be http://download.zfsonlinux.org/epel/7.7/x86_64/

[root@puppet .../environments-from-git/opi_production]# wget http://download.zfsonlinux.org/epel/7_7/x86_64/libnvpair1-0.8.2-1.el7.x86_64.rpm
--2019-10-07 12:46:19--  http://download.zfsonlinux.org/epel/7_7/x86_64/libnvpair1-0.8.2-1.el7.x86_64.rpm
Resolving download.zfsonlinux.org (download.zfsonlinux.org)... 2600:1fa0:40c0:791:34da:d4b9::, 52.218.229.72
Connecting to download.zfsonlinux.org (download.zfsonlinux.org)|2600:1fa0:40c0:791:34da:d4b9::|:80... connected.
HTTP request sent, awaiting response... 404 Not Found
2019-10-07 12:46:20 ERROR 404: Not Found.

[root@puppet .../environments-from-git/opi_production]# wget http://download.zfsonlinux.org/epel/7.7/x86_64/libnvpair1-0.8.2-1.el7.x86_64.rpm
--2019-10-07 12:46:24--  http://download.zfsonlinux.org/epel/7.7/x86_64/libnvpair1-0.8.2-1.el7.x86_64.rpm
Resolving download.zfsonlinux.org (download.zfsonlinux.org)... 2600:1fa0:40c0:791:34da:d4b9::, 52.218.229.72
Connecting to download.zfsonlinux.org (download.zfsonlinux.org)|2600:1fa0:40c0:791:34da:d4b9::|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 32652 (32K) [application/x-rpm]
Saving to: ‘libnvpair1-0.8.2-1.el7.x86_64.rpm’

libnvpair1-0.8.2-1.el7.x86_64.rpm                                          100%[=======================================================================================================================================================================================>]  31.89K   187KB/s    in 0.2s

2019-10-07 12:46:24 (187 KB/s) - ‘libnvpair1-0.8.2-1.el7.x86_64.rpm’ saved [32652/32652]

Update support for Puppet 6.x

Restrictive dependency on puppetlabs/apt

The dependency on puppetlabs-apt<5 is blocking stdlib being updated beyond 4.25.1. stdlib only officially supports puppet 6 from v5.1 and newer versions contain useful fixes.

Tests pass when the module dependency is updated to <8, however since I don't use this module on any apt-based systems, I can't provide any more testing than that.

Switch from params.pp to Hiera in module

Acceptance tests broken on Debian 8

Jessie backports have been removed from the main mirrors so you have to use the URL http://archive.debian.org/debian.

See https://unix.stackexchange.com/questions/508724/failed-to-fetch-jessie-backports-repository

CentOS 7.4

I have found that with the current kernel from 7.4 that a new zfs-release file is needed specific to that version:
http://download.zfsonlinux.org/epel/zfs-release.el7_4.noarch.rpm

I see that in manifests/install.pp the zfs-release file of zfs-release.el7.noarch.rpm is installed, but I'm finding that the 7_4 version is needed now with the current '3.10.0-693' kernels.

$_source = "http://download.zfsonlinux.org/epel/zfs-release.el${::operatingsystemmajrelease}.noarch.rpm"

Can you think of a good way to handle this other than write an if statement (if version is >= 7.3, then install the _$facts['os']['release']['minor]' version?

Manage ZFS Yum repos directly

Using my Yum module with repo purging enabled, unless you define the yumrepo resources, it will purge away the ZFS repos.

ZFS init fails on Centos due to not loaded module

I would like to get your opinion about following issue.
When the Puppet ZFS initializes on CentOS 7.7 (fresh system), it fails with zfs-import-scan due to not loaded zfs kernel module.

Do you have an idea how I can do the ZFS initial install in a way to workaround this issue? I see on Ubuntu or Centos 6 you are explicitly loading the kernel module. I guess ZFS module will only be loaded automatically on Centos7 if a ZFS volume already exists on the server?

What I wonder is, if this might be a new issue related to Centos 7.7 update, new ZFS 0.8.3 behaviour or new version of zfs puppet module. Because previously, the kernel module was loaded automatically and the zfs puppet init completed fine without manual intervention.

So this part already fails:
class { '::zfs': kmod_type => 'dkms', zfs_arc_max => to_bytes('16 G'), zfs_arc_min => to_bytes('1 G'), }

With following error "The ZFS modules are not loaded":

2020-02-03 15:38:13 +0100 /Stage[main]/Zfs::Install/Package[kernel-devel]/ensure (notice): created
2020-02-03 15:38:14 +0100 /Stage[main]/Fail2ban::Service/Service[fail2ban]/ensure (notice): ensure changed 'stopped' to 'running' (corrective)
2020-02-03 15:39:41 +0100 /Stage[main]/Zfs::Install/Package[zfs]/ensure (notice): created
2020-02-03 15:39:41 +0100 Puppet (err): Systemd start for zfs-import-scan failed!
journalctl log for zfs-import-scan:
-- Logs begin at Mon 2020-02-03 15:37:44 CET, end at Mon 2020-02-03 15:39:41 CET. --
Feb 03 15:39:41 servername systemd[1]: Starting Import ZFS pools by device scanning...
Feb 03 15:39:41 servername zpool[28269]: The ZFS modules are not loaded.
Feb 03 15:39:41 servername zpool[28269]: Try running '/sbin/modprobe zfs' as root to load them.
Feb 03 15:39:41 servername systemd[1]: zfs-import-scan.service: main process exited, code=exited, status=1/FAILURE
Feb 03 15:39:41 servername systemd[1]: Failed to start Import ZFS pools by device scanning.
Feb 03 15:39:41 servername systemd[1]: Unit zfs-import-scan.service entered failed state.
Feb 03 15:39:41 servername systemd[1]: zfs-import-scan.service failed.
2020-02-03 15:39:41 +0100 /Stage[main]/Zfs::Service/Service[zfs-import-scan]/ensure (err): change from 'stopped' to 'running' failed: Systemd start for zfs-import-scan failed!
journalctl log for zfs-import-scan:
-- Logs begin at Mon 2020-02-03 15:37:44 CET, end at Mon 2020-02-03 15:39:41 CET. --
Feb 03 15:39:41 servername systemd[1]: Starting Import ZFS pools by device scanning...
Feb 03 15:39:41 servername zpool[28269]: The ZFS modules are not loaded.
Feb 03 15:39:41 servername zpool[28269]: Try running '/sbin/modprobe zfs' as root to load them.
Feb 03 15:39:41 servername systemd[1]: zfs-import-scan.service: main process exited, code=exited, status=1/FAILURE
Feb 03 15:39:41 servername systemd[1]: Failed to start Import ZFS pools by device scanning.
Feb 03 15:39:41 servername systemd[1]: Unit zfs-import-scan.service entered failed state.
Feb 03 15:39:41 servername systemd[1]: zfs-import-scan.service failed.
2020-02-03 15:39:41 +0100 /Stage[main]/Zfs::Service/Service[zfs-mount] (notice): Dependency Service[zfs-import-scan] has failures: true
2020-02-03 15:39:41 +0100 /Stage[main]/Zfs::Service/Service[zfs-mount] (warning): Skipping because of failed dependencies
2020-02-03 15:39:41 +0100 /Stage[main]/Zfs::Service/Service[zfs-share] (warning): Skipping because of failed dependencies
2020-02-03 15:39:41 +0100 /Stage[main]/Profile::Redhat/Exec[zfs-bootstrap.sh] (warning): Skipping because of failed dependencies
2020-02-03 15:39:41 +0100 /Stage[main]/Profile::Redhat/Exec[zfs-swap.sh] (warning): Skipping because of failed dependencies

System: CentOS 7.7.1908
ZFS Version installed by bodgit::zfs: zfs-dkms-0.8.3-1.el7.noarch
ZFS module: mod 'bodgit-zfs', '2.2.7'

EL8 Support

CentOS 8 vagrant box
EPEL Puppet module support
ZFS packages available

Issue with DKMS and multiple kernel-header packages installed

System: CentOS 7.7

Puppet profile zfs module invocation:

    class { '::zfs':
      kmod_type   => 'dkms',
      zfs_arc_max => to_bytes('16 G'),
      zfs_arc_min => to_bytes('1 G'),
    }

When updating the kernel via regular yum update, the system will have multiple versions of kernel-header installed at the end:

# rpm -qa kernel-devel
kernel-devel-3.10.0-1062.4.1.el7.x86_64
kernel-devel-3.10.0-1062.4.3.el7.x86_64

This leads to a failure with puppet using the zfs module:

Error: Could not update: Failed to update to version 3.10.0-1062.4.3.el7.x86_64, got version 3.10.0-1062.4.1.el7; 3.10.0-1062.4.3.el7 instead
Error: /Stage[main]/Zfs::Install/Package[kernel-devel]/ensure: change from '3.10.0-1062.4.1.el7; 3.10.0-1062.4.3.el7' to '3.10.0-1062.4.3.el7.x86_64' failed: Could not update: Failed to update to version 3.10.0-1062.4.3.el7.x86_64, got version 3.10.0-1062.4.1.el7; 3.10.0-1062.4.3.el7 instead (corrective)

This is problematic, because the puppet run is failing until I remove the older kernel-devel package manually from all systems.