Detect zfs filesystem about booster HOT 33 CLOSED

Good news everyone. I have the initial version of zfs support. I was able to mount a zfs dataset as root mount and boot off it.

To try this functionality please pull and build wip branch. Then set enable_zfs: true in your booster config, then add zfs=$pool/$dataset boot parameter, where $pool and $dataset are for your root dataset. Enable debug log just in case. Then boot it. Let me know how does it work for you.

I see other systems have support for zfs=bootfs and legacy mount options but I am not sure how relevant these options are in the modern world. Is it something still actively used?

from booster.

Thanks @anatol, I will try to give it a try soon, and no I do not use any of those latest options you talked at the end personally.

from booster.

I do not use any of those latest options you talked at the end personally.

enable_zfs and zfs= options were added in my last commit. Note that ZFS support is under active development and its semantics might change.

from booster.

Here is the place where zfs toolset reads the label/uberblock structure https://github.com/openzfs/zfs/blob/e3e4c30b0a56553af643cfa5204f4dd69e3b8529/cmd/zdb/zdb.c#L4814

from booster.

I have never used zfs so I need some help with setting it up. Could you please help me to figure out what is:

1. the easiest way to create/mount zfs filesystem?
2. the standard way to setup a root filesystem just in case if it differs from the item 1.

from booster.

Are there any plans to support zfs in the future?

from booster.

Are there any plans to support zfs in the future?

Yes, I would like to see zfs support in booster. To add one a probeZfs function needs to be implemented. It is a relatively simple and straightforward code. See an example for swap partition type here d60d5df.

Could anyone provide an answer to my previous comment? I need a way to create a block device with ZFS type on it. I was expecting some sort of mkfs.zfs binary but it seems ZFS uses its own way to initialize a block device.

from booster.

Hey @anatol, in order to create some kind of ZFS block device, it is indeed true that ZFS uses its own binaries for that purpose.

You should find ways to install such a binary following this documentation : https://openzfs.github.io/openzfs-docs/Getting%20Started/index.html
There is information depending on your distro Linux on how to install ZFS's tool set.

Here is some quick info if you're afraid to break something, you can experiment kind of safely with ZFS : https://wiki.archlinux.org/title/ZFS#Experimenting_with_ZFS

(That wiki contains lots of great information, I heavily recommend you read parts of it if you're going to create a pool and ZFS filesystem that could be applied to other distros, I believe)

To give you a big picture idea, you'll need to:

1. Create a ZFS pool, inside you can have kind of like subvolumes, but they share the whole space of the partition all together, you can configure some quotas if needed, the device that can be chosen can be for a partition of fixed size, or a whole /dev/sdX device (a whole device is usually recommended but not mandatory), the tool used for this is zpool, zpool create, see here : https://wiki.archlinux.org/title/ZFS#Creating_ZFS_pools.
2. Create a ZFS filesystem inside the freshly made zpool, usually done with the command zfs create pool_name/filesystem_name.

Here are some widely used commands for zfs:

  - List all available zfs filesystems:
    zfs list

  - Create a new ZFS filesystem:
    zfs create pool_name/filesystem_name

  - Delete a ZFS filesystem:
    zfs destroy pool_name/filesystem_name

  - Create a Snapshot of a ZFS filesystem:
    zfs snapshot pool_name/filesystem_name@snapshot_name

  - Enable compression on a filesystem:
    zfs set compression=on pool_name/filesystem_name

  - Change mountpoint for a filesystem:
    zfs set mountpoint=/my/mount/path pool_name/filesystem_name

For completeness’s sake, also here is the same for zpool:

  - Show the configuration and status of all ZFS zpools:
    zpool status

  - Check a ZFS pool for errors (verifies the checksum of EVERY block). Very CPU and disk intensive:
    zpool scrub pool_name

  - List zpools available for import:
    zpool import

  - Import a zpool:
    zpool import pool_name

  - Export a zpool (unmount all filesystems):
    zpool export pool_name

  - Show the history of all pool operations:
    zpool history pool_name

  - Create a mirrored pool:
    zpool create pool_name mirror disk1 disk2 mirror disk3 disk4

  - Add a cache (L2ARC) device to a zpool:
    zpool add pool_name cache cache_disk

If there is anything else, you need, feel free to tag me on this issue, I'll do my best to help as I'm looking forward to ZFS support in order to give a try to booster.

from booster.

Thank you, this information is useful.

ZFS has some advanced features like snapshot and raid. It might be non-trivial to implement in booster. How often root-on-snapshot and root-on-raid are used?

from booster.

And how do people assemble ZFS filesystems at boot. Do you use zfs import or there are other ways to perform a fs assembly?

from booster.

For my daily use, on a laptop, I don't use root-on-snapshot, although not sure what you mean by that, is that when the root pool is backed up by snapshots and then these are remotely sent to another machine ?

About root-on-raid, I don't have enough HDDs to do such inside a laptop, on a server this would be a different story.

At boot, the ZFS filesystem get assembled by a systemctl unit (at least on my machine) I believe :

zfs-import-cache.service - Import ZFS pools by cache file
     Loaded: loaded (/usr/lib/systemd/system/zfs-import-cache.service; enabled; vendor preset: enabled)
     Active: active (exited) since Fri 2022-05-06 07:04:09 CEST; 23h ago
       Docs: man:zpool(8)
    Process: 2065 ExecStart=/usr/bin/zpool import -c /etc/zfs/zpool.cache -aN $ZPOOL_IMPORT_OPTS (code=exited, status=0/SUCCESS)
   Main PID: 2065 (code=exited, status=0/SUCCESS)
        CPU: 9ms

mai 06 07:04:09 495-new-phantom systemd[1]: Starting Import ZFS pools by cache file...
mai 06 07:04:09 495-new-phantom zpool[2065]: no pools available to import
mai 06 07:04:09 495-new-phantom systemd[1]: Finished Import ZFS pools by cache file.                                         

The message saying "no pools available to import" is somewhat a bug, a pool gets indeed imported on my machine.

from booster.

Also, maybe support can be added partially for ZFS filesystem, and then more complex features can be supported later, not everything must work/be supported from day one, I guess.

from booster.

I did some work on this issue and pushed it to wip branch.

Booster follows the same idea as implemented by zfs-import-cache.service - at the image generation time it copies /etc/zfs/zpool.cache into the image. And then at the boot time it runs zpool import -c /etc/zfs/zpool.cache -aN $ZPOOL_IMPORT_OPTS and expects that the root filesystem is mounted after it.

The problem is that I cannot test it. The integration tests fail due to hardcoded device paths (I think). The boot time import fails for me with Destroy and re-create the pool from a backup source. cachefile import failed, retrying

Is there a way to use disk UUID for the pool creation? Or maybe there is a way to modify the cache file and replace /dev/loop with /dev/sda path?

Anyway if there are people who are interested with debugging/testing please pull wip compile and then add the following booster.yaml config options:

enable_zfs: true
zfs_import_params: # add any params you need to mount the root partition

from booster.

Thanks, @anatol, for your work on this feature, I might give it a try but at the moment, I can't really test it for now, here is a pointer on how to use disk UUID/PARTUUIDs for pools at creation : https://wiki.archlinux.org/title/ZFS#Identify_disks
See also the next section about GPT labels, it might be relevant for what you're asking

It is also mentioned :

Warning
If you create zpools using device names (e.g. /dev/sda,/dev/sdb,...) ZFS might not be able to detect zpools intermittently on boot.

from booster.

• @techhazard who was interested in dracut+clevis+zfs support. Booster plans to add zfs and later look at clevis+zfs encryption integration.

from booster.

The experimental zfs support has been merged to master branch.

To enable the zfs support first modify /etc/booster.yaml and add enable_zfs: true, then regenerate the image and then add boot parameter zfs=$pool/$dataset that specifies your root dataset.

Please try it. Please report if you have any issues.

from booster.

Hi, I'm specifying zfs=zroot/ROOT/default per the Arch ZFS installation guide, and while it works for mkinitcpio, it doesn't seem booster like this syntax. When I boot I only get a "Press Enter to Reboot" message.

I see that the current parser uses a Split("/") which breaks the string above into 3 parts (the last one being ignored). Could this syntax be supported?

from booster.

Thank you for the feedback @ishitatsuyuki

I see that the current parser uses a Split("/") which breaks the string above into 3 parts (the last one being ignored). Could this syntax be supported?

In fact, only the first part (zpool in our case) is used to import a specific pool by its name. Later zroot/ROOT/default is used as a dataset name. Is it what you expect here?

When I boot I only get a "Press Enter to Reboot" message.

Could you please enable debug and post any logs related to zfs?

from booster.

It says booster: signal: aborted. Before that the ZFS module appears to be loaded but there doesn't seem to be anything else that is interesting.

EDIT: typo.

from booster.

There must be some information before the crash. If you don't mind could you please share the full debug logs with me?

Also please make sure you added enable_zfs: true to your booster.yaml config.

from booster.

I don't know if there's a way to take actual log files, but here is a slo-mo video capture of boot logs (kernel parameter debug).

https://drive.google.com/file/d/1wpQn6OfM5fXlm1vTu84iRMPeL2acQAwV/view?usp=sharing

Sorry, the camera quality isn't very great, but if you download (the inline player transcodes) and play it it should be sufficient to see what the log says.

from booster.

(I also checked the generated booster image. Yes, the zfs stuff is there.)

moduledependencies:
    aesni_intel:
        - crypto_simd
        - cryptd
    btrfs:
        - xor
        - raid6_pq
        - libcrc32c
        - blake2b_generic
        - crc32c_intel
        - crc32c_generic
    crypto_simd:
        - cryptd
    dm_multipath:
        - dm_mod
    ghash_clmulni_intel:
        - cryptd
    hid_logitech_dj:
        - usbhid
    icp:
        - zcommon
        - znvpair
        - spl
    libcrc32c:
        - crc32c_intel
        - crc32c_generic
    nvme:
        - nvme_core
    tpm:
        - rng_core
    tpm_crb:
        - tpm
        - rng_core
    tpm_tis:
        - tpm_tis_core
        - tpm
        - rng_core
    tpm_tis_core:
        - tpm
        - rng_core
    vfat:
        - fat
    xhci_pci:
        - xhci_pci_renesas
    zavl:
        - spl
    zcommon:
        - znvpair
        - spl
    zfs:
        - zunicode
        - zzstd
        - zlua
        - zavl
        - icp
        - zcommon
        - znvpair
        - spl
    znvpair:
        - spl
    zzstd:
        - spl
modulesforceload:
    - zfs
builtinmodules:
    88pm860x: true
    "8250": true
    8250_base: true
    8250_lpss: true
    8250_mid: true
    8250_pci: true
    8250_pericom: true
    ac: true
    acpiphp: true
    aead: true
    aes_generic: true
    af_packet: true
    agpgart: true
    ahci: true
    akcipher: true
    asn1_decoder: true
    asymmetric_keys: true
    autofs4: true
    backlight: true
    battery: true
    bfq: true
    binfmt_elf: true
    binfmt_misc: true
    binfmt_script: true
    bitrev: true
    braille_console: true
    bsg: true
    btree: true
    button: true
    cec: true
    cfbcopyarea: true
    cfbfillrect: true
    cfbimgblt: true
    charger_manager: true
    cma_heap: true
    cn: true
    compat_binfmt_elf: true
    configfs: true
    configs: true
    cpufreq_conservative: true
    cpufreq_ondemand: true
    cpufreq_performance: true
    cpufreq_powersave: true
    cpufreq_userspace: true
    cpuid: true
    crc_ccitt: true
    crc_t10dif: true
    crc32: true
    crc64: true
    crc64_rocksoft: true
    crc64_rocksoft_generic: true
    crct10dif_common: true
    crct10dif_generic: true
    crypto: true
    crypto_acompress: true
    crypto_algapi: true
    crypto_hash: true
    crypto_null: true
    cryptomgr: true
    ctr: true
    da903x: true
    da9052_core: true
    da9052_i2c: true
    da9052_spi: true
    da9055: true
    dax: true
    dh_generic: true
    digsig: true
    dmi_sysfs: true
    drbg: true
    drm: true
    drm_kms_helper: true
    drm_mipi_dsi: true
    drm_panel_orientation_quirks: true
    drm_shmem_helper: true
    drop_monitor: true
    dw_dmac_core: true
    dw_dmac_pci: true
    ecc: true
    ecdsa_generic: true
    edac_core: true
    efi_pstore: true
    efivarfs: true
    ehci_hcd: true
    ehci_pci: true
    evdev: true
    exportfs: true
    extcon_core: true
    ezx_pcap: true
    fan: true
    fb: true
    fb_sys_fops: true
    fbdev: true
    firmware_class: true
    font: true
    freq_table: true
    geniv: true
    glob: true
    hed: true
    hid: true
    hid_generic: true
    hmac: true
    hsu_dma: true
    hwmon: true
    hwspinlock_core: true
    i2c_algo_bit: true
    i2c_ccgx_ucsi: true
    i2c_core: true
    i2c_designware_core: true
    i2c_designware_pci: true
    i2c_designware_platform: true
    icc_core: true
    input_core: true
    input_leds: true
    intel_pmc_core: true
    intel_pmc_core_pltdrv: true
    intel_pstate: true
    intel_soc_pmic: true
    ioasid: true
    iommu_v2: true
    iosf_mbi: true
    iova: true
    ipv6: true
    jitterentropy_rng: true
    kdf_sp800108: true
    kpp: true
    kyber_iosched: true
    led_class: true
    libaes: true
    libahci: true
    libata: true
    libblake2s: true
    libblake2s_x86_64: true
    libnvdimm: true
    libsha256: true
    linear_ranges: true
    lp8788: true
    lz4: true
    lz4_compress: true
    lz4_decompress: true
    lzo_compress: true
    lzo_decompress: true
    mctp: true
    md5: true
    mfd_core: true
    mpi: true
    mq_deadline: true
    msr: true
    mt6323_poweroff: true
    n_null: true
    nfs_ssc: true
    nls_ascii: true
    nls_base: true
    nls_cp437: true
    nvmem_core: true
    nvram: true
    ohci_hcd: true
    ohci_pci: true
    oid_registry: true
    packing: true
    palmas: true
    pci_meson: true
    pci_stub: true
    pinctrl_alderlake: true
    pinctrl_broxton: true
    pinctrl_cannonlake: true
    pinctrl_cedarfork: true
    pinctrl_cherryview: true
    pinctrl_denverton: true
    pinctrl_elkhartlake: true
    pinctrl_emmitsburg: true
    pinctrl_geminilake: true
    pinctrl_icelake: true
    pinctrl_intel: true
    pinctrl_jasperlake: true
    pinctrl_lakefield: true
    pinctrl_lewisburg: true
    pinctrl_lynxpoint: true
    pinctrl_sunrisepoint: true
    pinctrl_tigerlake: true
    pkcs7_message: true
    pldmfw: true
    power_supply: true
    powercap_sys: true
    pps_core: true
    pretimeout_panic: true
    processor: true
    pstore: true
    ptp: true
    public_key: true
    pvcalls_back: true
    rational: true
    rc_core: true
    regmap_i2c: true
    regmap_mmio: true
    regmap_spi: true
    remoteproc: true
    restart_poweroff: true
    rng: true
    rsa_generic: true
    rtc_cmos: true
    sch_fq_codel: true
    scsi_common: true
    scsi_mod: true
    sd_mod: true
    serdev: true
    serial_core: true
    serial_mctrl_gpio: true
    sha1_generic: true
    sha256_generic: true
    sha512_generic: true
    shpchp: true
    simpledrm: true
    skcipher: true
    syscopyarea: true
    sysfillrect: true
    sysimgblt: true
    system_heap: true
    t10_pi: true
    tcp_cubic: true
    thermal: true
    tps6586x: true
    tps65086_restart: true
    twl4030_audio: true
    twl6040: true
    ucs2_string: true
    udmabuf: true
    uhci_hcd: true
    unicode: true
    unix: true
    usb_common: true
    usbcore: true
    usbserial: true
    utf8data: true
    vesafb: true
    vgacon: true
    virt_dma: true
    virtio: true
    virtio_ring: true
    vmgenid: true
    watch_queue: true
    watchdog: true
    wwan: true
    x509_key_parser: true
    xenbus: true
    xenbus_probe_frontend: true
    xfrm_algo: true
    xfrm_user: true
    xhci_hcd: true
    xxhash: true
    xz_dec: true
    z3fold: true
    zbud: true
    zlib_deflate: true
    zlib_inflate: true
    zpool: true
    zsmalloc: true
    zstd: true
    zstd_compress: true
    zstd_decompress: true
    zswap: true
kernel: 5.18.14-arch1-1
enablezfs: true

from booster.

Thank you. You did not enable booster debug though. Please add booster.log=debug,console to the boot parameters - it will print a lot more information from booster that hopefully give a hint of what is going on here.

from booster.

Sorry; missed that option somehow,

The debug log flooded console with udev message that are not so useful; and there was still no clue about where the abort occurred.

The info log level gives a more readable output:

One thing to note is that I'm running ZFS on a partition as opposed to the entire disk, could that maybe matter?

from booster.

OK, I put the busybox in and I can now debug from a shell. It was zpool import that is tripping an assert and failing. I'll let you know when I figure out anything.

from booster.

Could it be the device path perhaps? My zpool.cache refers to /dev/disk/by-id/nvme-eui.e8238fa6bf530001001b448b457d94f2-part2 but as far as I tried this symlink doesn't appear to be generated by booster (only parsed when passed to /boot).

from booster.

It was zpool import that is tripping an assert and failing.

Interesting, if zpool command is failing I would expect its stderr printed at the console. The booster code does unwrapExitError(err) that extracts the stderr and later it should be printed to console.

Could it be the device path perhaps? My zpool.cache refers to /dev/disk/by-id/nvme-eui.e8238fa6bf530001001b448b457d94f2-part2 but as far as I tried this symlink doesn't appear to be generated by booster (only parsed when passed to /boot).

Indeed it is a culprit. The by-id and by-path support are really limited at this point. What do you have when you print ls /dev/disk/by-id/?

from booster.

On a functional boot, the by-id is:

nvme-eui.e8238fa6bf530001001b444a4408ab4b        nvme-WD_BLACK_SN770_1TB_22073Z802747
nvme-eui.e8238fa6bf530001001b444a4408ab4b-part1  nvme-WD_BLACK_SN770_1TB_22073Z802747-part1
nvme-eui.e8238fa6bf530001001b444a4408ab4b-part2  nvme-WD_BLACK_SN770_1TB_22073Z802747-part2
nvme-eui.e8238fa6bf530001001b444a4408ab4b-part3  nvme-WDC_WDS100T2B0C-00PXH0_20070X461413
nvme-eui.e8238fa6bf530001001b444a4408ab4b-part4  nvme-WDC_WDS100T2B0C-00PXH0_20070X461413-part1
nvme-eui.e8238fa6bf530001001b444a4408ab4b-part5  nvme-WDC_WDS100T2B0C-00PXH0_20070X461413-part2
nvme-eui.e8238fa6bf530001001b444a4408ab4b-part6  nvme-WDC_WDS100T2B0C-00PXH0_20070X461413-part3
nvme-eui.e8238fa6bf530001001b444a4408ab4b-part7  nvme-WDC_WDS100T2B0C-00PXH0_20070X461413-part4
nvme-eui.e8238fa6bf530001001b448b457d94f2        nvme-WDC_WDS100T2B0C-00PXH0_20070X461413-part5
nvme-eui.e8238fa6bf530001001b448b457d94f2-part1  nvme-WDC_WDS100T2B0C-00PXH0_20070X461413-part6
nvme-eui.e8238fa6bf530001001b448b457d94f2-part2  nvme-WDC_WDS100T2B0C-00PXH0_20070X461413-part7

Meanwhile on booster busybox IIRC is:

nvme-eui.e8238fa6bf530001001b444a4408ab4b
nvme-WD_BLACK_SN770_1TB_22073Z802747
nvme-WDC_WDS100T2B0C-00PXH0_20070X461413
nvme-eui.e8238fa6bf530001001b448b457d94f2

from booster.

@ishitatsuyuki I pushed a possible fix to the wip branch. Could you please try it?

from booster.

It works! Thanks very much for your help and the prompt fix.

from booster.

@ishitatsuyuki I just pushed the fix to master branch. Could you please try it and let me know if everything works fine for you?

from booster.

Sure, just updated and it still works.

from booster.

@ishitatsuyuki thank you for confirming it!

from booster.