Maybe this is an example of seeing everything as a nail since we have a big hammer in objcopy.

I added support for stubby to use an 'allowed list' of command line arguments .
I occurs to me that that is a thing that might likely be changed (as it is, there is 'root=atomix' allowed, which is clearly not general purpose).

Should we allow putting that white list into the stubby.efi via objcopy? Then whoever is putting together a kernel/initrd can make the decision of what is acceptable. Ultimately, the list is signed by the signer, so its up to them what they want to do anyway.

I am somewhat weary of this running amuck into a general purpose "stuff configuration into stubby.efi" mechanism.

Thoughts?
Scott

Hi

Linux kernel 6.2.0 has added a new config setting EFI_HANDOVER_PROTOCOL and marks this as deprecated.

Whilst this setting defaults to Y to enable handover support I was helping someone recently using UKI booting where it turned out the distro kernel in use had this unset.

stubby appears to rely on EFI_HANDOVER for booting.

I had a look at the systemd-boot code and they appear to have added support for alternative methods: systemd/systemd#20918 and systemd/systemd#24777

I also see a PR was raised in the past to remove HANDOVER support from systemd-boot, although the PR was closed, not merged: https://github.com/systemd/systemd/pull/25126/files

It would make sense to add non-handover support in the near future before distros start disabling it in 6.2.x+ kernels.

Hi,

in rhboot/shim-review#182 (comment) it's mentioned that stubby's SBAT handling is "likely incorrect". This issue is to track either (a) verification that it is correct or (b) fixing it.

The only reference I'm aware of with respect to SBAT is https://github.com/rhboot/shim/blob/main/SBAT.md . Is there any other?

The UEFI shim SBAT changes are likely going to cause a change in the stubby-smash arguments so we lare going to have a break in how the script is run. While stubby doesn't really have an "installed base" to worry about, we should establish an approach to tagging/branching now so that people who make use of stubby can do so knowing we won't break things on them.

When stubby uses a builtin command line, the command line presented to the kernel would contain only the arguments present in the file (working as expected). However, when a stubby kernel.efi with no builtin cmdline is executed from the EFI SHELL the command line given to the kernel will have the name of the kernel efi in it.

The problem here is that the string 'kernel.efi' (or whatever the name) is then subjected to the kernel parameters requirements. The string 'kernel.efi' did not look like an allowed argument, so it would be denied.

Here is an example of the failing case (insecure boot, secureboot would fail rather than emitting 'would be rejected' message). EFI shell here comes from ovmf from Ubuntu 20.04 - 0~20191122.bd85bf54-2ubuntu3.3:

BdsDxe: failed to load Boot0001 "UEFI QEMU DVD-ROM QM00005 " from PciRoot(0x0)/Pci(...): Not Found
...
BdsDxe: starting Boot0003 "EFI Internal Shell" from Fv(...)/FvFile(7C04A583-9E
3E-4F1C-AD65-E05268D0B4D1)
UEFI Interactive Shell v2.2
EDK II
UEFI v2.70 (EDK II, 0x00010000)
Mapping table
      FS0: Alias(s):F1:;BLK1:
          PciRoot(0x0)/Pci(0x2,0x0)
     BLK0: Alias(s):
          PciRoot(0x0)/Pci(0x1F,0x2)/Sata(0x2,0xFFFF,0x0)

Press ESC in 5 seconds to skip startup.nsh or any other key to continue.
...
Press ESC in 1 seconds to skip startup.nsh or any other key to continue.
Shell> setvar SecureBoot
8BE4DF61-93CA-11D2-AA0D-00E098032B8C - SecureBoot - 0001 Bytes
00 
Shell> fs0:
FS0:\> cd fs0:\efi\boot
FS0:\efi\boot\> kernel.efi root=atomix console=ttyS0
token not allowed: kernel.efi
Custom kernel would be rejected in secure mode

shim's load-options.c parse_load_options has lots of comments on this, and how to parse loadoptions.

The table below tries to summarize the scope of the problem as I understand it, where 'kernel.efi' is the name of the stubby kernel efi.

• loader is either:
  • nvram: as loaded via boot-options (set by efibootmgr or bcfg)
  • uefi-shell: from a uefi shell either with a 'startup.nsh', another nsh script, or user typing
• 'shim' : indicates if the shim is used to load stubby (shim.efi stubby.efi console=ttyS0) or not (stubby.efi console=ttyS0)

#18 does have a fix for this issue, but it included copying code from shim's load-options.c. shim is licensed under BSD-2-Clause-Patent and stubby is LGPL-2.1-or-later (much of stubby was copied from systemd).

The way 'nvram' (boot entries written to nvram) based booting works as seen in STARTUP_NSH_NVRAM is:

1. write a 'boot.opts' file to esp
2. boot into a startup.nsh script
3. use bootcfg to add a boot entry to nvram with '-opt' boot.opts
4. move startup.nsh out of the way
5. reboot

the path works, but it is both slow (due to reboot) and somewhat convoluted.

We should be able to use virt-firmware-vars to modify the ovmf-vars.fd to have the given boot entry on first power on.

I would like to be able to place a file of cmdline options for stubby to read from instead of the baked-in values.
The other advantage here is that if stubby is booted via PXE or HTTP, then when the read of the file occurs, this would result in a fetch from the source server of the value allowing us to Network Boot a stubby kernel and pass options to it since we cannot append command line args without adding some sort of interposer (like grub, or ipxe.efi).

Currently, stubby allows either:

• builtin: adding a kernel command line via objcopy (stubby-smash -c "console=ttyS0 root=LABEL=rootdev")
• runtime: providing a kernel command line at runtime via arguments to the efi application (kernel.efi console=ttyS0 root=LABEL=rootdev).

If a the efi has a builtin command line and is provided with runtime arguments, the boot is rejected.

We would like to be able to support having both a builtin and runtime command lines.

Kernel parsing of its command line and options allowed is discussed here. Some things to note:

• The kernel parses parameters from the kernel command line up to --; if it doesn’t recognize a parameter and it doesn’t contain a ‘.’, the parameter gets passed to init: parameters with ‘=’ go into init’s environment, others are passed as command line arguments to init. Everything after -- is passed as an argument to init.
• Module parameters can specified on the kernel command line with a module name prefix (usbcore.blinkenlights=1).
• behavior of multiple values of the same type (key=value) arguments differs
  • console=ttyS0 console=tty1 : kernel output will go to both tty1 and ttyS0; /dev/console is set to the last entry (tty1)
  • acpi_osi (see doc)
  • root= (may depend on initramfs implementation)

Because of the complexity, we want to be able to tightly control the order of builtin and runtime arguments.

The following is suggested:

• if there is no builtin command line, use runtime. (current behavior)
• if builtin command line is not an empty string, does not contain a marker (STUBBY_RT_CLI1), and runtime value is not an empty string, then reject boot (current behavior)
• if builtin command line is set and contains a marker (STUBBY_RT_CLI1): replace the marker with the runtime arguments (using an empty string if none are given)
• stubby gives no special consideration to the value of builtin or runtime at this stage. For example, '--' is treated the same as 'console=X' or any other string.
• if the string STUBBY_RT is found in the builtin value, but not as part of STUBBY_RT_CLI1, then fail. This is to protect against accidental signing or potentially a different future version with different behavior.
• STUBBY_RT_CLI1 must be a complete token; if STUBBY_RT_CLI10 or STUBBY_RT_CLI1console=ttyS1 are found in the builtin cmdline , boot will be rejected.
• STUBBY_RT_CLI1 cannot occur more than once in the builtin ; if multiple instances occur boot will be rejected
• if STUBBY_RT is found in runtime arguments, boot will be rejected.

Above:

• 'allowed' indicates whether the command line will be passed to 'check_cmdline' function. If false execution will stop with error immediately.
• 'result cmdline' indicates the command line that is will be passed to validation.
• <empty> indicates empty string "" (no other value)
• <any> indicates any value including empty string.

We would like to have some way to provide automated "integration" test inside the stubby tree. I added some unit-test like functionality, but actually booting a UEFI system with a signed kernel.efi is ultimately what we need to test.

Goals

• Test boot of efi application (kernel.efi) that was built using the just-built stubby.efi and assembled via stubby-smash.2.sh.
• No need to test a shim, sign the kernel.efi with a key that is in the firmware's database.
• Verify boot of built-in command line
• Verify boot of cli provided command line for terms in the allowed_list
• Verify failure of cli provided command line for terms not in the allowed list.

Implementation suggestion

• Use github workflows and execute in an ubuntu focal environment. (see the build workflow for an example).
• install qemu-system-x86 and ovmf and use qemu-system-x86_64 to boot a system.
• get a kernel and initramfs from cirros as described in a gist I wrote recently. Its probably best to copy that content to this repo.
• Then, iterate over the following, varying kernel command lines

  • smash kernel, initramfs, cmdline into kernel.efi

  • sign kernel.efi with the snakeoil keys that are provided by ovmf package (doc

  • create a vfat/esp image with that kernel inside and a startup.nsh script.

    • Another gist contains gen-esp that allows a single command for making an esp image. It has usage like:

      $ ./tools/gen-esp create esp.img \
      kernel.efi:kernel.efi \
      startup.nsh:startup.nsh
      
      creating image 128MB in esp.img
      EFI/BOOT/STARTUP.NSH -> EFI/BOOT/STARTUP.NSH
      EFI/BOOT/KERNEL.EFI -> EFI/BOOT/KERNEL.EFI
      

  • start the VM, with the built esp.img as a disk. Log its console output. Expect it to shutdown itself or kill it and fail after ~2 minutes.

    • Note that starting with no attached network devices will probably boot faster as it wont' attempt to pxe.

  • inspect console output to determine what happened (was the boot allowed, did it boot correctly ..)

  • report PASS or FAIL

stubby's test harness uses some "firmwares" files that come from OVMF and from shim-signed packages.
These can be collected from an ubuntu system using './test/collect-firmwares'.

Using different firmware files over time adds variability to the test harness, and exposes it to bugs like in LP: #1986692.

It would be good to publish the firmware files in one way or another, so that we can use consistent versions.

I've attached

• gen-lxc-firmwares.txt - bash script to use lxc to generate the firmwares files from an ubuntu container
• focal.tar.gz - files from focal up to date as of today
• jammy.tar.gz - files from jammy up to date as of today (they suffer bug 1986692, so they're not really usable).

Some x86_64 system shipped with 32-bit EFI, and I've been trying to build a 32-bit EFI version of stubby on an x86_64 system without success. Have you attempted this, or have any guidance? I think clang can do it (with headers from gnu-efi), but I wasn't able to figure out how to build stubby with clang for x86_64 as a starting point.