I can't really get kexec to work on Raspberry Pi 4, even with upstream kernel. This article suggests that the ARM Trusted Framework might assist without requiring the user to limit their cpu for their initial boot.

https://www.thegoodpenguin.co.uk/blog/booting-linux-from-linux-with-kexec/

I don't know if this is true, though.

Again, leaving this as a note to self, I'll try to send this as documentation or something:

1. Use dd to copy tow-boot on the front of an SD card.
2. Use parted then print to get a prompt to expand GPT to the full disk
3. Then you can use gdisk to add a new "linux filesystem" partition behind the tow-boot partition
4. Format that ext4 partition

Then install your favorite way.

It'd be nice to have specific commands to execute to do step 2, but I couldn't figure it out quickly.

An installation on some installation locations on some platforms, e.g. SPI on RK3399, eMMC on Pinebook Pro*, can be hard to recover from.

One method that could be used to make the installation/upgrade more robust is to:

• Break valid boot signature by nulling it on the target storage media
• Install most of the IBF; probably all of it but the signature
• Finish by writing the signature

This would mean that an incomplete install, e.g. stopped before the end, would still "brick" the computer, but at the same time, the computer would be easier to recover from. That is, an alternative boot source could be used, like an SD card.

* eMMC on Pinebook Pro is about as inconvenient as SPI Flash to recover from when an IBF install has gone wrong. Both requires disassembling the Pinebook Pro. The main difference is that for SPI Flash you may have to disassemble a bit further.

It's unclear to me how U-Boot syncs, and when.

Syncing dts with upstream automatically in some form would be helpful since the built-in device tree is used in some boot schemes (e.g. UEFI) when no other dtb is provided.

Automatically syncing should be done in a manner that would likely be accepted by upstream.

I'm not talking about a task that runs automatically here, but about a script a maintainer would run for each kernel bump.

I guess the main issue is that automatically importing those changes in theory could lead to breakage in U-Boot. But really, it would mean there's an issue with correctness somewhere, I would assume. Any breakage would need to be fixed anyway. And it means the built-in DT is probably wrong according to the mainline kernel's opinion on DTs.

Though it should be relatively straightforward since the porting guide is "ensure you understand how mainline U-Boot gets built for your platform" and "then add it to the builds".

The former being a bit out of scope, but we can provide pointers.

The latter depending on the specific board and SoC family.

Idea from a discussion on the Pinebook channel.

[@samueldr] the generic distro boot scheme from U-Boot would benefit greatly from a couple echo telling the user what it is actually using

Trying boot source: mmc0
 looking for ESP... (not found)
 looking for legacy bootable partition... (found)
  looking for boot.scr (not found)
  looking for extlinux/extlinux.conf (found)
Attempting boot from extlinux/extlinux.conf  from partition mmc0 partition 2

Or anything with at least this level of detail

Something to try to upstream back to U-Boot.

Though I guess it would be nice if we could somewhat distinguish the "errors" from those messages... Not sure this is possible given the line-oriented nature of the output.

This appears to be somewhat of a u-boot problem, and not specifically tow-boot, but I figure an opinionated configuration might provision for this sort of thing:

I had Tow-Boot installed to SPI, and I was poking around the console, and I tried running saveenv, which proceeded to save environment settings to SPI, overwriting part of the loader. Which part? I don't know. It didn't say where in SPI it was saving, but I presume it was an important region because now the power LED is red upon power on and nothing else happens.

Maybe saveenv can write to a specific offset past the bootloader? I'm not sure what the best course of action is here. All I know is that I rendered my Pinebook Pro unbootable through a command with a relatively innocuous-sounding name. (Yes, I should have looked up what this did first…)

This would serve as a way to update the firmware on, let's say, an eMMC disk built into a device.

It would work basically like the SPI installer, except that it would

• Check firmware does not collide with unexpected partitions (in case it grew too big)
• dd into the board-specific shared offset

The goal is that it should be able to install Tow-Boot on a board where it wasn't installed previously. While it won't synthesize the partitions around a "legacy" installed U-Boot, it will at least give the end-user the benefits of using Tow-Boot.

Additionally, the installer should allow "completely erasing" the disk. This would be the main installation method for eMMC. The use case would be as follow:

• Fresh device with eMMC and no SPI
• Use the shared storage installer from an SD card
• Use "erase, format, partition and install Tow-Boot"

This would "just" write disk-image.img on the eMMC.

I have just flashed the '002' pre-release to a Pinebook Pro SPI flash. Happy user. Thanks!
Booting works. Except for the boot menu option "Boot to SD" does not actually boot from an SD card.

NOTE I can put the same SD card in a USB card-reader connected to USB, and "Boot to USB" will work correctly. (I have previously used the same SD card to install an OS, so it should be working. I originally flashed Tow-Boot to SPI flash from the same slot.)

The main place where it will be observable is scary is in the SPI installer images.

The cause is known: it's https://github.com/Tow-Boot/Tow-Boot/blob/30d52a6747f389043853cb59cdae2f5ed01f8af8/support/builders/tow-boot/patches/0001-HACK-video-sync-dirty.patch.

The board is most likely fine, wait long enough, SPI erase and flash can be a slow operation at times. Waiting up to 5 minutes should be sufficient. Though those operations shouldn't last 5 minutes.

Forcing a sync on scroll may help with this issue.

It might be helpful for people with existing installs that wants to install to another storage. E.g. a Pinebook Pro user that wants to remove the eMMC U-Boot from their distro from the firmware boot flow, so that SD card boot is usable.

SPI

Erasing the first 0x2000 bytes was verified to be enough on Allwinner A64, RK3399 and S805X (and assumed to be fine for other SoCs in their respective families).

eMMC / SD

RK3399

Replace /dev/mmcblkX by the block storage you want to neuter the install on. Note that running on any other storage can break the data it contains. On RK3399 platforms, generally (please verify on your device), the eMMC is at /dev/mmcblk2.

# dd if=/dev/zero of=/dev/mmcblkX bs=512 seek=64 count=16

This differs from U-Boot in some ways.

• Serial baud rate is always 115200
• Environment storage (read platform-specific notes)
• Program location offset in SPI (read platform-specific notes)
• Boot order is always static, does not differ depending on the firmware boot source (see sunxi patch)

Hey, there,
I am currently trying to boot from my usb thumbdrive and sdcard - And it just won't work; It always sais "Unrecognised filesystem type"
I have already tried either both inbuilt usb ports and my sdcard slot on the Pinebook Pro.
I have flashed clonezilla onto both devices via Etcher.
Trying yet another different device, this is what I got this time:

** Unrecognised filesystem type **
Scanning disk [email protected]
Scanning disk usb_mass_storage.lun0
Found 7 disks
** Unable to read file ubootefi.var **
Failed to load EFI variables
BootOrder not defined
EFI boot manager: Cannot load any image
USB Boot failed.
Press any key to continue...

After another usb rescan, I got this while trying to boot my usb device:

Device 0: Foo
... is now current device
libfdt fdt_check_header(): FDT_ERR_BADMAGIC
BootOrder not defined
EFI boot manager: Cannot load any image
USB Boot failed.
Press any key to continue...

While creating this issue, Tow-Boot just decided to freeze up - seems to happen, when it's not touched in a while (5-10min).

Little sidenote: Please set up github actions - At the moment I can't compile the package myself.

I hope this info is useful!
Thanks!

Some new updates from the EBBR repo:

• https://github.com/ARM-software/ebbr/wiki/U-Boot-Configuration-Guide
• https://github.com/ARM-software/ebbr/wiki/EBBR-Testing-Guide

The capsule updates will need more reading from me to better really internalize how it ends up being used. But that's, I guess, the main way we'll support upgrades.

It uses U-Boot's DFU tooling

• https://github.com/u-boot/u-boot/blob/master/doc/usage/dfu.rst

I noticed that it seems like you copied the ARM TF into this repo. As I was typing up #71, I was going to point out that automating nixpkgs updates means that I automate rpi4 firmware flowing into here eventually (which also notably wouldn't be covered by #71).

Presenting the question, does it make sense to copy the rpi firmware into the overlay (or otherwise package it in this repo) to control the version being deployed?

(If not, does it make sense to remove ARM TF or upstream whatever you did to it back to nixpkgs?)

For example, we write the config in this repo, but are dependent on the firmware from nixpkgs. Granted, I think this config format is basically stable, but just something for consideration.

Right now the built artifacts are built locally and uploaded manually to the GitHub release.

Big yikes!

Explore avenues for build artifacts.

Right now the baud rate is as provided by upstream projects.

For most boards it's 115200, except for Rockchip where it would be 1500000. About an order of magnitude more.

The most common default baud rate would be used for all boards.

Benefits

• Common instructions for all boards
• Assuming 115200, the most common speed out there
• Some distros have their getty default to 115200

Drawbacks

• Does not match defaults from other projects
• Does not match vendor defaults (meh!)

Does Tow-Boot support BOOT_LOADER_SPECIFICATION (originated from freedesktop)

This provides an already common standard out there, which should be sufficient enough even for embedded stuff.

This locked post serves as a wiki-like issue for maintainers to post links to in-progress work at upstream.

Patches to upstream next

• https://github.com/Tow-Boot/Tow-Boot/blob/38a0e77a2b79b3e95f3270f384a8913dd70248f8/support/u-boot/2022.07/patches/0001-smbios-enclosure-support.patch

Patches tracked for upstream

Patches accepted an merged upstream

• https://patchwork.ozlabs.org/project/uboot/patch/[email protected]/
• https://patchwork.ozlabs.org/project/uboot/patch/[email protected]/
• https://patchwork.ozlabs.org/project/uboot/patch/[email protected]/
• pause patch
  • https://patchwork.ozlabs.org/project/uboot/patch/[email protected]/
  • https://patchwork.ozlabs.org/project/uboot/patch/[email protected]/
  • https://patchwork.ozlabs.org/project/uboot/patch/[email protected]/

Hi, right now we're just getting started, and trying to bite a huge chunk all at once. Please politely pester us to get started sending changes back!

Tested on Pinebook Pro.

Research needs to be done to see where the issue lies, with U-Boot or with Linux.

For the time being, boot with any other method supported by the generic distro boot scheme.

Workaround

Add efifb=off to the kernel command-line.

It works since any fb module will get short-circuited when parsing its cmdline options:

• https://github.com/torvalds/linux/blob/df8c66c4cfb91f2372d138b9b714f6df6f506966/drivers/video/fbdev/core/fb_cmdline.c#L59-L60
• https://github.com/torvalds/linux/blob/df8c66c4cfb91f2372d138b9b714f6df6f506966/drivers/video/fbdev/efifb.c#L353-L354

Another workaround has been suggested, adding fbcon=map:1 to the command-line.

Upstream fix

Writeup by the patch author:

• https://blog.dowhile0.org/2021/07/21/a-framebuffer-hidden-in-plain-sight/

Accepted:

• torvalds/linux@3c1ed51

https://github.com/crust-firmware/crust/

For supported platforms only, obviously.

I wrote the image spi.installer.img to my SD card, shut down my PBP, disconnect the eMMC chip (as it takes precedence to boot), and boot from my SD card. And when I reached the boot menu, there is no option for flash firmware to SPI flash. Or is there any way to flash the firmware from the console?

Just opening this for a place to track, share notes, etc.

Hi.

Maybe this is because my SD came pre-formatted, but after I dd the rpi4 shared.disk-image.img to my SD Card... I figured I'd fire up fdisk to add a traditional /boot and /root for a shared media install.

However, fdisk gave me this warning, and I'm not actually sure what I want to do here:

/home/cole/downloads/raspberryPi-aarch64-2021.04-002> sudo dd if=shared.disk-image.img of=/dev/sdb bs=4M; sudo sync; sync
8+1 records in
8+1 records out
34620416 bytes (35 MB, 33 MiB) copied, 0.018304 s, 1.9 GB/s

/home/cole/downloads/raspberryPi-aarch64-2021.04-002> sudo fdisk /dev/sdb

Welcome to fdisk (util-linux 2.36.2).
Changes will remain in memory only, until you decide to write them.
Be careful before using the write command.

The backup GPT table is not on the end of the device. This problem will be corrected by write.
A hybrid GPT was detected. You have to sync the hybrid MBR manually (expert command 'M').

Any tips? Should I ignore it? Should I have "wiped" the sd card more thoroughly?

Right now using the wrong installer disk image on a board continues with installation.

E.g. it is possible to use the spi-installer.img from libreComputer-amlS805xAc on a pine64-pinebookPro. This particular instance shouldn't cause much harm, but it might be possible to have compatible-enough boards, where it puts the user in a hard situation.

The solution is to, as early as possible, check the target board in the installation script, and bail when it differs from the expectation.

Fedora fails to boot, I tried with both Workstation 34 and Rawhide, one on my eMMC and one on my microSD. (I forget which one is which).

Two screenshots:

It works (to an extent) when using Fedora's own uboot, package spec/source here: https://src.fedoraproject.org/fork/rdossant/rpms/uboot-tools/blob/rawhide/f/uboot-tools.spec

I know UEFI booting is incomplete, so I figure there's something either upstream or a patch in Fedora's repo that makes this work, but unfortunately I don't know enough about uboot to understand what's going on in either your nix build system or how Fedora's rpm spec works.

(Unrelated note, my own matrix server needs to be burned with fire and abandoned due to lack of maintenance; when I have a matrix home I'll be joining your matrix group).

Hi. It's not 100% exact, but it's close. It's almost always that first boot from cold power works, but on a reboot, u-boot fails to detect and properly scan my SSD.

However, I also have figured out how to utilize the early watchdog in latest rpi4 firmware, so I have a very reliable workaround. (If u-boot starts to tftp, it will fail to get anything responding to the WD and will be rebooted, the next boot always works.)

I do realize this is likely an upstream u-boot issue, but again, I don't really have any interaction with that community, so I'm posting here instead.

Currently the Tow-Boot environment customization is done through a gnarly patch that does environment init through C defines in a header. Just like done in U-Boot.

The main issue is we may need customization depending on factors. E.g. look at setup_leds.

The "correct" way to handle this would be to provide patches to add a "setup_leds" feature to U-Boot Kconfig and all that jazz. That is a bit cumbersome, and anyway it seems U-Boot themselves want to do away with the C mess for environment.

As a way to circumvent the issue, we can do better. Let's provide a patch that adds a shim environment header file, and build our custom environment entirely through Nix expressions. In turn, this should give us better semantics to provide better environment. The shim would be the current 0001-Tow-Boot-Provide-opinionated-boot-flow.patch, but without the environment defined. The shim is needed because it needs to be hooked into include/config_distro_bootcmd.h.

This is more important with the upcoming features I've been working on, where device-specific handling of boot steps is required. (These upcoming features are the Phone UX.)

The issue had been reported on the Matrix room, but never filed here.

Accidentally, or mindfully, touching the touchpad will break keyboard input on the Pinebook Pro.

The workaround is to not touch the touchpad.

TODO

• Verify if a keyboard+mouse exhibits the same issue on another board (could not reproduce)
• Verify with mouse on the same USB device (e.g. test with my "air mouse" remote thingy) (could not reproduce)
• Verify touchpad on Pinebook A64 (could not reproduce)

Here:

So I guess at CONFIG_AUTOBOOT_PROMPT it should eat all input?

Or since it also is a problem when canceling late, maybe we need a way to loop without canceling on ^C?

For now, prefer [Escape] for canceling boot I guess.

I updated my raspberrypi-eeprom package that I use with the nixpkgs that I build my system with, and updated the system.

But for the rpi to update its own eeprom "in-place" (without a separate SD card to host a recovery.bin), there's a bit more to it:

sudo mount /dev/{your_tow_boot_fw_part} /boot/firmware
sudo env FIRMWARE_RELEASE_STATUS=stable BOOTFS=/boot/firmware rpi-eeprom-update -a
sudo reboot

Also, it seems rpi-eeprom-update wants to use the "default" release which lags behind "stable" by quite a bit. Don't ask me.

Ahead of my nixpkgs changes merging, I thought I'd try my hand at a script to compare against a "LKG" (last known good) of the platform/board binary.

Some quick thoughts/questions:

1. Should there be a slight refactor so that something exposes a combination of { image = ...; tow-boot-binary = ....; } so that it's easy to walk to output boards and have the corresponding platform binary inside that we want to test?
2. Alternatively, it seems like some of the boards are basically just wraps around the builder, indicating maybe that we don't want to have a binary for each board necessarily? (This even applies to rpi4 where I have two board outputs, one with and one without ATF, where there's only a single TB binary to test).

Just stashing some thoughts, curious what you think or have a preference or if I'm overthinking it.

https://github.com/pengutronix/genimage

I'm trying to install NixOS on my PBP and I've built both Tow-Boot and your NixOS sdimage (the new feature branch). I can't seem to boot the spi-installer.img though; after writing it to the sd card the PBP never boots from it. The sdimage produced by your overlay works fine otoh, so I'm guessing it's an issue with the spi-installer.img. Since my PBP boots currently, can I just write the SPI image directly on the PBP instead of booting from the image?

I've recently installed Tow-Boot, as it has been my only solution to get working graphics on my already installed openSUSE. I installed oS via serial console, to the NVMe, LUKS encrypted,

With Tow-Boot I can type my LUKS password and log into my desktop, all seems well. Except I have noticed that there are no modules loaded for sound or battery (cw2015_battery), so I tried to load another dtb from upstream. I don't know how to do it from Tow-Boot, I always get the following:
libfdt fdt_check_header(): FDT_ERR_BADMAGIC

So I load it from GRUB with the following:
devicetree /boot/rk3399-pinebook-pro.dtb

Then the output is different, for instance I now see messages related to the cw2015_battery, so the new dtb seems to be loaded fine. However I have no graphical output again. So far I've tried efifb=off, blacklist=efifb, forcing video parameters similar to how other distros do, but I can't get any video output.

I guess I have one question, as the ARM land is quite new territory for me. Where can I get the dtb that Tow-Boot loads by default? If I could compare them, it would certainly provide some interesting info.

Sorry if this isn't related to Tow-Boot, the boot process is almost black magic to me. I'd be more than happy to provide any additional logs/output. Thanks.

This is our #1 issue.

This, or a future evolution of mainline U-Boot straight from them, should be the main, if not only, suggestion about how to get started booting something on an ARM SBC.

Tow-Boot is non-partisan

While this is built using Nix, this is not a NixOS project.

The goal will always be to support any standards-compliant operating systems.

We will always accept contributions from other distribution users and maintainers, and other operating systems. Don't hesitate to contribute installation notes for your operating system of choice!

This project uses Nix to coordinate builds.

It's great and all, but contributors may not want, or may be unable to install Nix.

Provide a pre-configured Docker-based environment to transparently use Nix to build this.

https://www.raspberrypi.org/documentation/computers/config_txt.html#upstream_kernel

Not sure if this would help Tow-Boot support both mainline + non-mainline users by having the DTBs for both in places?

But then I'm not sure how Tow-Boot's DTBs would stay in sync with the kernel?

Another angle of the usual weird problem with the RPi.

Right now things work for me, but I think it's a bit of luck. Also, I notice that when I boot the mainline kernel, I get slightly different device names and consoles depending on which bootloader is used. I think this is because extlinux is loading mainline kernel and re-loading the mainline DTB as the device tree. On the other hand, since NixOS's grub generator currently omits DTB lines, grub loads mainline with the firmware-mangled, vendor-sourced, foundation-kernel-derived DTBs.

It almost feels like the right thing is to have multiple variants... but I'm not sure.

Selfishly, I'd like to replace the vendor DTBs with (any recent version) of mainlines, but I don't know if one direction of version skew is more forgiving than another.

The latter (platform firmware) is what's being used more often than not in the EBBR documentation. Let's help this wording take its righteous place.

Note that dropping "boot" from here will probably help reducing confusion about bootloader vs. platform firmware!

(Side-note for me: fix this also at NixOS on ARM on the NixOS wiki...)

Allwinner

[smaeul] samueldr: allwinner does support boot partitions https://u-boot.readthedocs.io/en/latest/board/allwinner/sunxi.html#installing-on-emmc-on-board-flash-memory

[smaeul] according to https://patchwork.ozlabs.org/project/uboot/cover/[email protected]/ it is before the user data partition

• https://u-boot.readthedocs.io/en/latest/board/allwinner/sunxi.html#installing-on-emmc-on-board-flash-memory
• https://patchwork.ozlabs.org/project/uboot/cover/[email protected]/
• https://linux-sunxi.org/Bootable_eMMC

With this, supporting dedicated storage for Allwinner boards (range to be confiremd, but H3, and A64 at the very least) should be possible through the eMMC boot partitions.

Writing and managing should be pretty much equivalent to SPI installation.

Note that for some boards, eMMC is not fixed storage. When a board supports SPI, we shouldn't build the eMMC boot partition support in official builds, it provides no real benefit, especially when boards often treat eMMC as non-fixed storage.

• Proof of concept (DONE)

Amlogic

Implementation details from Radxa, unclear which families will actively support the feature.

• https://wiki.radxa.com/Zero/install/Boot_Troubleshooting

• Proof of concept (DONE)

Rockchip

Until proven otherwise, it is assumed the Boot ROM sequence cannot.

Duh...

The ANSI console provided by Tow-Boot is limited to cp437... emojis give some harmless gobble-dee-gook.

(See photo in #45)

At least, for installing on the "internal storage" of targets such as the Pinebook Pro.

Someone that wants to toy around with firmware development, testing many U-Boot builds and derivatives will be interested in installing the firmware to an SD card (and only the firmware to that SD card), such that it allows running a fresh build more easily, and a failed build is not a bother.

Really, for "final end-user products" like the Pinebook Pro, there is no benefit to installing to the internal eMMC, comapred to installing to the SPI, as recovering from a bad install (which is unlikely either way, and as likely either way), is just about as inconvenient.

Hello. I've been using Tow-Boot "pine64-pinebookPro-2021.04-001" with some success (and some issues but likely unrelated to Tow-Boot).
I'm trying to update to 002, as described in the release page it fails to recognize $board_identifier. The suggested workaround, env default board_identifier, didn't work ($board_identifier stays empty).
I had to do env set board_identifier pine64-pinebookPro to pass the check.

Then this happens:

Any idea?
I would like to avoid having to completely erase the SPI Flash as it's been annoying to force booting on the SD without Tow-Boot.

If left uninterrupted, it will boot correctly. However, if I press ctrl+C at the appropriate moment to enter the tow-boot menu, none of the boot options work. If I enter the "firmware console", none of the usual commands (such as run distro_bootcmd or run bootcmd_mmc0) work correctly.

If I run env default -a first, then run distro_bootcmdor run bootcmd_mmc0 will work, confirming that something in the environment is being set in a way that prevents it booting. When I have time, I'll print the env over serial before and after resetting to defaults, and try to narrow down the culprit.

My PBP has no eMMC or SD Card in it, it has Tow-Boot in the SPI flash, and Manjaro-ARM on an NVMe drive.

Cold booting the machine usually doesn't work, resulting in the boot menu appearing. From here, selecting reboot will normally allow the machine to successfully boot afterwards.

It's as if the NVMe drive does not have enough time to initialize the controller properly from a cold boot. Maybe an option to have a delay before attempting to boot from a device will fix this?

Tow-Boot version used: https://github.com/Tow-Boot/Tow-Boot/tree/release-2021.04-002
NVMe Drive used: KINGSTON RBUSNS8154P3256GJ1, as reported by nvme list-subsys

E.g.

If you use sgdisk, type

sgdisk -A 3:set:2 /dev/sda

which means set capability bit 2 (=the "legacy BIOS partition") of partition 3 of /dev/sda.

— https://www.lightofdawn.org/wiki/wiki.cgi/BIOSBootGPT

While this is built using Nix, this is not a NixOS project.

The goal will always be to support any standards-compliant operating systems.

We will always accept contributions from other distribution users and maintainers, and other operating systems. Don't hesitate to contribute installation notes for your operating system of choice!

After flashing Tow-Boot onto SPI I can't move around the menu with arrow keys. My best guess is either whole boot menu is frozen or keyboard just stops working. Issue happens after few seconds of accessing boot menu. I'm using latest build from releases tab