This setup has been mostly tested on Ubuntu. For other host operating systems see: Supported hosts.

Reserve 12Gb of disk and run:

You don’t need to clone recursively even though we have .git submodules: download-dependencies fetches just the submodules that you need for this build to save time.

If something goes wrong, see: Common build issues and use our issue tracker: https://github.com/************/linux-kernel-module-cheat/issues

The initial build will take a while (30 minutes to 2 hours) to clone and build, see Benchmark builds for more details.

If you don’t want to wait, you could also try the following faster but much more limited methods:

• Prebuilt setup

• Host kernel module setup

but you will soon find that they are simply not enough if you anywhere near serious about systems programming.

After ./run, QEMU opens up and you can start playing with the kernel modules inside the simulated system:

This should print to the screen:

which are printk messages from init and cleanup methods of those modules.

Sources:

• kernel_modules/hello.c

• kernel_modules/hello2.c

Quit QEMU with:

See also: Quit QEMU from text mode.

All available modules can be found in the kernel_modules directory.

It is super easy to build for different CPU architectures, just use the --arch option:

To avoid typing --arch aarch64 many times, you can set the default arch as explained at: Default command line arguments

I now urge you to read the following sections which contain widely applicable information:

• Run command after boot

• Clean the build

• Build the documentation

• Linux kernel

  • printk

  • Kernel command line parameters

Once you use GDB step debug and tmux, your terminal will look a bit like this:

Besides a seamless initial build, this project also aims to make it effortless to modify and rebuild several major components of the system, to serve as an awesome development setup.

1.1.2.1. Your first Linux kernel hack

vim submodules/linux/init/main.c

pr_info("I'VE HACKED THE LINUX KERNEL!!!");

./build-linux
./run

<6>[    0.000000] I'VE HACKED THE LINUX KERNEL!!!

./build --dry-run

1.1.2.2. Your first kernel module hack

pr_info("hello init hacked\n");

./build-modules

insmod /mnt/9p/out_rootfs_overlay/hello.ko

ls "$(./getvar out_rootfs_overlay_dir)"

./build-modules
./build-buildroot
./run --eval-after 'insmod /hello.ko'

./build --dry-run

./run --kvm

1.1.2.3. Your first QEMU hack

vim submodules/qemu/target/i386/cpu.c

.model_id = "QEMU Virtual CPU version " QEMU_HW_VERSION,

.model_id = "QEMU Virtual CPU version HACKED " QEMU_HW_VERSION,

./build-qemu
./run --eval-after 'grep "model name" /proc/cpuinfo'

1.1.2.4. Your first glibc hack

./build-buildroot -- glibc-reconfigure

diff --git a/libio/ioputs.c b/libio/ioputs.c
index 706b20b492..23185948f3 100644
--- a/libio/ioputs.c
+++ b/libio/ioputs.c
@@ -38,8 +38,9 @@ _IO_puts (const char *str)
   if ((_IO_vtable_offset (_IO_stdout) != 0
        || _IO_fwide (_IO_stdout, -1) == -1)
       && _IO_sputn (_IO_stdout, str, len) == len
+      && _IO_sputn (_IO_stdout, " hacked", 7) == 7
       && _IO_putc_unlocked ('\n', _IO_stdout) != EOF)
-    result = MIN (INT_MAX, len + 1);
+    result = MIN (INT_MAX, len + 1 + 7);

   _IO_release_lock (_IO_stdout);
   return result;

./run --eval-after '/hello.out'

hello hacked

./run --userland hello

1.1.2.5. Your first Binutils hack

./build-userland

binutils_hack.c:8: Error: no such instruction: `myinc %rax'

vim submodules/binutils-gdb/opcodes/i386-tbl.h

diff --git a/opcodes/i386-tbl.h b/opcodes/i386-tbl.h
index af583ce578..3cc341f303 100644
--- a/opcodes/i386-tbl.h
+++ b/opcodes/i386-tbl.h
@@ -1502,6 +1502,19 @@ const insn_template i386_optab[] =
     { { { 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
 	  0, 0, 1, 1, 1, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 1, 1, 1, 0,
 	  1, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0 } } } },
+  { "myinc", 1, 0xfe, 0x0, 1,
+    { { 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
+        0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
+        0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
+        0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
+        0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0 } },
+    { 0, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
+      0, 1, 0, 1, 0, 0, 0, 0, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0,
+      0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
+      0, 0, 0, 0, 0, 0 },
+    { { { 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
+	  0, 0, 1, 1, 1, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 1, 1, 1, 0,
+	  1, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0 } } } },
   { "sub", 2, 0x28, None, 1,
     { { 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
         0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,

./build-buildroot -- host-binutils-rebuild
./build-userland --static
./run --static --userland arch/x86_64/binutils_hack

This is our reference setup, and the best supported one, use it unless you have good reason not to.

It was historically the first one we did, and all sections have been tested with this setup unless explicitly noted.

Read the following sections for further introductory material:

• Introduction to QEMU

• Introduction to Buildroot

This setup is like the QEMU Buildroot setup, but it uses gem5 instead of QEMU as a system simulator.

QEMU tries to run as fast as possible and give correct results at the end, but it does not tell us how many CPU cycles it takes to do something, just the number of instructions it ran. This kind of simulation is known as functional simulation.

The number of instructions executed is a very poor estimator of performance because in modern computers, a lot of time is spent waiting for memory requests rather than the instructions themselves.

gem5 on the other hand, can simulate the system in more detail than QEMU, including:

• simplified CPU pipeline

• caches

• DRAM timing

and can therefore be used to estimate system performance, see: gem5 run benchmark for an example.

The downside of gem5 much slower than QEMU because of the greater simulation detail.

See gem5 vs QEMU for a more thorough comparison.

For the most part, if you just add the --emulator gem5 option or *-gem5 suffix to all commands and everything should magically work.

If you haven’t built Buildroot yet for QEMU Buildroot setup, you can build from the beginning with:

If you have already built previously, don’t be afraid: gem5 and QEMU use almost the same root filesystem and kernel, so ./build will be fast.

Remember that the gem5 boot is considerably slower than QEMU since the simulation is more detailed.

To get a terminal, either open a new shell and run:

You can quit the shell without killing gem5 by typing tilde followed by a period:

If you are inside tmux, which I highly recommend, you can both run gem5 stdout and open the guest terminal on a split window with:

See also: tmux gem5.

At the end of boot, it might not be very clear that you have the shell since some printk messages may appear in front of the prompt like this:

but if you look closely, the PS1 prompt marker # is there already, just hit enter and a clear prompt line will appear.

If you forgot to open the shell and gem5 exit, you can inspect the terminal output post-mortem at:

More gem5 information is present at: gem5

Good next steps are:

• gem5 run benchmark

• m5out directory

• m5ops

sudo apt-get install docker
./run-docker create && \
./run-docker sh -- ./build --download-dependencies qemu-buildroot
./run-docker sh

./run

./run-docker sh

./run-docker sh -- ./run-gdb start_kernel

./run --graphic --vnc

sudo apt-get install vinagre
./vnc

This setup uses prebuilt binaries that we upload to GitHub from time to time.

We don’t currently provide a full prebuilt because it would be too big to host freely, notably because of the cross toolchain.

Our prebuilts currently include:

• QEMU Buildroot setup binaries

  • Linux kernel

  • root filesystem

• Baremetal setup binaries for QEMU

For more details, see our our release procedure.

Advantage of this setup: saves time and disk space on the initial install, which is expensive in largely due to building the toolchain.

The limitations are severe however:

• can’t GDB step debug the kernel, since the source and cross toolchain with GDB are not available. Buildroot cannot easily use a host toolchain: Buildroot use prebuilt host toolchain.

  Maybe we could work around this by just downloading the kernel source somehow, and using a host prebuilt GDB, but we felt that it would be too messy and unreliable.

• you won’t get the latest version of this repository. Our Travis attempt to automate builds failed, and storing a release for every commit would likely make GitHub mad at us anyways.

• gem5 is not currently supported. The major blocking point is how to avoid distributing the kernel images twice: once for gem5 which uses vmlinux, and once for QEMU which uses arch/* images, see also: vmlinux vs bzImage vs zImage vs Image.

This setup might be good enough for those developing simulators, as that requires less image modification. But once again, if you are serious about this, why not just let your computer build the full featured setup while you take a coffee or a nap? :-)

Checkout to the latest tag and use the Ubuntu packaged QEMU to boot Linux:

Or to run a baremetal example instead:

You have to checkout to the latest tag to ensure that the scripts match the release format: https://stackoverflow.com/questions/1404796/how-to-get-the-latest-tag-name-in-current-branch-in-git

Be saner and use our custom built QEMU instead:

This also allows you to modify QEMU if you’re into that sort of thing.

To build the kernel modules as in Your first kernel module hack do:

TODO: for now the only way to test those modules out without building Buildroot is with 9p, since we currently rely on Buildroot to manipulate the root filesystem.

Command explanation:

• modules_prepare does the minimal build procedure required on the kernel for us to be able to compile the kernel modules, and is way faster than doing a full kernel build. A full kernel build would also work however.

• --gcc-which host selects your host Ubuntu packaged GCC, since you don’t have the Buildroot toolchain

• --no-modules-install is required otherwise the make modules_install target we run by default fails, since the kernel wasn’t built

To modify the Linux kernel, build and use it as usual:

./build-modules --gcc-which host --host

./build-modules --gcc-which host --host -- hello hello2

./build-modules \
  --gcc-which host \
  --host \
  -- \
  kernel_modules/hello.c \
  kernel_modules/hello2.c \
;

cd "$(./getvar kernel_modules_source_dir)"
mv broken.c broken.c~

cd "$(./getvar kernel_modules_build_host_subdir)"
sudo insmod hello.ko

# Our module is there.
sudo lsmod | grep hello

# Last message should be: hello init
dmesg -T

sudo rmmod hello

# Last message should be: hello exit
dmesg -T

# Not present anymore
sudo lsmod | grep hello

Minimal host build system example:

This setup does not use the Linux kernel nor Buildroot at all: it just runs your very own minimal OS.

x86_64 is not currently supported, only arm and aarch64: I had made some x86 bare metal examples at: https://github.com/************/x86-bare-metal-examples but I’m lazy to port them here now. Pull requests are welcome.

The main reason this setup is included in this project, despite the word "Linux" being on the project name, is that a lot of the emulator boilerplate can be reused for both use cases.

This setup allows you to make a tiny OS and that runs just a few instructions, use it to fully control the CPU to better understand the simulators for example, or develop your own OS if you are into that.

You can also use C and a subset of the C standard library because we enable Newlib by default. See also: https://electronics.stackexchange.com/questions/223929/c-standard-libraries-on-bare-metal/400077#400077

Our C bare-metal compiler is built with crosstool-NG. If you have already built Buildroot previously, you will end up with two GCCs installed. Unfortunately I don’t see a solution for this, since we need separate toolchains for Newlib on baremetal and glibc on Linux: https://stackoverflow.com/questions/38956680/difference-between-arm-none-eabi-and-arm-linux-gnueabi/38989869#38989869

Every .c file inside baremetal/ and .S file inside baremetal/arch/<arch>/ generates a separate baremetal image.

For example, to run baremetal/hello.c in QEMU do:

The terminal prints:

Now let’s run baremetal/arch/aarch64/add.S:

This time, the terminal does not print anything, which indicates success.

If you look into the source, you will see that we just have an assertion there.

You can see a sample assertion fail in baremetal/interactive/assert_fail.c:

and the terminal contains:

and the exit status of our script is 1:

To modify a baremetal program, simply edit the file, .g.

and rebuild:

./build qemu-baremetal had called build-baremetal for us previously, in addition to its requirements.

./build-baremetal uses crosstool-NG, and so it must be preceded by build-crosstool-ng, which ./build qemu-baremetal also calls.

Alternatively, for the sake of tab completion, we also accept relative paths inside baremetal/, for example the following also work:

Absolute paths however are used as is and must point to the actual executable:

To use gem5 instead of QEMU do:

and then as usual open a shell with:

Or as usual, tmux users can do both in one go with:

TODO: the carriage returns are a bit different than in QEMU, see: gem5 baremetal carriage return.

Note that ./build-baremetal requires the --emulator gem5 option, and generates separate executable images for both, as can be seen from:

This is unlike the Linux kernel that has a single image for both QEMU and gem5:

The reason for that is that on baremetal we don’t parse the device tress from memory like the Linux kernel does, which tells the kernel for example the UART address, and many other system parameters.

gem5 also supports the RealViewPBX machine, which represents an older hardware compared to the default VExpress_GEM5_V1:

This generates yet new separate images with new magic constants:

But just stick to newer and better VExpress_GEM5_V1 unless you have a good reason to use RealViewPBX.

When doing bare metal programming, it is likely that you will want to learn assembly language basics. Have a look at these tutorials for the userland part:

• https://github.com/************/x86-assembly-cheat

• https://github.com/************/arm-assembly-cheat

For more information on baremetal, see the section: Baremetal.

The following subjects are particularly important:

• Tracing

• Baremetal GDB step debug

./run --wait-gdb

./run-gdb start_kernel

./run-gdb init/main.c:1088

list
next
continue

Just don’t forget to pass --arch to ./run-gdb, e.g.:

and:

https://stackoverflow.com/questions/29151235/how-to-de-optimize-the-linux-kernel-to-and-compile-it-with-o0

O=0 is an impossible dream, O=2 being the default.

So get ready for some weird jumps, and <value optimized out> fun. Why, Linux, why.

./run

/count.sh

./run-gdb

Ctrl-C
break __x64_sys_write
continue
continue
continue

rbreak .*sys_write

tmux

./run --tmux --wait-gdb

./run --tmux --wait-gdb

./run --tmux-args start_kernel --wait-gdb

./run --wait-gdb
./run-gdb start_kernel

./run --tmux-args=--no-continue --wait-gdb

man tmux

If you are using gem5 instead of QEMU, --tmux has a different effect: it opens the gem5 terminal instead of the debugger:

If you also want to use the debugger with gem5, you will need to create new terminals as usual.

From inside tmux, you can do that with Ctrl-B C or Ctrl-B %.

To see the debugger by default instead of the terminal, run:

./run

insmod /timer.ko

./run-gdb

scanning for modules in /root/linux-kernel-module-cheat/out/kernel_modules/x86_64/kernel_modules
loading @0xffffffffc0000000: /root/linux-kernel-module-cheat/out/kernel_modules/x86_64/kernel_modules/timer.ko

break lkmc_timer_callback
continue
continue
continue

TODO on arm 51e31cdc2933a774c2a0dc62664ad8acec1d2dbe it does not always work, and lx-symbols fails with the message:

Can’t reproduce on x86_64 and aarch64 are fine.

It is kind of random: if you just insmod manually and then immediately ./run-gdb --arch arm, then it usually works.

But this fails most of the time: shell 1:

shell 2:

then hit Ctrl-C on shell 2, and voila.

Then:

says that the load address is:

so it is close to the failing 0xbf0000cc.

readelf:

does not give any interesting hits at cc, no symbol was placed that far.

TODO find a more convenient method. We have working methods, but they are not ideal.

This is not very easy, since by the time the module finishes loading, and lx-symbols can work properly, module_init has already finished running!

Possibly asked at:

• https://stackoverflow.com/questions/37059320/debug-a-kernel-module-being-loaded

• https://stackoverflow.com/questions/11888412/debug-the-init-module-call-of-a-linux-kernel-module

2.4.2.1. GDB module_init step into it

./run

./run-gdb do_one_initcall

833         ret = fn();

./run-gdb init/main.c:833

2.4.2.2. GDB module_init calculate entry address

./run --eval-after '/pr_debug.sh;insmod /fops.ko;/poweroff.out'

0xffffffffc0000000 .text

./run-toolchain readelf -- \
  -s "$(./getvar kernel_modules_build_subdir)/fops.ko" | \
  grep myinit

    30: 0000000000000240    43 FUNC    LOCAL  DEFAULT    2 myinit

0xffffffffc0000000 + 0x240 = 0xffffffffc0000240

./run --eval 'insmod /fops.ko;/poweroff.out' --wait-gdb

./run-gdb '*0xffffffffc0000240'

2.4.2.3. GDB module_init break at the end of sys_init_module

b sys_finit_module

n

b do_init_module

fin

2.4.2.4. GDB module_init add trap instruction

asm( " int $3");

Useless, but a good way to show how hardcore you are. Disable lx-symbols with:

From inside guest:

as mentioned at:

• https://stackoverflow.com/questions/6384605/how-to-get-address-of-a-kernel-module-loaded-using-insmod/6385818

• https://unix.stackexchange.com/questions/194405/get-base-address-and-size-of-a-loaded-kernel-module

This will give a line of form:

And then tell GDB where the module was loaded with:

Alternatively, if the module panics before you can read /proc/modules, there is a pr_debug which shows the load address:

And then search for a line of type:

Tested on 4f4749148273c282e80b58c59db1b47049e190bf + 1.

./run-gdb --no-continue

./run-gdb extract_kernel
./run-gdb main

One possibility is to run:

and then find the second address (the first one does not work, already too late maybe):

and break there:

but TODO: it does not show the source assembly under arch/arm: https://stackoverflow.com/questions/11423784/qemu-arm-linux-kernel-boot-debug-no-source-code

I also tried to hack run-gdb with:

and no I do have the symbols from arch/arm/boot/compressed/vmlinux', but the breaks still don’t work.

This is the userland debug setup most likely to work, since at init time there is only one userland executable running.

For executables from the userland directory such as userland/count.c:

• Shell 1:

  ./run --wait-gdb --kernel-cli 'init=/count.out'

• Shell 2:

  ./run-gdb-user count main

  Alternatively, we could also pass the full path to the executable:

  ./run-gdb-user "$(./getvar userland_build_dir)/sleep_forever.out" main

  Path resolution is analogous to that of ./run --baremetal.

Then, as soon as boot ends, we are left inside a debug session that looks just like what gdbserver would produce.

BusyBox custom init process:

• Shell 1:

  ./run --wait-gdb --kernel-cli 'init=/bin/ls'

• Shell 2:

  ./run-gdb-user "$(./getvar buildroot_build_build_dir)"/busybox-*/busybox ls_main

This follows BusyBox' convention of calling the main for each executable as <exec>_main since the busybox executable has many "mains".

BusyBox default init process:

• Shell 1:

  ./run --wait-gdb

• Shell 2:

  ./run-gdb-user "$(./getvar buildroot_build_build_dir)"/busybox-*/busybox init_main

init cannot be debugged with gdbserver without modifying the source, or else /sbin/init exits early with:

Non-init process:

• Shell 1:

  ./run --wait-gdb

• Shell 2:

  ./run-gdb-user myinsmod main

• Shell 1 after the boot finishes:

  /myinsmod.out /hello.ko

This is the least reliable setup as there might be other processes that use the given virtual address.

2.6.3.1. GDB step debug userland non-init without --wait-gdb

Cannot access memory at address 0x10604

./run-toolchain --arch arm readelf -- \
  -s "$(./getvar --arch arm kernel_modules_build_subdir)/fops.ko" | \
  grep main

info line main

000105fc

b *0x000105fc

>>> call printk(0, "asdf")
Could not fetch register "orig_rax"; remote failure reply 'E14'
>>> b printk
Breakpoint 2 at 0xffffffff81091bca: file kernel/printk/printk.c, line 1824.
>>> call fdget_pos(fd)
No symbol "fdget_pos" in current context.
>>> b fdget_pos
Breakpoint 3 at 0xffffffff811615e3: fdget_pos. (9 locations)
>>>

581 SYSCALL_DEFINE3(write, unsigned int, fd, const char __user *, buf,
582         size_t, count)
583 {
584     struct fd f = fdget_pos(fd);

Could not fetch register "orig_rax"; remote failure reply 'E14'

fin

./run --cpus 2 --eval-after '/sched_getaffinity.out'

sched_getaffinity = 1 1
sched_getcpu = 1
sched_getaffinity = 1 0
sched_getcpu = 0

./build-buildroot \
  --config 'BR2_PACKAGE_UTIL_LINUX=y' \
  --config 'BR2_PACKAGE_UTIL_LINUX_SCHEDUTILS=y' \
;
./run --eval-after 'taskset -c 1,1 /sched_getaffinity.out'

sched_getaffinity = 0 1
sched_getcpu = 1
sched_getaffinity = 1 0
sched_getcpu = 0

./run \
  --cpus 2 \
  --wait-gdb \
  --eval-after 'i=0; while true; do taskset -c $i,$i /sched_getaffinity.out; i=$((! $i)); done' \
;

./run-gdb-user "$(./getvar userland_build_dir)/sched_getaffinity.out" main

(gdb) info threads
  Id   Target Id         Frame
* 1    Thread 1 (CPU#0 [running]) main () at sched_getaffinity.c:30
  2    Thread 2 (CPU#1 [halted ]) native_safe_halt () at ./arch/x86/include/asm/irqflags.h:55
(gdb) c
(gdb) info threads
  Id   Target Id         Frame
  1    Thread 1 (CPU#0 [halted ]) native_safe_halt () at ./arch/x86/include/asm/irqflags.h:55
* 2    Thread 2 (CPU#1 [running]) main () at sched_getaffinity.c:30
(gdb) c

./run --cpus 2 --eval-after '/sched_getaffinity_threads.out'

./run-gdb-user "$(./getvar userland_build_dir)/sched_getaffinity_threads.out"

b main_thread_0

lx-dmesg

lx-cmdline

lx-fdtdump
pwd

lx-lsmod

Address            Module                  Size  Used by
0xffffff80006d0000 hello                  16384  0

List all processes:

Sample output:

The second and third fields are obviously PID and process name.

The first one is more interesting, and contains the address of the task_struct in memory.

This can be confirmed with:

which contains the correct PID for all threads I’ve tried:

TODO get the PC of the kthreads: https://stackoverflow.com/questions/26030910/find-program-counter-of-process-in-kernel Then we would be able to see where the threads are stopped in the code!

On ARM, I tried:

but task_pt_regs is a #define and GDB cannot see defines without -ggdb3: https://stackoverflow.com/questions/2934006/how-do-i-print-a-defined-constant-in-gdb which are apparently not set?

Bibliography:

• https://stackoverflow.com/questions/9561546/thread-aware-gdb-for-kernel

• https://wiki.linaro.org/LandingTeams/ST/GDB

• https://events.static.linuxfound.org/sites/events/files/slides/Debugging%20the%20Linux%20Kernel%20with%20GDB.pdf presentation: https://www.youtube.com/watch?v=pqn5hIrz3A8

./run --debug
./run-gdb --before '-ex "set remotetimeout 99999" -ex "set debug remote 1"' start_kernel

This error means that the GDB server, e.g. in QEMU, sent more registers than the GDB client expected.

This can happen for the following reasons:

• you set the architecture of the client wrong, often 32 vs 64 bit as mentioned at: https://stackoverflow.com/questions/4896316/gdb-remote-cross-debugging-fails-with-remote-g-packet-reply-is-too-long

• there is a bug in the GDB server and the XML description does not match the number of registers actually sent

• the GDB server does not send XML target descriptions and your GDB expects a different number of registers by default. E.g., gem5 d4b3e064adeeace3c3e7d106801f95c14637c12f does not send the XML files

The XML target description format is described a bit further at: https://stackoverflow.com/questions/46415059/how-to-observe-aarch64-system-registers-in-qemu/53043044#53043044

insmod /timer.ko
/kgdb.sh

break lkmc_timer_callback
continue
continue
continue

./run --kdb

[0]kdb> go

/count.sh &
/kgdb.sh

[0]kdb> bp __x64_sys_write
[0]kdb> go
[0]kdb> go
[0]kdb> go
[0]kdb> go

[0]kdb> help

You can also use KDB directly from the graphic window with:

This setup could be used to debug the kernel on machines without serial, such as modern desktops.

This works because --graphics adds kbd (which stands for KeyBoarD!) to kgdboc.

TODO neither arm and aarch64 are working as of 1cd1e58b023791606498ca509256cc48e95e4f5b + 1.

arm seems to place and hit the breakpoint correctly, but no matter how many go commands I do, the count.sh stdout simply does not show.

aarch64 seems to place the breakpoint correctly, but after the first go the kernel oopses with warning:

and stack trace:

My theory is that every serious ARM developer has JTAG, and no one ever tests this, and the kernel code is just broken.

/gdbserver.sh ls

./run-gdbserver "$(./getvar buildroot_build_build_dir)"/busybox-*/busybox ls_main

/gdbserver.sh /count.out

./run-gdbserver count

break sleep
continue

step

set sysroot ${common_buildroot_build_dir}/staging

This example illustrates how reading from the x86 control registers with mov crX, rax can only be done from kernel land on ring0.

From kernel land:

works and output the registers, for example:

However if we try to do it from userland:

stdout gives:

and dmesg outputs:

Sources:

• kernel_modules/ring0.c

• kernel_modules/ring0.h

• userland/ring0.c

In both cases, we attempt to run the exact same code which is shared on the ring0.h header file.

Bibliography:

• https://stackoverflow.com/questions/7415515/how-to-access-the-control-registers-cr0-cr2-cr3-from-a-program-getting-segmenta/7419306#7419306

• https://stackoverflow.com/questions/18717016/what-are-ring-0-and-ring-3-in-the-context-of-operating-systems/44483439#44483439

TODO Can you run arm executables in the aarch64 guest? https://stackoverflow.com/questions/22460589/armv8-running-legacy-32-bit-applications-on-64-bit-os/51466709#51466709

I’ve tried:

but it fails with:

./run --kernel-cli 'init=/count.sh'

./run --eval 'echo "asdf qwer";insmod /hello.ko;/poweroff.out'

./run --kernel-cli 'init=/eval_base64.sh - lkmc_eval="insmod /hello.ko;/poweroff.out"'

echo '
insmod /hello.ko
/poweroff.out
' > gitignore.sh
./run --eval "$(cat gitignore.sh)"

echo '#!/bin/sh
insmod /hello.ko
/poweroff.out
' > rootfs_overlay/gitignore.sh
chmod +x rootfs_overlay/gitignore.sh
./build-buildroot
./run --kernel-cli 'init=/gitignore.sh'

Just using BusyBox' poweroff at the end of the init does not work and the kernel panics:

because BusyBox' poweroff tries to do some fancy stuff like killing init, likely to allow userland to shutdown nicely.

But this fails when we are init itself!

poweroff works more brutally and effectively if you add -f:

but why not just use our minimal /poweroff.out and be done with it?

Source: userland/poweroff.c

This also illustrates how to shutdown the computer from C: https://stackoverflow.com/questions/28812514/how-to-shutdown-linux-using-c-or-qt-without-call-to-system

I dare you to guess what this does:

Source: userland/sleep_forever.c

This executable is a convenient simple init that does not panic and sleeps instead.

Get a reasonable answer to "how long does boot take?":

Dmesg contains a message of type:

which tells us that boot took 2.188242 seconds.

Bibliography: https://stackoverflow.com/questions/12683169/measure-time-taken-for-linux-kernel-from-bootup-to-userpace/46517014#46517014

./run --eval-after 'echo asdf;ls /proc;ls /sys;echo qwer'

./run --kernel-cli-after-dash 'lkmc_eval="insmod /hello.ko;poweroff.out;"'

cp rootfs_overlay/etc/init.d/S98 rootfs_overlay/etc/init.d/S99.gitignore
vim rootfs_overlay/etc/init.d/S99.gitignore
./build-buildroot
./run

./run --kernel-cli 'init=/init_env_poweroff.out - asdf=qwer zxcv'

args:
/init_env_poweroff.out
-
zxcv

env:
HOME=/
TERM=linux
asdf=qwer

The annoying dash - gets passed as a parameter to init, which makes it impossible to use this method for most non custom executables.

Arguments with dots that come after - are still treated specially (of the form subsystem.somevalue) and disappear, from args, e.g.:

outputs:

so see how a.b is gone.

The simple workaround is to just create a shell script that does it, e.g. as we’ve done at: rootfs_overlay/gem5_exit.sh.

Wait, where do HOME and TERM come from? (greps the kernel). Ah, OK, the kernel sets those by default: https://github.com/torvalds/linux/blob/94710cac0ef4ee177a63b5227664b38c95bbf703/init/main.c#L173

On top of the Linux kernel, the BusyBox /bin/sh shell will also define other variables.

We can explore the shenanigans that the shell adds on top of the Linux kernel with:

From there we observe that:

gives:

therefore adding SHLVL and PWD to the default kernel exported variables.

Furthermore, to increase confusion, if you list all non-exported shell variables https://askubuntu.com/questions/275965/how-to-list-all-variables-names-and-their-current-values with:

then it shows more variables, notably:

Finally, login shells will source some default files, notably:

We currently control /root/.profile at rootfs_overlay/root/.profile, and use the default BusyBox /etc/profile.

The shell knows that it is a login shell if the first character of argv[0] is -, see also: https://stackoverflow.com/questions/2050961/is-argv0-name-of-executable-an-accepted-standard-or-just-a-common-conventi/42291142#42291142

When we use just init=/bin/sh, the Linux kernel sets argv[0] to /bin/sh, which does not start with -.

However, if you use ::respawn:-/bin/sh on inttab described at TTY, BusyBox' init sets argv[0] to -, and so does getty. This can be observed with:

where $$ is the PID of the shell itself: https://stackoverflow.com/questions/21063765/get-pid-in-shell-bash

initrd /initrd.img-4.4.0-108-generic

./build-buildroot --initramfs
./build-linux --initramfs
./run --initramfs

cat "$(./getvar run_dir)/run.sh"

ls -lh "$(./getvar linux_image)"

./build-linux
./run

./build-buildroot --initramfs
./build-linux --initramfs --linux-build-id initramfs
./run --initramfs --linux-build-id

See: rootfs

src/arch/arm/linux/atag.hh

VFS: Cannot open root device "sda" or unknown-block(8,0): error -5

"$(./getvar buildroot_host_dir)"/bin/dtc -I dtb -O dts -o a.dts a.dtb
"$(./getvar buildroot_host_dir)"/bin/dtc -I dts -O dtb -o a.dtb a.dts

sudo apt-get install device-tree-compiler

/dts-v1/;

/ {
    a;
};

dtc a.dts

/dts-v1/;

/ {
    a;
};

/ {
    b;
};

/dts-v1/;

/ {
        a;
        b;
};

./build-buildroot \
  --arch aarch64 \
  --config 'BR2_PACKAGE_DTC=y' \
  --config 'BR2_PACKAGE_DTC_PROGRAMS=y' \
;

./run --arch aarch64

dtc -I fs -O dts /sys/firmware/devicetree/base

        cpus {
                #address-cells = <0x1>;
                #size-cells = <0x0>;

                cpu@0 {
                        compatible = "arm,cortex-a57";
                        device_type = "cpu";
                        reg = <0x0>;
                };
        };

./run --arch aarch64 --cpus 2

                cpu@0 {
                cpu@1 {

./run --arch aarch64 -- -machine dumpdtb=dtb.dtb

sudo apt-get install git
git clone https://github.com/************/linux-kernel-module-cheat
cd linux-kernel-module-cheat
sudo ./setup -y

sudo apt-get install gcc-aarch64-linux-gnu
./build-userland \
  --arch aarch64 \
  --gcc-which host \
  --userland-build-id host \
;
./run \
  --arch aarch64 \
  --userland-build-id host \
  --userland print_argv \
  --userland-args 'asdf "qw er"' \
;

asdf
qw er

./build --arch aarch64 user-mode-qemu
./run \
  --arch aarch64 \
  --userland print_argv \
  --userland-args 'asdf "qw er"' \
;

10.1.1.1. FATAL: kernel too old

./run --arch aarch64 --userland uname -- -r v4.17.0

10.1.1.2. stack smashing detected

./run --userland hello
./run --userland hello --qemu-which host

*** stack smashing detected ***: <unknown> terminated
qemu: uncaught target signal 6 (Aborted) - core dumped

./run --arch aarch64 --userland hello
./run --static --userland hello

qemu: uncaught target signal 4 (Illegal instruction) - core dumped

Running dynamically linked executables in QEMU requires pointing it to the root filesystem with the -L option so that it can find the dynamic linker and shared libraries.

We pass -L by default, so everything just works.

However, in case something goes wrong, you can also try statically linked executables with:

gem5 user mode currently only supports static executables: gem5 syscall emulation mode.

It’s nice when the obvious just works, right?

and on another shell:

Or alternatively, if you are using tmux, do everything in one go with:

To stop at the very first instruction of a freestanding program, just use --no-continue TODO example.

fatal: Unable to open dynamic executable's interpreter.

./build-userland \
  --arch aarch64 \
  --static \
;
./run \
  --arch aarch64 \
  --emulator gem5 \
  --userland print_argv \
  --userland-args 'asdf "qw er"' \
;

./run \
  --arch aarch64 \
  --emulator gem5 \
  --static \
  --userland print_argv \
  --userland-args 'asdf "qw er"' \
  --wait-gdb \
;
./run-gdb \
  --arch aarch64 \
  --emulator gem5 \
  --static \
  --userland print_argv \
  main \
;

As of gem5 7fa4c946386e7207ad5859e8ade0bbfc14000d91, the crappy se.py script does not forward the exit status of syscall emulation mode, you can test it with:

Source: userland/false.

Then manually run the generated gem5 CLI, and do:

and the output is always 0.

Instead, it just outputs a message to stdout just like for m5 fail:

which we parse in run and then exit with the correct result ourselves…​

Let’s see if user mode runs considerably faster than full system or not.

gem5 user mode:

gem5 full system:

QEMU user mode:

QEMU full system:

Result on P51 at bad30f513c46c1b0995d3a10c0d9bc2a33dc4fa0:

• gem5 user: 33 seconds

• gem5 full system: 51 seconds

• QEMU user: 45 seconds

• QEMU full system: 223 seconds

./build --all-archs test-user-mode
./test-user-mode --all-archs --all-emulators

./build --all-archs test-user-mode-qemu
./test-user-mode --all-archs --emulator qemu

./run --eval-after 'insmod /hello.ko'

# init_module
/myinsmod.out /hello.ko
# finit_module
/myinsmod.out /hello.ko "" 1
/myrmmod.out hello

man init_module

ls /lib/modules/<kernel_version>

ls -l /bin/lsmod

lrwxrwxrwx 1 root root 4 Jul 25 15:35 /bin/lsmod -> kmod

dpkg -l | grep -Ei

ii  kmod                                        22-1ubuntu5                                         amd64        tools for managing Linux kernel modules

Name of a predecessor set of tools.

kmod’s modprobe can also load modules under different names to avoid conflicts, e.g.:

./run --vnc

./vnc

https://superuser.com/questions/1087859/how-to-quit-the-qemu-monitor-when-not-using-a-gui

However, our QEMU setup captures Ctrl + C and other common signals and sends them to the guest, which makes it hard to quit QEMU for the first time since there is no GUI either.

The simplest way to quit QEMU, is to do:

Alternative methods include:

• quit command on the QEMU monitor

• pkill qemu

./run --graphic

./qemu-monitor info qtree

cat /dev/urandom > /dev/fb0

Scroll up in QEMU graphic mode:

but I never managed to increase that buffer:

• https://askubuntu.com/questions/709697/how-to-increase-scrollback-lines-in-ubuntu14-04-2-server-edition

• https://unix.stackexchange.com/questions/346018/how-to-increase-the-scrollback-buffer-size-for-tty

The superior alternative is to use text mode and GNU screen or tmux.

13.2.2.1. QEMU graphic mode arm terminal

./run --arch aarch64 --graphic

cat /dev/urandom > /dev/fb0

cat: write error: No space left on device

13.2.2.2. QEMU graphic mode arm terminal implementation

-device virtio-gpu-pci

CONFIG_DRM=y
CONFIG_DRM_VIRTIO_GPU=y

13.2.2.3. QEMU graphic mode arm VGA

-device VGA

# We use virtio-gpu because the legacy VGA framebuffer is
# very troublesome on aarch64, and virtio-gpu is the only
# video device that doesn't implement it.

./build-linux \
  --arch arm \
  --custom-config-file-gem5 \
  --linux-build-id gem5-v4.15 \
;
./run --arch arm --emulator gem5 --linux-build-id gem5-v4.15

vinagre localhost:5900

[    0.152755] [drm] found ARM HDLCD version r0p0
[    0.152790] hdlcd 2b000000.hdlcd: bound virt-encoder (ops 0x80935f94)
[    0.152795] [drm] Supports vblank timestamp caching Rev 2 (21.10.2013).
[    0.152799] [drm] No driver support for vblank timestamp query.
[    0.215179] Console: switching to colour frame buffer device 240x67
[    0.230389] hdlcd 2b000000.hdlcd: fb0:  frame buffer device
[    0.230509] [drm] Initialized hdlcd 1.0.0 20151021 for 2b000000.hdlcd on minor 0

system.vncserver: Listening for connections on port 5900

info: VNC client attached

./run \
  --arch arm \
  --emulator gem5 \
  --linux-build-id gem5-v4.15 \
  -- --frame-capture \
;

frames_system.vncserver/fb.<frame-index>.<timestamp>.png.gz

kmscube[706]: unhandled level 2 translation fault (11) at 0x00000000, esr 0x92000006, in libgbm.so.1.0.0[7fbf6a6000+e000]

For aarch64 we also need to configure the kernel with linux_config/display:

This is because the gem5 aarch64 defconfig does not enable HDLCD like the 32 bit one arm one for some reason.

TODO get working. There is an unmerged patchset at: https://gem5-review.googlesource.com/c/public/gem5/+/11036/1

The DP650 is a newer display hardware than HDLCD. TODO is its interface publicly documented anywhere? Since it has a gem5 model and in-tree Linux kernel support, that information cannot be secret?

The key option to enable support in Linux is DRM_MALI_DISPLAY=y which we enable at linux_config/display.

Build the kernel exactly as for Graphic mode gem5 aarch64 and then run with:

We cannot use mainline Linux because the gem5 arm Linux kernel patches are required at least to provide the CONFIG_DRM_VIRT_ENCODER option.

gem5 emulates the HDLCD ARM Holdings hardware for arm and aarch64.

The kernel uses HDLCD to implement the DRM interface, the required kernel config options are present at: linux_config/display.

TODO: minimize out the --custom-config-file. If we just remove it on arm: it does not work with a failing dmesg:

So what other options are missing from gem5_defconfig? It would be cool to minimize it out to better understand the options.

./build-buildroot --config-fragment buildroot_config/x11
./run --graphic

startx

xcalc
xeyes

./build-buildroot --config-fragment buildroot_config/x11 -- xserver_xorg-server-reconfigure

[    2.809104] WARNING: CPU: 0 PID: 51 at drivers/gpu/drm/ttm/ttm_bo_vm.c:304 ttm_bo_vm_open+0x37/0x40

TODO 9076c1d9bcc13b6efdb8ef502274f846d8d4e6a1 I’m 100% sure that it was working before, but I didn’t run it forever, and it stopped working at some point. Needs bisection, on whatever commit last touched x11 stuff.

• https://askubuntu.com/questions/730891/how-can-i-get-a-mouse-cursor-in-qemu

• https://stackoverflow.com/questions/19665412/mouse-and-keyboard-not-working-in-qemu-emulator

-show-cursor did not help, I just get to see the host cursor, but the guest cursor still does not move.

Doing:

shows that interrupts do happen when mouse and keyboard presses are done, so I expect that it is some wrong either with:

• QEMU. Same behaviour if I try the host’s QEMU 2.10.1 however.

• X11 configuration. We do have BR2_PACKAGE_XDRIVER_XF86_INPUT_MOUSE=y.

/var/log/Xorg.0.log contains the following interesting lines:

The file /dev/inputs/mice does not exist.

Note that our current link:kernel_confi_fragment sets:

for gem5, so you might want to remove those lines to debug this.

On ARM, startx hangs at a message:

and nothing shows on the screen, and:

says:

A friend told me this but I haven’t tried it yet:

• xf86-video-modesetting is likely the missing ingredient, but it does not seem possible to activate it from Buildroot currently without patching things.

• xf86-video-fbdev should work as well, but we need to make sure fbdev is enabled, and maybe add some line to the Xorg.conf

ifup -a

wget google.com
cat index.html

ifdown -a

ping google.com

PING google.com (216.58.204.46): 56 data bytes

14.3.1.1. nc host to guest

nc -l -p 45455

echo asdf | nc localhost 45455

14.3.1.2. ssh into guest

./build-buildroot --config 'BR2_PACKAGE_OPENSSH=y'

/sshd.sh

ssh root@localhost -p 45456

14.3.1.3. gem5 host to guest networking

First Enable networking.

Then in the host, start a server:

And then in the guest, find the IP we need to hit with:

which gives:

so we use in the guest:

Bibliography: https://serverfault.com/questions/769874/how-to-forward-a-port-from-guest-to-host-in-qemu-kvm/951835#951835

All of 9P and NFS (and sshfs) allow sharing directories between guest and host.

Advantages of 9P

• requires sudo on the host to mount

• we could share a guest directory to the host, but this would require running a server on the guest, which adds simulation overhead

  Furthermore, this would be inconvenient, since what we usually want to do is to share host cross built files with the guest, and to do that we would have to copy the files over after the guest starts the server.

• QEMU implements 9P natively, which makes it very stable and convenient, and must mean it is a simpler protocol than NFS as one would expect.

  This is not the case for gem5 7bfb7f3a43f382eb49853f47b140bfd6caad0fb8 unfortunately, which relies on the diod host daemon, although it is not unfeasible that future versions could implement it natively as well.

Advantages of NFS:

• way more widely used and therefore stable and available, not to mention that it also works on real hardware.

• the name does not start with a digit, which is an invalid identifier in all programming languages known to man. Who in their right mind would call a software project as such? It does not even match the natural order of Plan 9; Plan then 9: P9!

As usual, we have already set everything up for you. On host:

Guest:

Host:

The main ingredients for this are:

• 9P settings in our kernel configs

• 9p entry on our rootfs_overlay/etc/fstab

  Alternatively, you could also mount your own with:

  mkdir /mnt/my9p
  mount -t 9p -o trans=virtio,version=9p2000.L host0 /mnt/my9p

• Launch QEMU with -virtfs as in your run script

  When we tried:

  security_model=mapped

  writes from guest failed due to user mismatch problems: https://serverfault.com/questions/342801/read-write-access-for-passthrough-9p-filesystems-with-libvirt-qemu

Bibliography:

• https://superuser.com/questions/628169/how-to-share-a-directory-with-the-host-without-networking-in-qemu

• https://wiki.qemu.org/Documentation/9psetup

TODO seems possible! Lets do it:

• http://gem5.org/wiki/images/b/b8/Summit2017_wa_devlib.pdf

• http://gem5.org/WA-gem5

TODO: get working.

9P is better with emulation, but let’s just get this working for fun.

First make sure that this works: Guest to host networking.

Then, build the kernel with NFS support:

Now on host:

Now edit /etc/exports to contain:

and restart the server:

Now on guest:

TODO: failing with:

And now the /tmp directory from host is not mounted on guest!

If you don’t want to start the NFS server after the next boot automatically so save resources, do:

To modify a single option on top of our default kernel configs, do:

Kernel modules depend on certain kernel configs, and therefore in general you might have to clean and rebuild the kernel modules after changing the kernel config:

and then proceed as in Your first kernel module hack.

You might often get way without rebuilding the kernel modules however.

To use an extra kernel config fragment file on top of our defaults, do:

To use just your own exact .config instead of our defaults ones, use:

There is also a shortcut --custom-config-file to use the gem5 arm Linux kernel patches.

The following options can all be used together, sorted by decreasing config setting power precedence:

• --config

• --config-fragment

• --custom-config-file

To do a clean menu config yourself and use that for the build, do:

But remember that every new build re-configures the kernel by default, so to keep your configs you will need to use on further builds:

So what you likely want to do instead is to save that as a new defconfig and use it later as:

You can also use other config generating targets such as defconfig with the same method as shown at: Linux kernel defconfig.

Get the build config in guest:

or with our shortcut:

or to conveniently grep for a specific option case insensitively:

Source: rootfs_overlay/conf.sh.

This is enabled by:

From host:

Just for fun https://stackoverflow.com/questions/14958192/how-to-get-the-config-from-a-linux-kernel-image/14958263#14958263:

although this can be useful when someone gives you a random image.

By default, build-linux generates a .config that is a mixture of:

• a base config extracted from Buildroot’s minimal per machine .config, which has the minimal options needed to boot: About Buildroot’s kernel configs.

• small overlays put top of that

To find out which kernel configs are being used exactly, simply run:

and look for the merge_config.sh call. This script from the Linux kernel tree, as the name suggests, merges multiple configuration files into one as explained at: https://unix.stackexchange.com/questions/224887/how-to-script-make-menuconfig-to-automate-linux-kernel-build-configuration/450407#450407

For each arch, the base of our configs are named as:

e.g.: linux_config/buildroot-x86_64.

These configs are extracted directly from a Buildroot build with update-buildroot-kernel-config.

Note that Buildroot can sed override some of the configurations, e.g. it forces CONFIG_BLK_DEV_INITRD=y when BR2_TARGET_ROOTFS_CPIO is on. For this reason, those configs are not simply copy pasted from Buildroot files, but rather from a Buildroot kernel build, and then minimized with make savedefconfig: https://stackoverflow.com/questions/27899104/how-to-create-a-defconfig-file-from-a-config

On top of those, we add the following by default:

• linux_config/min: see: Linux kernel min config

• linux_config/default: other optional configs that we enable by default because they increase visibility, or expose some cool feature, and don’t significantly increase build time nor add significant runtime overhead

  We have since observed that the kernel size itself is very bloated compared to defconfig: Linux kernel defconfig.

15.1.3.1. About Buildroot’s kernel configs

./build-linux \
  --linux-build-id defconfig \
  --custom-config-target defconfig \
  --config CONFIG_VIRTIO_PCI=y \
  --config CONFIG_VIRTIO_BLK=y \
;
./run --linux-build-id defconfig

du -h \
  "$(./getvar vmlinux)" \
  "$(./getvar --linux-build-id defconfig vmlinux)" \
;

360M    /path/to/linux-kernel-module-cheat/out/linux/default/x86_64/vmlinux
47M     /path/to/linux-kernel-module-cheat/out/linux/defconfig/x86_64/vmlinux

./build-linux \
  --linux-build-id defconfig \
  --custom-config-target defconfig \
;
./run --initrd --linux-build-id defconfig

./build-linux \
  --arch aarch64 \
  --linux-build-id defconfig \
  --custom-config-target defconfig \
;
./run \
  --arch aarch64 \
  --initrd \
  --linux-build-id defconfig \
  --memory 2G \
;

118M    /path/to/linux-kernel-module-cheat/out/linux/default/aarch64/vmlinux
240M    /path/to/linux-kernel-module-cheat/out/linux/defconfig/aarch64/vmlinux

./build-linux \
  --arch aarch64 \
  --config-fragment linux_config/min \
  --custom-config-file linux_config/buildroot-aarch64 \
  --linux-build-id min \
;

15.1.3.2. Notable alternate gem5 kernel configs

We try to use the latest possible kernel major release version.

In QEMU:

or in the source:

During update all you kernel modules may break since the kernel API is not stable.

They are usually trivial breaks of things moving around headers or to sub-structs.

The userland, however, should simply not break, as Linus enforces strict backwards compatibility of userland interfaces.

This backwards compatibility is just awesome, it makes getting and running the latest master painless.

This also makes this repo the perfect setup to develop the Linux kernel.

In case something breaks while updating the Linux kernel, you can try to bisect it to understand the root cause: Bisection.

The kernel is not forward compatible, however, so downgrading the Linux kernel requires downgrading the userland too to the latest Buildroot branch that supports it.

The default Linux kernel version is bumped in Buildroot with commit messages of type:

So you can try:

Those commits change BR2_LINUX_KERNEL_LATEST_VERSION in /linux/Config.in.

You should then look up if there is a branch that supports that kernel. Staying on branches is a good idea as they will get backports, in particular ones that fix the build as newer host versions come out.

Finally, after downgrading Buildroot, if something does not work, you might also have to make some changes to how this repo uses Buildroot, as the Buildroot configuration options might have changed.

We don’t expect those changes to be very difficult. A good way to approach the task is to:

• do a dry run build to get the equivalent Bash commands used:

  ./build-buildroot --dry-run

• build the Buildroot documentation for the version you are going to use, and check if all Buildroot build commands make sense there

Then, if you spot an option that is wrong, some grepping in this repo should quickly point you to the code you need to modify.

It also possible that you will need to apply some patches from newer Buildroot versions for it to build, due to incompatibilities with the host Ubuntu packages and that Buildroot version. Just read the error message, and try:

• git log master — packages/<pkg>

• Google the error message for mailing list hits

Successful port reports:

• v3.18: ************#39 (comment)

./run --kernel-cli 'foo bar'

cat /proc/cmdline

dmesg | grep "Command line"

Double quotes can be used to escape spaces as in opt="a b", but double quotes themselves cannot be escaped, e.g. opt"a\"b"

This even lead us to use base64 encoding with --eval!

There are two methods:

• __setup as in:

  __setup("console=", console_setup);

• core_param as in:

  core_param(panic, panic_timeout, int, 0644);

core_param suggests how they are different:

By default, the Linux kernel mounts the root filesystem as readonly. TODO rationale?

This cannot be observed in the default BusyBox init, because by default our rootfs_overlay/etc/inittab does:

Analogously, Ubuntu 18.04 does in its fstab something like:

which uses default mount rw flags.

We have however removed those setups init setups to keep things more minimal, and replaced them with the rw kernel boot parameter makes the root mounted as writable.

To observe the default readonly behaviour, hack the run script to remove replace init, and then run on a raw shell:

Now try to do:

which fails with:

We can also observe the read-onlyness with:

which contains:

and so it is Read Only as shown by ro.

Disable userland address space randomization. Test it out by running rand_check.out twice:

If we remove it from our run script by hacking it up, the addresses shown by rand_check.out vary across boots.

Equivalent to:

dmesg -n 1

echo 1 > /proc/sys/kernel/printk

./run --kernel-cli 'loglevel=5'

<LEVEL>[TIMESTAMP] MESSAGE

<6>[    2.979121] Freeing unused kernel memory: 2024K

enables all log levels, and is basically the same as:

except that you don’t need to know what is the maximum level.

https://stackoverflow.com/questions/28936199/why-is-pr-debug-of-the-linux-kernel-not-giving-any-output/49835405#49835405

Debug messages are not printable by default without recompiling.

But the awesome CONFIG_DYNAMIC_DEBUG=y option which we enable by default allows us to do:

and we have a shortcut at:

Source: rootfs_overlay/pr_debug.sh.

Syntax: https://www.kernel.org/doc/html/v4.11/admin-guide/dynamic-debug-howto.html

Wildcards are also accepted, e.g. enable all messages from all files:

TODO: why is this not working:

Enable messages in specific modules:

Source: kernel_modules/myprintk.c

This outputs the pr_debug message:

but TODO: it also shows debug messages even without enabling them explicitly:

and it shows as enabled:

Enable pr_debug for boot messages as well, before we can reach userland and write to /proc:

Get ready for the noisiest boot ever, I think it overflows the printk buffer and funny things happen.

15.4.2.1. pr_debug != printk(KERN_DEBUG

./run --kernel-cli 'initcall_debug logleve=8'

<7>[    1.756680] calling  clk_disable_unused+0x0/0x130 @ 1
<7>[    1.757003] initcall clk_disable_unused+0x0/0x130 returned 0 after 111 usecs

/* If you are writing a driver, please use dev_dbg instead */
#if defined(CONFIG_DYNAMIC_DEBUG)
#include <linux/dynamic_debug.h>

/* dynamic_pr_debug() uses pr_fmt() internally so we don't need it here */
#define pr_debug(fmt, ...) \
    dynamic_pr_debug(fmt, ##__VA_ARGS__)
#elif defined(DEBUG)
#define pr_debug(fmt, ...) \
    printk(KERN_DEBUG pr_fmt(fmt), ##__VA_ARGS__)
#else
#define pr_debug(fmt, ...) \
    no_printk(KERN_DEBUG pr_fmt(fmt), ##__VA_ARGS__)
#endif

The Linux kernel allows passing module parameters at insertion time through the init_module and finit_module system calls:

Outcome: the test passes:

Sources:

• kernel_modules/params.c

• rootfs_overlay/params.sh

As shown in the example, module parameters can also be read and modified at runtime from sysfs.

We can obtain the help text of the parameters with:

The output contains:

15.6.1.1. modprobe.conf

modprobe params
cat /sys/kernel/debug/lkmc_params

12 34

One module can depend on symbols of another module that are exported with EXPORT_SYMBOL:

Outcome: the test passes:

Sources:

• kernel_modules/dep.c

• kernel_modules/dep2.c

• rootfs_overlay/dep.sh

The kernel deduces dependencies based on the EXPORT_SYMBOL that each module uses.

Symbols exported by EXPORT_SYMBOL can be seen with:

sample output:

This requires CONFIG_KALLSYMS_ALL=y.

Dependency information is stored by the kernel module build system in the .ko files' MODULE_INFO, e.g.:

contains:

We can double check with:

The output contains:

Module dependencies are also stored at:

Output:

TODO: what for, and at which point point does Buildroot / BusyBox generate that file?

15.6.2.1. Kernel module dependencies with modprobe

modprobe buildroot_dep2

42

lsmod

Module                  Size  Used by    Tainted: G
buildroot_dep2         16384  0
buildroot_dep          16384  1 buildroot_dep2

modprobe -r buildroot_dep2

extra/buildroot_dep2.ko: extra/buildroot_dep.ko

Module metadata is stored on module files at compile time. Some of the fields can be retrieved through the THIS_MODULE struct module:

Dmesg output:

Source: kernel_modules/module_info.c

Some of those are also present on sysfs:

Output:

And we can also observe them with the modinfo command line utility:

sample output:

Module information is stored in a special .modinfo section of the ELF file:

contains:

and:

gives:

I think a dedicated section is used to allow the Linux kernel and command line tools to easily parse that information from the ELF file as we’ve done with readelf.

Bibliography:

• https://stackoverflow.com/questions/19467150/significance-of-this-module-in-linux-driver/49812248#49812248

• https://stackoverflow.com/questions/4839024/how-to-find-the-version-of-a-compiled-kernel-module/42556565#42556565

• https://unix.stackexchange.com/questions/238167/how-to-understand-the-modinfo-output

Vermagic is a magic string present in the kernel and on MODULE_INFO of kernel modules. It is used to verify that the kernel module was compiled against a compatible kernel version and relevant configuration:

Possible dmesg output:

Source: kernel_modules/vermagic.c

If we artificially create a mismatch with MODULE_INFO(vermagic, the insmod fails with:

and dmesg says the expected and found vermagic found:

Source: kernel_modules/vermagic_fail.c

The kernel’s vermagic is defined based on compile time configurations at include/linux/vermagic.h:

The SMP part of the string for example is defined on the same file based on the value of CONFIG_SMP:

TODO how to get the vermagic from running kernel from userland? https://lists.kernelnewbies.org/pipermail/kernelnewbies/2012-October/006306.html

kmod modprobe has a flag to skip the vermagic check:

This option just strips modversion information from the module before loading, so it is not a kernel feature.

init_module and cleantup_module are an older alternative to the module_init and module_exit macros:

Dmesg output:

Source: kernel_modules/module_init.c

TODO why were module_init and module_exit created? https://stackoverflow.com/questions/3218320/what-is-the-difference-between-module-init-and-init-module-in-a-linux-kernel-mod

insmod /panic.ko
insmod /oops.ko

echo c > /proc/sysrq-trigger

On panic, the kernel dies, and so does our terminal.

The panic trace looks like:

Notice how our panic message hello panic is visible at:

15.7.1.1. Kernel module stack trace to source line

myinit+0x1d/0x20 [panic]

./run-gdb

info line *(myinit+0x1d)

Line 7 of "/root/linux-kernel-module-cheat/out/kernel_modules/x86_64/kernel_modules/panic.c" starts at address 0xbf00001c <myinit+28> and ends at 0xbf00002c <myexit>.

./run-toolchain gdb -- \
  -batch \
  -ex 'info line *(myinit+0x1d)' \
  "$(./getvar kernel_modules_build_subdir)/panic.ko" \
;

15.7.1.2. BUG_ON

15.7.1.3. Exit emulator on panic

Kernel panic in simulated kernel

AttributeError: Class LinuxX86System has no parameter panic_on_panic

panic: i8042 "System reset" command not implemented.

        kernelPanicEvent = addKernelFuncEventOrPanic<Linux::KernelPanicEvent>(
            "panic", "Kernel panic in simulated kernel", dmesg_output);

15.7.1.4. Reboot on panic

echo 1 > /proc/sys/kernel/panic

15.7.1.5. Panic trace show addresses instead of symbols

./run --eval-after 'insmod /dump_stack.ko' --wait-gdb --tmux-args dump_stack

static void printk_stack_address(unsigned long address, int reliable,
                 char *log_lvl)
{
    touch_nmi_watchdog();
    printk("%s %s%pB\n", log_lvl, reliable ? "" : "? ", (void *)address);
}

If KALLSYMS are disabled then the symbol address is printed instead.

On oops, the shell still lives after.

However we:

• leave the normal control flow, and oops after never gets printed: an interrupt is serviced

• cannot rmmod oops afterwards

It is possible to make oops lead to panics always with:

An oops stack trace looks like:

To find the line that oopsed, look at the RIP register:

and then on GDB:

run

which gives us the correct line:

This-did not work on arm due to GDB step debug kernel module insmodded by init on ARM so we need to either:

• GDB module_init

• Kernel module stack trace to source line post-mortem method

The dump_stack function produces a stack trace much like panic and oops, but causes no problems and we return to the normal control flow, and can cleanly remove the module afterwards:

Source: kernel_modules/dump_stack.c

The WARN_ON macro basically just calls dump_stack.

One extra side effect is that we can make it also panic with:

Source: kernel_modules/warn_on.c

Can also be activated with the panic_on_warn boot parameter.

Debugfs is the simplest pseudo filesystem to play around with:

Outcome: the test passes:

Sources:

• kernel_modules/debugfs.c

• rootfs_overlay/debugfs.sh

Debugfs is made specifically to help test kernel stuff. Just mount, set File operations, and we are done.

For this reason, it is the filesystem that we use whenever possible in our tests.

debugfs.sh explicitly mounts a debugfs at a custom location, but the most common mount point is /sys/kernel/debug.

This mount not done automatically by the kernel however: we, like most distros, do it from userland with our fstab.

Debugfs support requires the kernel to be compiled with CONFIG_DEBUG_FS=y.

Only the more basic file operations can be implemented in debugfs, e.g. mmap never gets called:

• https://patchwork.kernel.org/patch/9252557/

• https://github.com/torvalds/linux/blob/v4.9/fs/debugfs/file.c#L212

Bibliography: https://github.com/chadversary/debugfs-tutorial

Procfs is just another fops entry point:

Outcome: the test passes:

Procfs is a little less convenient than debugfs, but is more used in serious applications.

Procfs can run all system calls, including ones that debugfs can’t, e.g. mmap.

Sources:

• kernel_modules/procfs.c

• rootfs_overlay/procfs.sh