This is the best setup if you are on one of the supported systems: Ubuntu 16.04 or Ubuntu 18.04.

Everything will likely also work on other Linux distros if you install the analogous required packages for your distro from configure, but this is not currently well tested. Compatibility patches are welcome. You can also try Getting started with Docker if you are on other distros.

Reserve 12Gb of disk and run:

It is also trivial to build for different supported CPU architectures.

The first configure will take a while (30 minutes to 2 hours) to clone and build, see Benchmark builds for more details.

If you don’t want to wait, you could also try the following faster but much more limited methods:

• Getting started with prebuilts

• Getting started on host

but you will soon find that they are simply not enough if you anywhere near serious about systems programming.

After QEMU opens up, you can start playing with the kernel modules:

This should print to the screen:

which are printk messages from init and cleanup methods of those modules.

Source:

• kernel_module/hello.c

• kernel_module/hello2.c

Once you use GDB step debug and tmux, your terminal will look a bit like this:

All available modules can be found in the kernel_module directory.

We will try to support the following Ubuntu versions at least:

• if the latest release is an LTS, support both the latest LTS and the previous one

• otherwise, support both latest LTS and the latest non-LTS

This is a good option if you are on a Linux host, but the native build failed due to your weird host distribution.

Before anything, you must get rid of any host build files on out/ if you have any. A simple way to do this it to:

A cleaner option is to make a separate clone of this repository just for Docker, although this will require another submodule update.

Then install Docker, e.g. on Ubuntu:

The very first time you launch Docker, create the container with:

You are now left inside a shell in the Docker guest.

From there, run the exact same commands that you would on a native install: Getting started

The host git top level directory is mounted inside the guest, which means for example that you can use your host’s GUI text editor directly on the files.

Just don’t forget that if you nuke that directory on the guest, then it gets nuked on the host as well!

Trying to run the output from Docker from host won’t however, I think the main reason is that the absolute paths inside Docker are will be different than the host ones, but even if we fix that there will likely be other problems.

TODO make files created inside Docker be owned by the current user in host instead of root: https://stackoverflow.com/questions/23544282/what-is-the-best-way-to-manage-permissions-for-docker-shared-volumes

Quit and stop the container:

Restart the container:

Open a second shell in a running container:

You will need this for example to use GDB step debug.

Start a second shell, and run a command in it at the same time:

Docker stops if and only if you quit the initial shell, you can quit this one without consequences.

If you mistakenly run ./rundocker twice, it opens two mirrored terminals. To quit one of them do https://stackoverflow.com/questions/19688314/how-do-you-attach-and-detach-from-dockers-process:

To use Graphic mode from Docker:

and then on host:

Destroy the docker container:

Since we mount the guest’s working directory on the host git top-level, you will likely not lose data from doing this, just the apt-get installs.

To get back to a host build, don’t forget to clean up out/ again:

After this, to start using Docker again will you need another:

We don’t currently provide a full prebuilt because it would be too big to host freely, notably because of the cross toolchain.

However, we do provide a prebuilt filesystem and kernel, which allows you to quickly try out running our kernel modules:

1. Download QEMU and this repo:

   sudo apt-get install qemu-system-x86
   git clone https://github.com/cirosantilli/linux-kernel-module-cheat
   cd linux-kernel-module-cheat

2. go to the latest release https://github.com/cirosantilli/linux-kernel-module-cheat/releases, download the images-*.zip file and extract it into the repository:

   unzip images-*.zip

   It is not possible to automate this step without the API, and I’m not venturing there at this time, pull requests welcome.

3. checkout to the prebuilt repo version so that the scripts and documentation will be compatible with it, and run with the -P option:

   git checkout <release-sha>
   ./run -P

Limitations of this method:

• can’t GDB step debug the kernel, since the source and cross toolchain with GDB are not available. Buildroot cannot easily use a host toolchain: Buildroot use prebuilt host toolchain.

  Maybe we could work around this by just downloading the kernel source somehow, and using a host prebuilt GDB, but we felt that it would be too messy and unreliable.

• can’t create new modules or modify the existing ones, since no cross toolchain

• can’t use things that rely on our QEMU fork, e.g. in-fork Device models or Tracing

• you won’t get the latest version of this repository. Our Travis attempt to automate builds failed, and storing a release for every commit would likely make GitHub mad at us.

• gem5 is not currently supported, although it should not be too hard to do. One annoyance is that there is no Debian package for it, so you have to compile your own, so you might as well just build the image itself.

This method runs the kernel modules directly on your host computer without a VM, and saves you the compilation time and disk usage of the virtual machine method.

It has however severe limitations, and you will soon see that the compilation time and disk usage are well worth it:

• can’t control which kernel version and build options to use. So some of the modules will likely not compile because of kernel API changes, since the Linux kernel does not have a stable kernel module API.

• bugs can easily break you system. E.g.:

  • segfaults can trivially lead to a kernel crash, and require a reboot

  • your disk could get erased. Yes, this can also happen with sudo from userland. But you should not use sudo when developing newbie programs. And for the kernel you don’t have the choice not to use sudo.

  • even more subtle system corruption such as not being able to rmmod

• can’t control which hardware is used, notably the CPU architecture

• can’t step debug it with GDB easily. The alternatives are JTAG or KGDB, but those are less reliable, and JTAG requires extra hardware.

Still interested?

If the compilation of any of the C files fails because of kernel or toolchain differences that we don’t control on the host, just rename it to remove the .c extension and try again:

Once you manage to compile, and have come to terms with the fact that this may blow up your host, try it out with:

Once you are done with this method, you must clean up the in-tree build objects before you decide to do the right thing and move on to the superior ./build Buildroot method:

otherwise they will cause problems.

Minimal host build system example:

Ctrl-A X

./run -x

./qemumonitor info qtree

cat /dev/urandom > /dev/fb0

TODO could not get it working on x86_64, only ARM.

Overview: https://stackoverflow.com/questions/50364863/how-to-get-graphical-gui-output-and-user-touch-keyboard-mouse-input-in-a-ful/50364864#50364864

More concretely:

and then on another shell:

The CONFIG_LOGO penguin only appears after several seconds, together with kernel messages of type:

The port 5900 is incremented by one if you already have something running on that port, gem5 stdout tells us the right port on stdout as:

and when we connect it shows a message:

Alternatively, you can also view the frames with --frame-capture:

This option dumps one compressed PNG whenever the screen image changes inside m5out, indexed by the cycle ID. This allows for more controlled experiments.

It is fun to see how we get one new frame whenever the white underscore cursor appears and reappears under the penguin.

TODO kmscube failed on aarch64 with:

Tested on: 38fd6153d965ba20145f53dc1bb3ba34b336bde9

For aarch64 we also need -c kernel_config_fragment/display:

This is because the gem5 aarch64 defconfig does not enable HDLCD like the 32 bit one arm one for some reason.

/modulename.sh

dmesg -n 1

echo 1 > /proc/sys/kernel/printk

./run -e 'loglevel=5'

<LEVEL>[TIMESTAMP] MESSAGE

<6>[    2.979121] Freeing unused kernel memory: 2024K

Shift-PgUp

https://stackoverflow.com/questions/28936199/why-is-pr-debug-of-the-linux-kernel-not-giving-any-output/49835405#49835405

Debug messages are not printable by default without recompiling.

But the awesome CONFIG_DYNAMIC_DEBUG=y option which we enable by default allows us to do:

and we have a shortcut at:

Source: rootfs_overlay/pr_debug.sh.

Syntax: https://www.kernel.org/doc/html/v4.11/admin-guide/dynamic-debug-howto.html

Wildcards are also accepted, e.g. enable all messages from all files:

TODO: why is this not working:

Enable messages in specific modules:

Source: kernel_module/myprintk.c

This outputs the pr_debug message:

but TODO: it also shows debug messages even without enabling them explicitly:

and it shows as enabled:

Enable pr_debug for boot messages as well, before we can reach userland and write to /proc:

Get ready for the noisiest boot ever, I think it overflows the printk buffer and funny things happen.

enables all log levels, and is basically the same as:

except that you don’t need to know what is the maximum level.

./build -k

./build -- kernel_module-reconfigure

./build -l -q -g

./build -- linux-reconfigure host-qemu-reconfigure gem5-reconfigure

./build -- <pkg>-reconfigure

For example, if you decide to Enable compiler optimizations after an initial build is finished, you must first clean the build before rebuilding:

as explained at: https://buildroot.org/downloads/manual/manual.html#rebuild-pkg

The clean is necessary because the source files didn’t change, so make would just check the timestamps and not build anything.

cp cli.example data/cli

cd buildroot
git checkout -- .
git clean -xdf .

rm -rf out/x86_64/buildroot

rm -rf out/x86_64/buildroot/build/host-qemu-custom
./build

./run -F 'date >f;poweroff'

./run -F 'cat f'

sync

./build

TODO how to make gem5 disk writes persistent?

As of cadb92f2df916dbb47f428fd1ec4932a2e1f0f48 there are some read_only entries in the config.ini under cow sections, but hacking them to true did not work:

The directory of interest is src/dev/storage.

qcow2 does not appear supported, there are not hits in the source tree, and there is a mention on Nate’s 2009 wishlist: http://gem5.org/Nate%27s_Wish_List

./run -e 'foo bar'

cat /proc/cmdline

dmesg | grep "Command line"

Double quotes can be used to escape spaces as in opt="a b", but double quotes themselves cannot be escaped, e.g. opt"a\"b"

This even lead us to use base64 encoding with -E!

There are two methods:

• __setup as in:

  __setup("console=", console_setup);

• core_param as in:

  core_param(panic, panic_timeout, int, 0644);

core_param suggests how they are different:

Disable userland address space randomization. Test it out by running rand_check.out twice:

If we remove it from our run script by hacking it up, the addresses shown by rand_check.out vary across boots.

Equivalent to:

If you are feeling fancy, you can also insert modules with:

which insmods kernel_module/hello.c.

modprobe searches for modules under:

Kernel modules built from the Linux mainline tree with CONFIG_SOME_MOD=m, are automatically available with modprobe, e.g.:

If you are feeling raw, you can insert and remove modules with our own minimal module inserter and remover!

which teaches you how it is done from C code.

Source:

• kernel_module/user/myinsmod.c

• kernel_module/user/myrmmod.c

The Linux kernel offers two system calls for module insertion:

• init_module

• finit_module

and:

documents that:

The finit_module() system call is like init_module(), but reads the module to be loaded from the file descriptor fd. It is useful when the authenticity of a kernel module can be determined from its location in the filesystem; in cases where that is possible, the overhead of using cryptographically signed modules to determine the authenticity of a module can be avoided. The param_values argument is as for init_module().

finit is newer and was added only in v3.8. More rationale: https://lwn.net/Articles/519010/

Bibliography: https://stackoverflow.com/questions/5947286/how-to-load-linux-kernel-modules-from-c-code

./run

./run -n 1

./run -aA -g -n 0 &>/dev/null &
./run -aA -g -n 1 &>/dev/null &

less out/aarch64/gem5/default/0/m5out
less out/aarch64/gem5/default/1/m5out

less out/aarch64/gem5/default/0/termout.txt
less out/aarch64/gem5/default/1/termout.txt

./run -aA -g -n some-experiment.1

less out/aarch64/gem5/default/some-experiment.1/m5out

./run -aA -g -n 1

./run -n 1
./rungdb -n 1

./run -d

./rungdb start_kernel

./rungdb init/main.c:1088

l
n
c

https://stackoverflow.com/questions/29151235/how-to-de-optimize-the-linux-kernel-to-and-compile-it-with-o0

O=0 is an impossible dream, O=2 being the default.

So get ready for some weird jumps, and <value optimized out> fun. Why, Linux, why.

./run

/count.sh

./rungdb

Ctrl-C
break __x64_sys_write
continue
continue
continue

tmux

./run -du

./run -du

./run -du -U start_kernel

man tmux

If you are using gem5 instead of QEMU, -u has a different effect: it opens the gem5 terminal instead of the debugger:

If you also want to use the debugger with gem5, you will need to create new terminals as usual.

From inside tmux, you can do that with Ctrl-B C or Ctrl-B %.

To see the debugger by default instead of the terminal, run:

./run

insmod /timer.ko

./rungdb

scanning for modules in /linux-kernel-module-cheat//out/x86_64/buildroot/build/linux-custom
loading @0xffffffffc0000000: ../kernel_module-1.0//timer.ko

b lkmc_timer_callback
c
c
c

TODO on arm 51e31cdc2933a774c2a0dc62664ad8acec1d2dbe it does not always work, and lx-symbols fails with the message:

Can’t reproduce on x86_64 and aarch64 are fine.

It is kind of random: if you just insmod manually and then immediately ./rungdb -a arm, then it usually works.

But this fails most of the time: shell 1:

shell 2:

then hit Ctrl-C on shell 2, and voila.

Then:

says that the load address is:

so it is close to the failing 0xbf0000cc.

readelf:

does not give any interesting hits at cc, no symbol was placed that far.

TODO find a more convenient method. We have working methods, but they are not ideal.

This is not very easy, since by the time the module finishes loading, and lx-symbols can work properly, module_init has already finished running!

Possibly asked at:

• https://stackoverflow.com/questions/37059320/debug-a-kernel-module-being-loaded

• https://stackoverflow.com/questions/11888412/debug-the-init-module-call-of-a-linux-kernel-module?rq=1

Useless, but a good way to show how hardcore you are. Disable lx-symbols with:

From inside guest:

as mentioned at:

• https://stackoverflow.com/questions/6384605/how-to-get-address-of-a-kernel-module-loaded-using-insmod/6385818

• https://unix.stackexchange.com/questions/194405/get-base-address-and-size-of-a-loaded-kernel-module

This will give a line of form:

And then tell GDB where the module was loaded with:

Alternatively, if the module panics before you can read /proc/modules, there is a pr_debug which shows the load address:

And then search for a line of type:

./rungdb extract_kernel
./rungdb main

One possibility is to run:

and then find the second address (the first one does not work, already too late maybe):

and break there:

but TODO: it does not show the source assembly under arch/arm: https://stackoverflow.com/questions/11423784/qemu-arm-linux-kernel-boot-debug-no-source-code

I also tried to hack rungdb with:

and no I do have the symbols from arch/arm/boot/compressed/vmlinux', but the breaks still don’t work.

• Shell 1:

  ./run -d -e 'init=/sleep_forever.out'

• Shell 2:

  ./rungdb-user kernel_module-1.0/user/sleep_forever.out main

BusyBox custom init process:

• Shell 1:

  ./run -d -e 'init=/bin/ls'

• Shell 2:

  ./rungdb-user busybox-1.26.2/busybox ls_main

This follows BusyBox' convention of calling the main for each executable as <exec>_main since the busybox executable has many "mains".

BusyBox default init process:

• Shell 1:

  ./run -d

• Shell 2:

  ./rungdb-user busybox-1.26.2/busybox init_main

This cannot be debugged in another way without modifying the source, or /sbin/init exits early with:

Non-init process:

• Shell 1:

  ./run -d

• Shell 2:

  ./rungdb-user kernel_module-1.0/user/myinsmod.out main

• Shell 1 after the boot finishes:

  /myinsmod.out /hello.ko

This is the least reliable setup as there might be other processes that use the given virtual address.

>>> call printk(0, "asdf")
Could not fetch register "orig_rax"; remote failure reply 'E14'
>>> b printk
Breakpoint 2 at 0xffffffff81091bca: file kernel/printk/printk.c, line 1824.
>>> call fdget_pos(fd)
No symbol "fdget_pos" in current context.
>>> b fdget_pos
Breakpoint 3 at 0xffffffff811615e3: fdget_pos. (9 locations)
>>>

581 SYSCALL_DEFINE3(write, unsigned int, fd, const char __user *, buf,
582         size_t, count)
583 {
584     struct fd f = fdget_pos(fd);

Could not fetch register "orig_rax"; remote failure reply 'E14'

fin

./run -c2 -F '/sched_getaffinity.out'

sched_getaffinity = 1 1
sched_getcpu = 1
sched_getaffinity = 1 0
sched_getcpu = 0

taskset -c 1,1 /sched_getaffinity.out

sched_getaffinity = 0 1
sched_getcpu = 1
sched_getaffinity = 1 0
sched_getcpu = 0

./run -c2 -d -F 'i=0; while true; do taskset -c $i,$i /sched_getaffinity.out; i=$((! $i)); done'

./rungdb-user kernel_module-1.0/user/sched_getaffinity.out main

(gdb) info threads
  Id   Target Id         Frame
* 1    Thread 1 (CPU#0 [running]) main () at sched_getaffinity.c:30
  2    Thread 2 (CPU#1 [halted ]) native_safe_halt () at ./arch/x86/include/asm/irqflags.h:55
(gdb) c
(gdb) info threads
  Id   Target Id         Frame
  1    Thread 1 (CPU#0 [halted ]) native_safe_halt () at ./arch/x86/include/asm/irqflags.h:55
* 2    Thread 2 (CPU#1 [running]) main () at sched_getaffinity.c:30
(gdb) c

./run -c2 -F '/sched_getaffinity_threads.out'

./rungdb-user kernel_module-1.0/user/sched_getaffinity_threads.out

b main_thread_0

lx-dmesg

lx-cmdline

lx-fdtdump
pwd

lx-lsmod

Address            Module                  Size  Used by
0xffffff80006d0000 hello                  16384  0

List all processes:

Sample output:

The second and third fields are obviously PID and process name.

The first one is more interesting, and contains the address of the task_struct in memory.

This can be confirmed with:

which contains the correct PID for all threads I’ve tried:

TODO get the PC of the kthreads: https://stackoverflow.com/questions/26030910/find-program-counter-of-process-in-kernel Then we would be able to see where the threads are stopped in the code!

On ARM, I tried:

but task_pt_regs is a #define and GDB cannot see defines without -ggdb3: https://stackoverflow.com/questions/2934006/how-do-i-print-a-defined-constant-in-gdb which are apparently not set?

Bibliography:

• https://stackoverflow.com/questions/9561546/thread-aware-gdb-for-kernel

• https://wiki.linaro.org/LandingTeams/ST/GDB

• https://events.static.linuxfound.org/sites/events/files/slides/Debugging%20the%20Linux%20Kernel%20with%20GDB.pdf presentation: https://www.youtube.com/watch?v=pqn5hIrz3A8

Entering kdb (current=0xf8ce07d3, pid 1) due to Keyboard Entry
kdb>

Reading symbols from vmlinux...done.
Remote debugging using localhost:1234
Ignoring packet error, continuing...
warning: unrecognized item "timeout" in "qSupported" response
Ignoring packet error, continuing...
Remote replied unexpectedly to 'vMustReplyEmpty': timeout

/kgdb-mod.sh

lx-symbols ../kernel_module-1.0/
b fop_write
c
c
c

-append kgdboc=kbd

[0]kdb> go

/kgdb.sh

[0]kdb> help
[0]kdb> bp __x64_sys_write
[0]kdb> go

./rungdbserver -a arm kernel_module-1.0/user/myinsmod.out

/gdbserver.sh ls

./rungdbserver busybox-1.26.2/busybox

b open
c

s

set sysroot ${buildroot_out_dir}/staging

This example illustrates how reading from the x86 control registers with mov crX, rax can only be done from kernel land on ring0.

From kernel land:

works and output the registers, for example:

However if we try to do it from userland:

stdout gives:

and dmesg outputs:

Sources:

• kernel_module/ring0.c

• kernel_module/ring0.h

• kernel_module/user/ring0.c

In both cases, we attempt to run the exact same code which is shared on the ring0.h header file.

Bibliography:

• https://stackoverflow.com/questions/7415515/how-to-access-the-control-registers-cr0-cr2-cr3-from-a-program-getting-segmenta/7419306#7419306

• https://stackoverflow.com/questions/18717016/what-are-ring-0-and-ring-3-in-the-context-of-operating-systems/44483439#44483439

TODO Can you run arm executables in the aarch64 guest? https://stackoverflow.com/questions/22460589/armv8-running-legacy-32-bit-applications-on-64-bit-os/51466709#51466709

I’ve tried:

but it fails with:

Haven’t tried it, doubt it will work out of the box! :-)

Maybe: https://stackoverflow.com/questions/47857023/booting-a-graphical-mips-qemu-machine

Haven’t tried.

./run -E 'echo "asdf qwer";insmod /hello.ko;/poweroff.out'

./run -e 'init=/eval.sh - lkmc_eval="insmod /hello.ko;/poweroff.out"'

echo '
insmod /hello.ko
/poweroff.out
' > gitignore.sh
./run -E "$(cat gitignore.sh)"

echo '#!/bin/sh
insmod /hello.ko
/poweroff.out
' > rootfs_overlay/gitignore.sh
chmod +x rootfs_overlay/gitignore.sh
./build
./run -e 'init=/gitignore.sh'

Just using BusyBox' poweroff at the end of the init does not work and the kernel panics:

because BusyBox' poweroff tries to do some fancy stuff like killing init, likely to allow userland to shutdown nicely.

But this fails when we are init itself!

poweroff works more brutally and effectively if you add -f:

but why not just use our minimal /poweroff.out and be done with it?

Source: kernel_module/user/poweroff.c

This also illustrates how to shutdown the computer from C: https://stackoverflow.com/questions/28812514/how-to-shutdown-linux-using-c-or-qt-without-call-to-system

I dare you to guess what this does:

Source: kernel_module/user/sleep_forever.c

This executable is a convenient simple init that does not panic and sleeps instead.

Get a reasonable answer to "how long does boot take?":

Dmesg contains a message of type:

which tells us that boot took 2.188242 seconds.

Bibliography: https://stackoverflow.com/questions/12683169/measure-time-taken-for-linux-kernel-from-bootup-to-userpace/46517014#46517014

./run -F 'echo asdf;ls /proc;ls /sys;echo qwer'

./run -f 'lkmc_eval="insmod /hello.ko;poweroff.out;"'

cp rootfs_overlay/etc/init.d/S98 rootfs_overlay/etc/init.d/S99.gitignore
vim rootfs_overlay/etc/init.d/S99.gitignore
./build
./run

./run -e 'init=/init_env_poweroff.sh - asdf=qwer zxcv'

./run -e 'init=/init_env_poweroff.sh - /poweroff.out'

/sbin/ifup -a

/sbin/ifdown -a

wget google.com

ping google.com

watch -n 1 grep i8042 /proc/interrupts

[    27.549] (II) LoadModule: "mouse"
[    27.549] (II) Loading /usr/lib/xorg/modules/input/mouse_drv.so
[    27.590] (EE) <default pointer>: Cannot find which device to use.
[    27.590] (EE) <default pointer>: cannot open input device
[    27.590] (EE) PreInit returned 2 for "<default pointer>"
[    27.590] (II) UnloadModule: "mouse"

# CONFIG_INPUT_MOUSE is not set
# CONFIG_INPUT_MOUSEDEV_PSAUX is not set

vgaarb: this pci device is not a vga device

grep EE /var/log/Xorg.0.log

(EE) Failed to load module "modesetting" (module does not exist, 0)

initrd /initrd.img-4.4.0-108-generic

./build -I -l && ./run -I

./build -I && ./run -I

ls -lh out/x86_64/buildroot/images/bzImage

By default, we use a .config that is a mixture of:

• Buildroot’s minimal per machine .config, which has the minimal options needed to boot

• our kernel configs which enables options we want to play with

Use just your own exact .config instead:

Beware that Buildroot can sed override some of the configurations we make no matter what, e.g. it forces CONFIG_BLK_DEV_INITRD=y when BR2_TARGET_ROOTFS_CPIO is on, so you might want to double check as explained at Find the kernel config. TODO check if there is a way to prevent that patching and maybe patch Buildroot for it, it is too fuzzy. People should be able to just build with whatever .config they want.

Modify a single option:

Use an extra kernel config fragment file:

-K, -c, -C can all be used at the same time. Options passed via -C take precedence over -c, which takes precedence over -K.

Ge the build config in guest:

or with our shortcut:

or to conveniently grep for a specific option case insensitively:

Source: rootfs_overlay/conf.sh.

This is enabled by:

From host:

Just for fun https://stackoverflow.com/a/14958263/895245:

although this can be useful when someone gives you a random image.

We have managed to come up with minimalistic kernel configs that work for both QEMU and gem5 (oh, the hours of bisection).

Our configs are all based on Buildroot’s configs, which were designed for QEMU, and then on top of those we also add:

• kernel_config_fragment/min: minimal tweaks required to boot gem5 or for using our slightly different QEMU command line options than Buildroot

• kernel_config_fragment/default: optional configs that we add by default to our kernel build because they increase visibility, and don’t significantly increase build time nor add significant runtime overhead

Changes to those files automatically trigger kernel reconfigures even without using the linux-reconfigure target, since timestamps are used to decide if changes happened or not.

Having the same config working for both QEMU and gem5 means that you can deal with functional matters in QEMU, which runs much faster, and switch to gem5 only for performance issues.

To see Buildroot’s base configs, have a look at buildroot/configs/qemu_x86_64_defconfig, which our ./build script uses.

That file contains BR2_LINUX_KERNEL_CUSTOM_CONFIG_FILE="board/qemu/x86_64/linux-4.11.config", which points to the base config file used.

arm, on the other hand, uses buildroot/configs/qemu_arm_vexpress_defconfig, which contains BR2_LINUX_KERNEL_DEFCONFIG="vexpress", and therefore just does a make vexpress_defconfig.

Other configs which we had previously tested at 4e0d9af81fcce2ce4e777cb82a1990d7c2ca7c1e are:

• Jason’s magic x86_64 config: http://web.archive.org/web/20171229121642/http://www.lowepower.com/jason/files/config which is referenced at: http://web.archive.org/web/20171229121525/http://www.lowepower.com/jason/setting-up-gem5-full-system.html. QEMU boots with that by removing # CONFIG_VIRTIO_PCI is not set

• arm and aarch64 configs present in the official ARM gem5 Linux kernel fork: https://gem5.googlesource.com/arm/linux, e.g. for arm v4.9: https://gem5.googlesource.com/arm/linux/+/917e007a4150d26a0aa95e4f5353ba72753669c7/arch/arm/configs/gem5_defconfig. The patches there are just simple optimizations and instrumentation, but they are not needed to boot.

On one hand, we would like to have our configs as a single git file tracked on this repo, to be able to easily refer people ot them. However, that would lose use the ability to:

• reuse Buildroot’s configs

• split our configs into min and default

We try to use the latest possible kernel major release version.

In QEMU:

or in the source:

During update all you kernel modules may break since the kernel API is not stable.

They are usually trivial breaks of things moving around headers or to sub-structs.

The userland, however, should simply not break, as Linus enforces strict backwards compatibility of userland interfaces.

This backwards compatibility is just awesome, it makes getting and running the latest master painless.

This also makes this repo the perfect setup to develop the Linux kernel.

When we had a local patchset on top of mainline, this is how we updated it:

But we have since moved to running just mainline, which makes the update simpler.

The kernel is not forward compatible, however, so downgrading the Linux kernel requires downgrading the userland too to the latest Buildroot branch that supports it.

The default Linux kernel version is bumped in Buildroot with commit messages of type:

So you can try:

Those commits change BR2_LINUX_KERNEL_LATEST_VERSION in /linux/Config.in.

You should then look up if there is a branch that supports that kernel. Staying on branches is a good idea as they will get backports, in particular ones that fix the build as newer host versions come out.

The Linux kernel allows passing module parameters at insertion time through the init_module and finit_module system calls:

Outcome: the test passes:

Sources:

• kernel_module/params.c

• rootfs_overlay/params.sh

As shown in the example, module parameters can also be read and modified at runtime from sysfs.

We can obtain the help text of the parameters with:

The output contains:

One module can depend on symbols of another module that are exported with EXPORT_SYMBOL:

Outcome: the test passes:

Sources:

• kernel_module/dep.c

• kernel_module/dep2.c

• rootfs_overlay/dep.sh

The kernel deduces dependencies based on the EXPORT_SYMBOL that each module uses.

Symbols exported by EXPORT_SYMBOL can be seen with:

sample output:

This requires CONFIG_KALLSYMS_ALL=y.

Dependency information is stored by the kernel module build system in the .ko files' modinfo, e.g.:

contains:

We can double check with:

The output contains:

Module dependencies are also stored at:

Output:

TODO: what for, and at which point point does Buildroot / BusyBox generate that file?

Module metadata is stored on module files at compile time. Some of the fields can be retrieved through the THIS_MODULE struct module:

Dmesg output:

Source: kernel_module/module_info.c

Some of those are also present on sysfs:

Output:

And we can also observe them with the modinfo command line utility:

sample output:

Module information is stored in a special .modinfo section of the ELF file:

contains:

and:

gives:

I think a dedicated section is used to allow the Linux kernel and command line tools to easily parse that information from the ELF file as we’ve done with readelf.

Bibliography:

• https://stackoverflow.com/questions/19467150/significance-of-this-module-in-linux-driver/49812248#49812248

• https://stackoverflow.com/questions/4839024/how-to-find-the-version-of-a-compiled-kernel-module/42556565#42556565

• https://unix.stackexchange.com/questions/238167/how-to-understand-the-modinfo-output

Vermagic is a magic string present in the kernel and on modinfo of kernel modules. It is used to verify that the kernel module was compiled against a compatible kernel version and relevant configuration:

Possible dmesg output:

Source: kernel_module/vermagic.c

If we artificially create a mismatch with MODULE_INFO(vermagic, the insmod fails with:

and dmesg says the expected and found vermagic found:

Source: kernel_module/vermagic_fail.c

The kernel’s vermagic is defined based on compile time configurations at include/linux/vermagic.h:

The SMP part of the string for example is defined on the same file based on the value of CONFIG_SMP:

TODO how to get the vermagic from running kernel from userland? https://lists.kernelnewbies.org/pipermail/kernelnewbies/2012-October/006306.html

kmod modprobe has a flag to skip the vermagic check:

This option just strips modversion information from the module before loading, so it is not a kernel feature.

init_module and cleantup_module are an older alternative to the module_init and module_exit macros:

Dmesg output:

Source: kernel_module/module_init.c

TODO why were module_init and module_exit created? https://stackoverflow.com/questions/3218320/what-is-the-difference-between-module-init-and-init-module-in-a-linux-kernel-mod

insmod /panic.ko
insmod /oops.ko

echo c > /proc/sysrq-trigger

On panic, the kernel dies, and so does our terminal.

Make the kernel reboot after n seconds after panic:

Can also be controlled with the panic= kernel boot parameter.

0 to disable: https://unix.stackexchange.com/questions/29567/how-to-configure-the-linux-kernel-to-reboot-on-panic/29569#29569

The panic trace looks like:

Notice how our panic message hello panic is visible at:

On oops, the shell still lives after.

However we:

• leave the normal control flow, and oops after never gets printed: an interrupt is serviced

• cannot rmmod oops afterwards

It is possible to make oops lead to panics always with:

An oops stack trace looks like:

To find the line that oopsed, look at the RIP register:

and then on GDB:

run

which gives us the correct line:

This-did not work on arm due to GDB step debug kernel module ARM so we need to either:

• GDB module_init

• Kernel module stack trace to source line post-mortem method

The dump_stack function produces a stack trace much like panic and oops, but causes no problems and we return to the normal control flow, and can cleanly remove the module afterwards:

Source: kernel_module/dump_stack.c

The WARN_ON macro basically just calls dump_stack.

One extra side effect is that we can make it also panic with:

Source: kernel_module/warn_on.c

Can also be activated with the panic_on_warn boot parameter.

Debugfs is the simplest pseudo filesystem to play around with:

Outcome: the test passes:

Sources:

• kernel_module/debugfs.c

• rootfs_overlay/debugfs.sh

Debugfs is made specifically to help test kernel stuff. Just mount, set File operations, and we are done.

For this reason, it is the filesystem that we use whenever possible in our tests.

debugfs.sh explicitly mounts a debugfs at a custom location, but the most common mount point is /sys/kernel/debug.

This mount not done automatically by the kernel however: we, like most distros, do it from userland with our fstab.

Debugfs support requires the kernel to be compiled with CONFIG_DEBUG_FS=y.

Only the more basic file operations can be implemented in debugfs, e.g. mmap never gets called:

• https://patchwork.kernel.org/patch/9252557/

• https://github.com/torvalds/linux/blob/v4.9/fs/debugfs/file.c#L212

Bibliography: https://github.com/chadversary/debugfs-tutorial

Procfs is just another fops entry point:

Outcome: the test passes:

Procfs is a little less convenient than debugfs, but is more used in serious applications.

Procfs can run all system calls, including ones that debugfs can’t, e.g. mmap.

Sources:

• kernel_module/procfs.c

• rootfs_overlay/procfs.sh

Bibliography: https://stackoverflow.com/questions/8516021/proc-create-example-for-kernel-module/18924359#18924359

Sysfs is more restricted than procfs, as it does not take an arbitrary file_operations:

Outcome: the test passes:

Sources:

• kernel_module/sysfs.c

• rootfs_overlay/sysfs.sh

Vs procfs:

• https://unix.stackexchange.com/questions/4884/what-is-the-difference-between-procfs-and-sysfs

• https://stackoverflow.com/questions/37237835/how-to-attach-file-operations-to-sysfs-attribute-in-platform-driver

You basically can only do open, close, read, write, and lseek on sysfs files.

It is similar to a seq_file file operation, except that write is also implemented.

TODO: what are those kobject structs? Make a more complex example that shows what they can do.

Bibliography:

• https://github.com/t3rm1n4l/kern-dev-tutorial/blob/1f036ef40fc4378f5c8d2842e55bcea7c6f8894a/05-sysfs/sysfs.c

• https://www.kernel.org/doc/Documentation/kobject.txt

• https://www.quora.com/What-are-kernel-objects-Kobj

• http://www.makelinux.net/ldd3/chp-14-sect-1

• https://www.win.tue.nl/~aeb/linux/lk/lk-13.html

Character devices can have arbitrary File operations associated to them:

Outcome: the test passes:

Sources:

• rootfs_overlay/character_device.sh

• rootfs_overlay/mknoddev.sh

• kernel_module/character_device.c

Unlike procfs entires, character device files are created with userland mknod or mknodat syscalls:

Intuitively, for physical devices like keyboards, the major number maps to which driver, and the minor number maps to which device it is.

A single driver can drive multiple compatible devices.

The major and minor numbers can be observed with:

Output:

which means:

• c (first letter): this is a character device. Would be b for a block device.

• 1, 9: the major number is 1, and the minor 9

To avoid device number conflicts when registering the driver we:

• ask the kernel to allocate a free major number for us with: register_chrdev(0

• find ouf which number was assigned by grepping /proc/devices for the kernel module name

Bibliography: https://unix.stackexchange.com/questions/37829/understanding-character-device-or-character-special-files/371758#371758

File operations are the main method of userland driver communication. struct file_operations determines what the kernel will do on filesystem system calls of Pseudo filesystems.

This example illustrates the most basic system calls: open, read, write, close and lseek:

Outcome: the test passes:

Sources:

• kernel_module/fops.c

• rootfs_overlay/fops.sh

Then give this a try:

We have put printks on each fop, so this allows you to see which system calls are being made for each command.

No, there no official documentation: http://stackoverflow.com/questions/15213932/what-are-the-struct-file-operations-arguments

Writing trivial read File operations is repetitive and error prone. The seq_file API makes the process much easier for those trivial cases:

Outcome: the test passes:

Sources:

• kernel_module/seq_file.c

• rootfs_overlay/seq_file.sh

In this example we create a debugfs file that behaves just like a file that contains:

However, we only store a single integer in memory and calculate the file on the fly in an iterator fashion.

seq_file does not provide write: https://stackoverflow.com/questions/30710517/how-to-implement-a-writable-proc-file-by-using-seq-file-in-a-driver-module

Bibliography:

• Documentation/filesystems/seq_file.txt

• https://stackoverflow.com/questions/25399112/how-to-use-a-seq-file-in-linux-modules

The poll system call allows an user process to do a non-busy wait on a kernel event:

Outcome: jiffies gets printed to stdout every second from userland.

Sources:

• kernel_module/poll.c

• kernel_module/poll.c

• rootfs_overlay/poll.sh

Typically, we are waiting for some hardware to make some piece of data available available to the kernel.

The hardware notifies the kernel that the data is ready with an interrupt.

To simplify this example, we just fake the hardware interrupts with a kthread that sleeps for a second in an infinite loop.

Bibliography: https://stackoverflow.com/questions/30035776/how-to-add-poll-function-to-the-kernel-module-code/44645336#44645336

The ioctl system call is the best way to pass an arbitrary number of parameters to the kernel in a single go:

Outcome: the test passes:

Sources:

• kernel_module/ioctl.c

• kernel_module/ioctl.h

• kernel_module/user/ioctl.c

• rootfs_overlay/ioctl.sh

ioctl is one of the most important methods of communication with real device drivers, which often take several fields as input.

ioctl takes as input:

• an integer request : it usually identifies what type of operation we want to do on this call

• an untyped pointer to memory: can be anything, but is typically a pointer to a struct

  The type of the struct often depends on the request input

  This struct is defined on a uapi-style C header that is used both to compile the kernel module and the userland executable.

  The fields of this struct can be thought of as arbitrary input parameters.

And the output is:

• an integer return value. man ioctl documents:

  Usually, on success zero is returned. A few ioctl() requests use the return value as an output parameter and return a nonnegative value on success. On error, -1 is returned, and errno is set appropriately.

• the input pointer data may be overwritten to contain arbitrary output

Bibliography:

• https://stackoverflow.com/questions/2264384/how-do-i-use-ioctl-to-manipulate-my-kernel-module/44613896#44613896

• https://askubuntu.com/questions/54239/problem-with-ioctl-in-a-simple-kernel-module/926675#926675

The mmap system call allows us to share memory between user and kernel space without copying:

Outcome: the test passes:

Sources:

• kernel_module/mmap.c

• kernel_module/user/mmap.c

• rootfs_overlay/mmap.sh

In this example, we make a tiny 4 byte kernel buffer available to user-space, and we then modify it on userspace, and check that the kernel can see the modification.

mmap, like most more complex File operations, does not work with debugfs as of 4.9, so we use a procfs file for it.

Example adapted from: https://coherentmusings.wordpress.com/2014/06/10/implementing-mmap-for-transferring-data-from-user-space-to-kernel-space/

Bibliography:

• https://stackoverflow.com/questions/10760479/mmap-kernel-buffer-to-user-space/10770582#10770582

• https://stackoverflow.com/questions/1623008/allocating-memory-for-user-space-from-kernel-thread

• https://stackoverflow.com/questions/6967933/mmap-mapping-in-user-space-a-kernel-buffer-allocated-with-kmalloc

• https://github.com/jeremytrimble/ezdma

• https://github.com/simonjhall/dma

• https://github.com/ikwzm/udmabuf

Anonymous inodes allow getting multiple file descriptors from a single filesystem entry, which reduces namespace pollution compared to creating multiple device files:

Outcome: the test passes:

Sources:

• kernel_module/anonymous_inode.c

• kernel_module/anonymous_inode.h

• kernel_module/user/anonymous_inode.c

• rootfs_overlay/anonymous_inode.sh

This example gets an anonymous inode via ioctl from a debugfs entry by using anon_inode_getfd.

Reads to that inode return the sequence: 1, 10, 100, …​ 10000000, 1, 100, …​

Bibliography: https://stackoverflow.com/questions/4508998/what-is-an-anonymous-inode-in-linux/44388030#44388030

Netlink sockets offer a socket API for kernel / userland communication:

Outcome: the test passes:

Sources:

• kernel_module/netlink.c

• kernel_module/netlink.h

• kernel_module/user/netlink.c

• rootfs_overlay/netlink.sh

Launch multiple user requests in parallel to stress our socket:

TODO: what is the advantage over read, write and poll? https://stackoverflow.com/questions/16727212/how-netlink-socket-in-linux-kernel-is-different-from-normal-polling-done-by-appl

Bibliography:

• https://stackoverflow.com/questions/3299386/how-to-use-netlink-socket-to-communicate-with-a-kernel-module

• https://en.wikipedia.org/wiki/Netlink

insmod /kthread.ko

0
1
2
...
8
9
0
1
2
...

rmmod kthread

Let’s launch two threads and see if they actually run in parallel:

Source: kernel_module/kthreads.c

Outcome: two threads count to dmesg from 0 to 9 in parallel.

Each line has output of form:

Possible very likely outcome:

The threads almost always interleaved nicely, thus confirming that they are actually running in parallel.

Count to dmesg every one second from 0 up to n - 1:

Source: kernel_module/sleep.c

The sleep is done with a call to usleep_range directly inside module_init for simplicity.

Bibliography:

• https://stackoverflow.com/questions/15994603/how-to-sleep-in-the-linux-kernel/44153288#44153288

• https://github.com/torvalds/linux/blob/v4.17/Documentation/timers/timers-howto.txt

A more convenient front-end for kthread:

Outcome: count from 0 to 9 infinitely many times

Stop counting:

Source: kernel_module/workqueue_cheat.c

The workqueue thread is killed after the worker function returns.

We can’t call the module just workqueue.c because there is already a built-in with that name: https://unix.stackexchange.com/questions/364956/how-can-insmod-fail-with-kernel-module-is-already-loaded-even-is-lsmod-does-not

Bibliography: https://github.com/torvalds/linux/blob/v4.17/Documentation/core-api/workqueue.rst

Let’s block the entire kernel! Yay:

Outcome: the system hangs, the only way out is to kill the VM.

Source: kernel_module/schedule.c

kthreads only allow interrupting if you call schedule(), and the schedule=0 kernel module parameter turns it off.

Sleep functions like usleep_range also end up calling schedule.

If we allow schedule() to be called, then the system becomes responsive:

and we can observe the counting with:

The system also responds if we add another core:

Wait queues are a way to make a thread sleep until an event happens on the queue:

Dmesg output:

Stop the count:

Source: kernel_module/wait_queue.c

This example launches three threads:

• one thread generates events every with wake_up

• the other two threads wait for that with wait_event, and print a dmesg when it happens.

  The wait_event macro works a bit like:

  while (!cond)
      sleep_until_event

insmod /timer.ko

rmmod timer

Brute force monitor every shared interrupt that will accept us:

Source: kernel_module/irq.c.

Now try the following:

• press a keyboard key and then release it after a few seconds

• press a mouse key, and release it after a few seconds

• move the mouse around

Outcome: dmesg shows which IRQ was fired for each action through messages of type:

dev is the character device for the module and never changes, as can be confirmed by:

The IRQs that we observe are:

• 1 for keyboard press and release.

  If you hold the key down for a while, it starts firing at a constant rate. So this happens at the hardware level!

• 12 mouse actions

This only works if for IRQs for which the other handlers are registered as IRQF_SHARED.

We can see which ones are those, either via dmesg messages of type:

which indicate that 0 is not, but 1 is, or with:

which shows:

so only 1 has myirqhandler0 attached but not 0.

The QEMU monitor also has some interrupt statistics for x86_64:

TODO: properly understand how each IRQ maps to what number.

The Linux kernel v4.16 mainline also has a dummy-irq module at drivers/misc/dummy-irq.c for monitoring a single IRQ.

We build it by default with:

And then you can do

and in guest:

Outcome: when you click a key on the keyboard, dmesg shows:

However, this module is intended to fire only once as can be seen from its source:

and furthermore interrupt 1 and 12 happen immediately TODO why, were they somehow pending?

So so see something interesting, you need to monitor an interrupt that is more rare than the keyboard, e.g. platform_device.

In the guest on Graphic mode:

Then see how clicking the mouse and keyboard affect the interrupt counts.

This confirms that:

• 1: keyboard

• 12: mouse click and drags

The module also shows which handlers are registered for each IRQ, as we have observed at irq.ko

When in text mode, we can also observe interrupt line 4 with handler ttyS0 increase continuously as IO goes through the UART.

Convert a string to an integer:

Outcome: the test passes:

Sources:

• kernel_module/kstrto.c

• rootfs_overlay/kstrto.sh

Bibliography: https://stackoverflow.com/questions/6139493/how-convert-char-to-int-in-linux-kernel/49811658#49811658

Convert a virtual address to physical:

Source: kernel_module/virt_to_phys.c

Sample output:

We can confirm that the kmalloc_ptr translation worked with:

which reads four bytes from a given physical address, and gives the expected:

TODO it only works for kmalloc however, for the static variable:

it gave a wrong value of 00000000.

Bibliography:

• https://stackoverflow.com/questions/5748492/is-there-any-api-for-determining-the-physical-address-from-virtual-address-in-li/45128487#45128487

• https://stackoverflow.com/questions/39134990/mmap-of-dev-mem-fails-with-invalid-argument-for-virt-to-phys-address-but-addre/45127582#45127582

• https://stackoverflow.com/questions/43325205/can-we-use-virt-to-phys-for-user-space-memory-in-kernel-module