Intro to Using `gdb` to Debug the Linux Kernel

There are many great tools that are useful for debugging the Linux kernel, including good old-fashioned printk, ftrace, and kgdb. In this post we’ll be exploring how to use the kernel debugger (kgdb) to debug a QEMU VM, although some of the techniques below may be applied to debugging via hardware interfaces like JTAG. Using gdb as a front-end for the kernel debugger allows us to debug the kernel in the familiar and powerful debugging interface of gdb.

Building the Kernel

When building a kernel for debugging with gdb, I would advise using the following configuration options to make debugging a bit more pleasant.

CONFIG_DEBUG_INFO Including debug information in the kernel and kernel modules will make both the image and the modules larger in size, but is a required option for debugging the kernel or kernel modules with gdb. Under the hood this option adds the -g option to the compiler flags used by gcc.

CONFIG_DEBUG_INFO_SPLIT Split out the debug info for the kernel and kernel modules into separate .dwo files. This significantly reduces the size of the kernel image and kernel modules installed on the device or VM we will be debugging. Note that this option requires a gcc version greater than or equal to version 4.7, as it adds the option -gsplit-dwarfto the compiler flags.

CONFIG_GDB_SCRIPTS Adds links to the GDB helper scripts. I have found this option to be extremely useful. I find it particularly useful when debugging a kernel module, when I need to inspect the kernel log buffer or VFS mounts, or when I have to do anything with tasks (more on this later).

CONFIG_KGDB Enables the built in kernel debugger, which allows for remote debugging. Technically this option is the only one that is strictly required, but attempting to debug without debug symbols will make debugging much harder.

CONFIG_FRAME_POINTER Depending on the architecture of the VM you’re running, you may also want to enable CONFIG_FRAME_POINTER. This option adds the compiler flag -fno-omit-frame-pointer and greatly improves the reliability of backtraces.

A Note on Kernel Address Space Layout Randomization (KASLR)

KASLR changes the base address where the kernel code is placed at boot time. If KASLR is enabled (CONFIG_RANDOMIZE_BASE is set to y) in your kernel configuration, setting breakpoints from gdb will not work unless you also later add nokaslr to the kernel command-line parameters.

Running the VM and Setting Your First Breakpoint

Set up `gdb`

Before starting the VM and attempting to attach gdb, set up gdb to load the Linux helper scripts by adding add-auto-load-safe-path to your ~/.gdbinit.

Start the VM

Before we start the VM there are a few QEMU command-line parameters that are worth reviewing:

-gdb tcp::<port> Port to run the gdbserver on

-S Freeze the CPU on startup (useful for debugging early steps in the kernel)

-kernel <path> Path to kernel image to debug

-initrd <path> Path to initial ramdisk

-append <cmdline> Linux kernel command-line parameters

Note that the kernel, initrd, and append parameters are not necessarily needed if you are not using direct kernel boot, but instead are booting from a disk with a bootloader installed. Regardless of your method of booting, if KASLR is enabled in your kernel configuration, you need to add nokaslr to your command-line parameters as noted in the previous section.

After building the kernel you can start the VM with something like:

qemu-system-x86_64 \
    -kernel $KERNEL_SRC/arch/x86/boot/bzImage \
    -append "console=ttyS0,115200 nokaslr" \
    -gdb tcp::1234 \
    -S

Note the use of the -S parameter to halt the CPU until we attach a debugger.

Setting your first breakpoint

In a separate terminal window we can attach gdb to this halted VM and set a breakpoint. In the example below we will break at start_kernel.

1. Load the symbols for the kernel.

gdb ./vmlinux

2. Attach gdb to the halted VM instance.

(gdb) target remote :1234

3. Set a breakpoint at `start_kernel` and resume execution

(gdb) hbreak start_kernel
Hardware assisted breakpoint 1 at <addr>: file init/main.c, line <line number>.
(gdb) continue

If you’ve configured everything properly you should see something like the following:

(gdb) c
Continuing.

Breakpoint 1, start_kernel () at init/main.c:<line number>
<line number>     {

Note the use of hbreak to set a hardware assisted breakpoint instead of the more commonly used break. To debug earlier steps in the kernel you’ll need to use a hardware assisted breakpoint. Since the kernel has not had a chance to install any exception handlers for software breakpoints you cannot use the more typical break command to halt execution at start_kernel in the example above.

Utility functions

Several common Linux functions like container_of are implemented as gdb functions and commands. To list these run:

apropos lx

The lx-symbols function is particularly important if you want to debug a kernel module. After starting the VM and loading the module, use the lx-symbols function to load symbols for all modules currently loaded in the kernel. If debugging an out-of-tree module, use the first argument to provide the path of the root directory to search.

Practical advice

The ability to set breakpoints in the kernel does kind of feel like a superpower, but it also gets a bit daunting when you need to set a breakpoint in vfs_open which may be triggered thousands of times for completely unrelated tasks. This is where gdb breakpoint conditionsbecome critical.

For example, if you wanted to set a breakpoint in do_exit for the process with the pid 42, you might set a breakpoint like the following:

br do_exit if $lx_current()->pid == 42

Or if you’re setting a breakpoint in vfs_open, but only care about the file named test, you might use something like the following:

br vfs_open if $_streq(path.dentry->d_iname, "test")

You might also check out the gdb convenience functions for other useful functions like $_streq() implemented in gdb.

Creating debug-able VMs from `libvirt` and `Vagrant`

`libvirt`

When managing a QEMU VM with libvirt it is possible to use the qemu:commandline and qemu:arg tags to append the necessary command line arguments to QEMU to start the gdbserver.

<domain type='kvm' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'>
  ...
  <qemu:commandline>
    <qemu:arg value='-gdb'/>
    <qemu:arg value='tcp::1234'/>
  </qemu:commandline>
</domain>

`Vagrant`

Since we’re able to create a debuggable VM with libvirt it is also possible to create a VM for debugging the kernel with vagrant-libvirt. After installing the vagrant-libvirt plugin you should be able to use a Vagrantfile with the following to configure a libvirt VM with a configuration similar to what is listed in the previous section:

Vagrant.configure(2) do |config|
  ...
  config.vm.provider :libvirt do |virt|
    ...
    # GDB args
    virt.qemuargs :value => "-gdb"
    virt.qemuargs :value => "tcp::1234"
  end
end