1.. highlight:: none
2
3Debugging kernel and modules via gdb
4====================================
5
6The kernel debugger kgdb, hypervisors like QEMU or JTAG-based hardware
7interfaces allow to debug the Linux kernel and its modules during runtime
8using gdb. Gdb comes with a powerful scripting interface for python. The
9kernel provides a collection of helper scripts that can simplify typical
10kernel debugging steps. This is a short tutorial about how to enable and use
11them. It focuses on QEMU/KVM virtual machines as target, but the examples can
12be transferred to the other gdb stubs as well.
13
14
15Requirements
16------------
17
18- gdb 7.2+ (recommended: 7.4+) with python support enabled (typically true
19  for distributions)
20
21
22Setup
23-----
24
25- Create a virtual Linux machine for QEMU/KVM (see www.linux-kvm.org and
26  www.qemu.org for more details). For cross-development,
27  http://landley.net/aboriginal/bin keeps a pool of machine images and
28  toolchains that can be helpful to start from.
29
30- Build the kernel with CONFIG_GDB_SCRIPTS enabled, but leave
31  CONFIG_DEBUG_INFO_REDUCED off. If your architecture supports
32  CONFIG_FRAME_POINTER, keep it enabled.
33
34- Install that kernel on the guest.
35  Alternatively, QEMU allows to boot the kernel directly using -kernel,
36  -append, -initrd command line switches. This is generally only useful if
37  you do not depend on modules. See QEMU documentation for more details on
38  this mode.
39
40- Enable the gdb stub of QEMU/KVM, either
41
42    - at VM startup time by appending "-s" to the QEMU command line
43
44  or
45
46    - during runtime by issuing "gdbserver" from the QEMU monitor
47      console
48
49- cd /path/to/linux-build
50
51- Start gdb: gdb vmlinux
52
53  Note: Some distros may restrict auto-loading of gdb scripts to known safe
54  directories. In case gdb reports to refuse loading vmlinux-gdb.py, add::
55
56    add-auto-load-safe-path /path/to/linux-build
57
58  to ~/.gdbinit. See gdb help for more details.
59
60- Attach to the booted guest::
61
62    (gdb) target remote :1234
63
64
65Examples of using the Linux-provided gdb helpers
66------------------------------------------------
67
68- Load module (and main kernel) symbols::
69
70    (gdb) lx-symbols
71    loading vmlinux
72    scanning for modules in /home/user/linux/build
73    loading @0xffffffffa0020000: /home/user/linux/build/net/netfilter/xt_tcpudp.ko
74    loading @0xffffffffa0016000: /home/user/linux/build/net/netfilter/xt_pkttype.ko
75    loading @0xffffffffa0002000: /home/user/linux/build/net/netfilter/xt_limit.ko
76    loading @0xffffffffa00ca000: /home/user/linux/build/net/packet/af_packet.ko
77    loading @0xffffffffa003c000: /home/user/linux/build/fs/fuse/fuse.ko
78    ...
79    loading @0xffffffffa0000000: /home/user/linux/build/drivers/ata/ata_generic.ko
80
81- Set a breakpoint on some not yet loaded module function, e.g.::
82
83    (gdb) b btrfs_init_sysfs
84    Function "btrfs_init_sysfs" not defined.
85    Make breakpoint pending on future shared library load? (y or [n]) y
86    Breakpoint 1 (btrfs_init_sysfs) pending.
87
88- Continue the target::
89
90    (gdb) c
91
92- Load the module on the target and watch the symbols being loaded as well as
93  the breakpoint hit::
94
95    loading @0xffffffffa0034000: /home/user/linux/build/lib/libcrc32c.ko
96    loading @0xffffffffa0050000: /home/user/linux/build/lib/lzo/lzo_compress.ko
97    loading @0xffffffffa006e000: /home/user/linux/build/lib/zlib_deflate/zlib_deflate.ko
98    loading @0xffffffffa01b1000: /home/user/linux/build/fs/btrfs/btrfs.ko
99
100    Breakpoint 1, btrfs_init_sysfs () at /home/user/linux/fs/btrfs/sysfs.c:36
101    36              btrfs_kset = kset_create_and_add("btrfs", NULL, fs_kobj);
102
103- Dump the log buffer of the target kernel::
104
105    (gdb) lx-dmesg
106    [     0.000000] Initializing cgroup subsys cpuset
107    [     0.000000] Initializing cgroup subsys cpu
108    [     0.000000] Linux version 3.8.0-rc4-dbg+ (...
109    [     0.000000] Command line: root=/dev/sda2 resume=/dev/sda1 vga=0x314
110    [     0.000000] e820: BIOS-provided physical RAM map:
111    [     0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable
112    [     0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved
113    ....
114
115- Examine fields of the current task struct::
116
117    (gdb) p $lx_current().pid
118    $1 = 4998
119    (gdb) p $lx_current().comm
120    $2 = "modprobe\000\000\000\000\000\000\000"
121
122- Make use of the per-cpu function for the current or a specified CPU::
123
124    (gdb) p $lx_per_cpu("runqueues").nr_running
125    $3 = 1
126    (gdb) p $lx_per_cpu("runqueues", 2).nr_running
127    $4 = 0
128
129- Dig into hrtimers using the container_of helper::
130
131    (gdb) set $next = $lx_per_cpu("hrtimer_bases").clock_base[0].active.next
132    (gdb) p *$container_of($next, "struct hrtimer", "node")
133    $5 = {
134      node = {
135        node = {
136          __rb_parent_color = 18446612133355256072,
137          rb_right = 0x0 <irq_stack_union>,
138          rb_left = 0x0 <irq_stack_union>
139        },
140        expires = {
141          tv64 = 1835268000000
142        }
143      },
144      _softexpires = {
145        tv64 = 1835268000000
146      },
147      function = 0xffffffff81078232 <tick_sched_timer>,
148      base = 0xffff88003fd0d6f0,
149      state = 1,
150      start_pid = 0,
151      start_site = 0xffffffff81055c1f <hrtimer_start_range_ns+20>,
152      start_comm = "swapper/2\000\000\000\000\000\000"
153    }
154
155
156List of commands and functions
157------------------------------
158
159The number of commands and convenience functions may evolve over the time,
160this is just a snapshot of the initial version::
161
162 (gdb) apropos lx
163 function lx_current -- Return current task
164 function lx_module -- Find module by name and return the module variable
165 function lx_per_cpu -- Return per-cpu variable
166 function lx_task_by_pid -- Find Linux task by PID and return the task_struct variable
167 function lx_thread_info -- Calculate Linux thread_info from task variable
168 lx-dmesg -- Print Linux kernel log buffer
169 lx-lsmod -- List currently loaded modules
170 lx-symbols -- (Re-)load symbols of Linux kernel and currently loaded modules
171
172Detailed help can be obtained via "help <command-name>" for commands and "help
173function <function-name>" for convenience functions.
174