linux/Documentation/BUG-HUNTING
<<
>>
Prefs
   1Table of contents
   2=================
   3
   4Last updated: 20 December 2005
   5
   6Contents
   7========
   8
   9- Introduction
  10- Devices not appearing
  11- Finding patch that caused a bug
  12-- Finding using git-bisect
  13-- Finding it the old way
  14- Fixing the bug
  15
  16Introduction
  17============
  18
  19Always try the latest kernel from kernel.org and build from source. If you are
  20not confident in doing that please report the bug to your distribution vendor
  21instead of to a kernel developer.
  22
  23Finding bugs is not always easy. Have a go though. If you can't find it don't
  24give up. Report as much as you have found to the relevant maintainer. See
  25MAINTAINERS for who that is for the subsystem you have worked on.
  26
  27Before you submit a bug report read REPORTING-BUGS.
  28
  29Devices not appearing
  30=====================
  31
  32Often this is caused by udev. Check that first before blaming it on the
  33kernel.
  34
  35Finding patch that caused a bug
  36===============================
  37
  38
  39
  40Finding using git-bisect
  41------------------------
  42
  43Using the provided tools with git makes finding bugs easy provided the bug is
  44reproducible.
  45
  46Steps to do it:
  47- start using git for the kernel source
  48- read the man page for git-bisect
  49- have fun
  50
  51Finding it the old way
  52----------------------
  53
  54[Sat Mar  2 10:32:33 PST 1996 KERNEL_BUG-HOWTO lm@sgi.com (Larry McVoy)]
  55
  56This is how to track down a bug if you know nothing about kernel hacking.
  57It's a brute force approach but it works pretty well.
  58
  59You need:
  60
  61        . A reproducible bug - it has to happen predictably (sorry)
  62        . All the kernel tar files from a revision that worked to the
  63          revision that doesn't
  64
  65You will then do:
  66
  67        . Rebuild a revision that you believe works, install, and verify that.
  68        . Do a binary search over the kernels to figure out which one
  69          introduced the bug.  I.e., suppose 1.3.28 didn't have the bug, but
  70          you know that 1.3.69 does.  Pick a kernel in the middle and build
  71          that, like 1.3.50.  Build & test; if it works, pick the mid point
  72          between .50 and .69, else the mid point between .28 and .50.
  73        . You'll narrow it down to the kernel that introduced the bug.  You
  74          can probably do better than this but it gets tricky.
  75
  76        . Narrow it down to a subdirectory
  77
  78          - Copy kernel that works into "test".  Let's say that 3.62 works,
  79            but 3.63 doesn't.  So you diff -r those two kernels and come
  80            up with a list of directories that changed.  For each of those
  81            directories:
  82
  83                Copy the non-working directory next to the working directory
  84                as "dir.63".
  85                One directory at time, try moving the working directory to
  86                "dir.62" and mv dir.63 dir"time, try
  87
  88                        mv dir dir.62
  89                        mv dir.63 dir
  90                        find dir -name '*.[oa]' -print | xargs rm -f
  91
  92                And then rebuild and retest.  Assuming that all related
  93                changes were contained in the sub directory, this should
  94                isolate the change to a directory.
  95
  96                Problems: changes in header files may have occurred; I've
  97                found in my case that they were self explanatory - you may
  98                or may not want to give up when that happens.
  99
 100        . Narrow it down to a file
 101
 102          - You can apply the same technique to each file in the directory,
 103            hoping that the changes in that file are self contained.
 104
 105        . Narrow it down to a routine
 106
 107          - You can take the old file and the new file and manually create
 108            a merged file that has
 109
 110                #ifdef VER62
 111                routine()
 112                {
 113                        ...
 114                }
 115                #else
 116                routine()
 117                {
 118                        ...
 119                }
 120                #endif
 121
 122            And then walk through that file, one routine at a time and
 123            prefix it with
 124
 125                #define VER62
 126                /* both routines here */
 127                #undef VER62
 128
 129            Then recompile, retest, move the ifdefs until you find the one
 130            that makes the difference.
 131
 132Finally, you take all the info that you have, kernel revisions, bug
 133description, the extent to which you have narrowed it down, and pass
 134that off to whomever you believe is the maintainer of that section.
 135A post to linux.dev.kernel isn't such a bad idea if you've done some
 136work to narrow it down.
 137
 138If you get it down to a routine, you'll probably get a fix in 24 hours.
 139
 140My apologies to Linus and the other kernel hackers for describing this
 141brute force approach, it's hardly what a kernel hacker would do.  However,
 142it does work and it lets non-hackers help fix bugs.  And it is cool
 143because Linux snapshots will let you do this - something that you can't
 144do with vendor supplied releases.
 145
 146Fixing the bug
 147==============
 148
 149Nobody is going to tell you how to fix bugs. Seriously. You need to work it
 150out. But below are some hints on how to use the tools.
 151
 152To debug a kernel, use objdump and look for the hex offset from the crash
 153output to find the valid line of code/assembler. Without debug symbols, you
 154will see the assembler code for the routine shown, but if your kernel has
 155debug symbols the C code will also be available. (Debug symbols can be enabled
 156in the kernel hacking menu of the menu configuration.) For example:
 157
 158    objdump -r -S -l --disassemble net/dccp/ipv4.o
 159
 160NB.: you need to be at the top level of the kernel tree for this to pick up
 161your C files.
 162
 163If you don't have access to the code you can also debug on some crash dumps
 164e.g. crash dump output as shown by Dave Miller.
 165
 166>    EIP is at ip_queue_xmit+0x14/0x4c0
 167>     ...
 168>    Code: 44 24 04 e8 6f 05 00 00 e9 e8 fe ff ff 8d 76 00 8d bc 27 00 00
 169>    00 00 55 57  56 53 81 ec bc 00 00 00 8b ac 24 d0 00 00 00 8b 5d 08
 170>    <8b> 83 3c 01 00 00 89 44  24 14 8b 45 28 85 c0 89 44 24 18 0f 85
 171>
 172>    Put the bytes into a "foo.s" file like this:
 173>
 174>           .text
 175>           .globl foo
 176>    foo:
 177>           .byte  .... /* bytes from Code: part of OOPS dump */
 178>
 179>    Compile it with "gcc -c -o foo.o foo.s" then look at the output of
 180>    "objdump --disassemble foo.o".
 181>
 182>    Output:
 183>
 184>    ip_queue_xmit:
 185>        push       %ebp
 186>        push       %edi
 187>        push       %esi
 188>        push       %ebx
 189>        sub        $0xbc, %esp
 190>        mov        0xd0(%esp), %ebp        ! %ebp = arg0 (skb)
 191>        mov        0x8(%ebp), %ebx         ! %ebx = skb->sk
 192>        mov        0x13c(%ebx), %eax       ! %eax = inet_sk(sk)->opt
 193
 194In addition, you can use GDB to figure out the exact file and line
 195number of the OOPS from the vmlinux file. If you have
 196CONFIG_DEBUG_INFO enabled, you can simply copy the EIP value from the
 197OOPS:
 198
 199 EIP:    0060:[<c021e50e>]    Not tainted VLI
 200
 201And use GDB to translate that to human-readable form:
 202
 203  gdb vmlinux
 204  (gdb) l *0xc021e50e
 205
 206If you don't have CONFIG_DEBUG_INFO enabled, you use the function
 207offset from the OOPS:
 208
 209 EIP is at vt_ioctl+0xda8/0x1482
 210
 211And recompile the kernel with CONFIG_DEBUG_INFO enabled:
 212
 213  make vmlinux
 214  gdb vmlinux
 215  (gdb) p vt_ioctl
 216  (gdb) l *(0x<address of vt_ioctl> + 0xda8)
 217or, as one command
 218  (gdb) l *(vt_ioctl + 0xda8)
 219
 220If you have a call trace, such as :-
 221>Call Trace:
 222> [<ffffffff8802c8e9>] :jbd:log_wait_commit+0xa3/0xf5
 223> [<ffffffff810482d9>] autoremove_wake_function+0x0/0x2e
 224> [<ffffffff8802770b>] :jbd:journal_stop+0x1be/0x1ee
 225> ...
 226this shows the problem in the :jbd: module. You can load that module in gdb
 227and list the relevant code.
 228  gdb fs/jbd/jbd.ko
 229  (gdb) p log_wait_commit
 230  (gdb) l *(0x<address> + 0xa3)
 231or
 232  (gdb) l *(log_wait_commit + 0xa3)
 233
 234
 235Another very useful option of the Kernel Hacking section in menuconfig is
 236Debug memory allocations. This will help you see whether data has been
 237initialised and not set before use etc. To see the values that get assigned
 238with this look at mm/slab.c and search for POISON_INUSE. When using this an
 239Oops will often show the poisoned data instead of zero which is the default.
 240
 241Once you have worked out a fix please submit it upstream. After all open
 242source is about sharing what you do and don't you want to be recognised for
 243your genius?
 244
 245Please do read Documentation/SubmittingPatches though to help your code get
 246accepted.
 247
lxr.linux.no kindly hosted by Redpill Linpro AS, provider of Linux consulting and operations services since 1995.