Using ADB to Debug the UNIX* Kernel Samuel J. Leffler and William N. Joy Computer Systems Research Group Department of Electrical Engineering and Computer Science University of California, Berkeley Berkeley, California 94720 _A_B_S_T_R_A_C_T This document describes the facilities found in the 4.3BSD version of the VAX* UNIX debugger _a_d_b which may be used to debug the UNIX kernel. It discusses how standard _a_d_b commands may be used in examining the kernel and introduces the basics necessary for users to write _a_d_b command scripts which can augment the standard _a_d_b command set. The examination techniques described here may be applied both to running systems and the post- mortem dumps automatically created by the _s_a_v_e_c_o_r_e(8) program after a system crash. The reader is expected to have at least a passing fam- iliarity with the debugger command language. Revised June 3, 1986 _1. _I_n_t_r_o_d_u_c_t_i_o_n Modifications have been made to the standard VAX UNIX debugger _a_d_b to simplify examination of post-mortem dumps automatically generated following a system crash. These changes may also be used when examining UNIX in its normal operation. This document serves as an introduction to the _u_s_e of these facilities, and should not be construed as a description of _h_o_w _t_o _d_e_b_u_g _t_h_e _k_e_r_n_e_l. __________________________ *UNIX is a Trademark of Bell Laboratories. *DEC and VAX are trademarks of Digital Equipment Cor- poration. September 2, 1987 - 2 - Introduction _1._1. _I_n_v_o_c_a_t_i_o_n When examining post-mortem dumps of the UNIX kernel the -k option should be used, e.g. % adb -k vmunix.? vmcore.? where the appropriate version of the saved operating system image and core dump are supplied in place of ``?''. This flag causes _a_d_b to partially simulate the VAX virtual memory hardware when accessing the _c_o_r_e file. In addition the internal state maintained by the debugger is initialized from data structures maintained by the kernel explicitly for debugging|=. A running kernel may be examined in a similar fashion, % adb -k /vmunix /dev/mem _1._2. _E_s_t_a_b_l_i_s_h_i_n_g _C_o_n_t_e_x_t During initialization _a_d_b attempts to establish the context of the ``currently active process'' by examining the value of the kernel variable _m_a_s_t_e_r_p_a_d_d_r. This variable contains the virtual address of the process context block of the last process which was set executing by the _S_w_t_c_h rou- tine. _M_a_s_t_e_r_p_a_d_d_r normally provides sufficient information to locate the current stack frame (via the stack pointers found in the context block). By locating the process con- text block for the process _a_d_b may then perform virtual to physical address translation using that process's in-core page tables. When examining post-mortem dumps locating the most recent stack frame of the last currently active process can be nontrivial. This is due to the different ways in which state may be saved after a nonrecoverable error. Crashes may or may not be ``clean'' (i.e. the top of the interrupt stack contains a pointer to the process's kernel mode stack pointer and program counter); an ``unclean'' crash will occur, for instance, if the interrupt stack overflows. When _a_d_b is invoked on a post-mortem crash dump it tries to automatically establish the proper stack frame. This is done by first checking the stack pointer normally saved in the restart parameter block at _r_p_b+1fc (or _s_c_b-4). If this value does not point to a valid stack frame, _a_d_b searches the interrupt stack looking for a valid stack frame. Should this also fail _a_d_b then searches the kernel stack located in __________________________ |= If the -k flag is not used when invoking _a_d_b the user must explicitly calculate virtual addresses. With the -_k option _a_d_b interprets page tables to automatically perform virtual to physical address translation. September 2, 1987 - 3 - Introduction the user structure associated with the last executing pro- cess. If _a_d_b is able to locate a valid stack frame using this procedure the command $c will generate a stack trace from the last point at which the kernel was executing on behalf of the user process all the way to the top of the user process's stack (e.g. to the _m_a_i_n routine in the user process). Should _a_d_b be unable to locate a valid stack frame it prints a message and the current state is left undefined. When a stack trace of a particular process (other than that which was currently exe- cuting) is desired, an alternate method, described in 2.4, should be used. Additional information may be obtained from the kernel stack. Discussion of that subject is postponed until com- mand scripts have been introduced; see 2.2. _2. _C_o_m_m_a_n_d _S_c_r_i_p_t_s _2._1. _E_x_t_e_n_d_i_n_g _t_h_e _F_o_r_m_a_t_t_i_n_g _F_a_c_i_l_i_t_i_e_s Once the process context has been established, the com- plete _a_d_b command set is available for interpreting data structures. In addition, a number of _a_d_b scripts have been created to simplify the structured printing of commonly referenced kernel data structures. The scripts normally reside in the directory /_u_s_r/_l_i_b/_a_d_b, and are invoked with the ``$<'' operator. (A later table lists the standard scripts distributed with the system.) As an example, consider the following listing which contains a dump of a faulty process's state (our typing is shown emboldened). % adb -k vmunix.175 vmcore.175 sbr 5868 slr 2770 p0br 5a00 p0lr 236 p1br 6600 p1lr fff0 panic: dup biodone $c _boot() from _boot+f3 _boot(0,0) from _panic+3a _panic(800413d0) from _biodone+17 _biodone(800791e8) from _rxpurge+23 _rxpurge(80044754) from _rxstart+5a _rxstart(80044754) from 80031df8 _rxintr(0) from _Xrxintr0+11 _Xrxintr0(45b01,3aaf4) from 457f _Syssize(3aaf4) from 365a _Syssize() from 19a8 ?() from 2ff3 _Syssize(4,7fffe834) from 9cf3 September 2, 1987 - 4 - Command Scripts _Syssize(4,7fffe834,7fffe848) from 37 ?() u$l ,#l Place the value of the ``link'' in the _a_d_b variable ``4 *nproc>l *proc>f $l f ,#