January 7, 2009 ---------------------------------------------------------- panic: initiate_write_inodeabdlock0_:u fFsA2I:L UaRlEr e-a dlyo asdt adrattead acdp0u:i ds e=t tin2g KuDpB :D MeAn tfeari:l epdan "ad0: FAILURE - load data" panic: initiate_write_inodeblock_ufs2: already started "setting up DMA failed" KDB: enter: panic panic: cpu ident 2 --------------------------------------------------------- In KDB: (different panic?) kdb_enter panic softdep_disk_io_initiation ffs_geom_strategy bufwrite ffs_update ffs_fsync fsync syscall XFast_syscall -------------------------------------------------------- The next day: panic: iandi2t:i aFtAeI_LwUrRiEt e-_ ilnooadde bdlaotcak auafds22:: saeltrteiandgy uspt aDrMtAe df iilcepdu d-0 tEKODMB_:M IeRnRtOeRr:: pCaanninco u ad2: FAILURE - load data panic: initiate_write_inodeblock_ufs2: already started --------- 4xbonnie++, old kernelstill new kernel mirror complete: spew of: ad2: setting up DMA failed ad2: FAILURE - load data (ad2 drops out of array) --- rebuild mirror, still on oldnew kernel: 2xbonnie++ spew of: ad0: setting up DMA failed ad0: FAILURE - load data (ad0 drops out of array) stop bonnie++; rebuild array: ad2: FAILURE - load data ad2: setting up DMA failed [repeats several times] GEOM_MIRROR: Request failed (error=5). ad2s4[WRITE(offset=1733902336, length=16384)] g_vfs_done():mirror/gm0a[WRITE(offset={multiple entries}, $length)]error = 5 [repeats] GEOM_MIRROR: Synchronization request failed (error=5). mirror/gm0[READ[...]] [...] (rebuild halts? ... syslogd is pegging a CPU) (`less /var/log/messages` hangs (/var being on the mirror in question)) Hm, might be a deadlock between giant taskq and syslogd, but maybe it was just spinning ... [obtain mini-dump of possibly-deadlocked state, vmcore.0] ---- 9 January 2009 Rebuild finishes; reboot into single-user with intent to fsck. First disk access causes DMA setup failure on ad2, mirror degrades. Start rebuild. ad0 DMA setup failure, lots of g_vfs_done console messages. GEOM_MIRROR cannot update metadata on ad0s4. panic() ad0p: anFiAcI:L URiEn it-i altoea_dw rdiattea_ inaodd0e:b lsoectkt_iunfgs 2u:p aDlMrAe afdayi lsetda Gted EOcMp_uMiIdR R=O R1: KCDaBn:n oetn tueprd:a tep amneitca da --------------- could not use debugger; reboot-switch Array is auto-rebuilding on boot. -- rebuild finished 4xbonnie++; run for a few minutes, then massive spew and panic and deadlock g_vfs_done error=5 as well as DMA failure panic: initiate_inode_ufs2 or something like that; messages intertwined on rebood ad2 broken, mirror degraded forget/insert ad2 and rebuild -------------- 12 Jan 2009 Actually boot old kernel; mirror rebuilds fine. 4xbonnie++ complete successfully Boot to new (CURRENT) kernel, array degrades on boot with FAILED - load data and DMA spew. ad2 failed, this time. Try a new kernel with trasz's patch to avoid panics (possibly relevant), but the array still degrades. I didn't try very hard to make it panic, so that may still be present. ------------- 13 Jan 2009 Build new CURRENT kernel with WITNESS and INVARIANTS. 4xbonnie++ produces similar output, panic drops to debugger. While scrolling back and looking at history, something else hangs?! at db> prompt, but unresponsive (no keyboard LED change, even) On screen: sc_cngetc cncheckc cngetc db_readline db_read_line db_command_loop db_trap kdb_trap trap calltrap --- trap 0x3, rip = -xffffffff802e1c81, rsp = 0xfffffffe4005d9c0, rbp = 0xfffffffe4005d9e0 --- kdb_enter panic bufinit brelse bufdone ffs_backgroundwritedone bufdone g_io_schedule_up g_up_procbody fork_exit fork_trampoline ------------------- 16 Jan 2009 Revert geom_vfs.c to version 1.12 and rebuild (still with WITNESS and INVARIANTS) [lots of g_vfs_done write failures for ad2] panic: bundirty: buffer 0xfffffffe933eefd0 still on queue 1 kdb_enter panic bufinit brelse+0x8c2 bufdone+0x5c ffs_backgroundwritedone+0xac bufdone+0x37 g_io_schedule_up+0xec g_up_procbody+0x6f fork_exit fork+trampoline could not dump; failure setting up DMA ---