Sep 29 18:11:26 cooker kernel: [ 297.924302] end_request: I/O error, dev sda, sector 3040 Sep 29 18:11:26 cooker kernel: [ 297.924359] device-mapper: multipath: Failing path 8:0. Sep 29 18:11:26 cooker kernel: [ 297.924608] Sep 29 18:11:26 cooker kernel: [ 297.924611] ================================= Sep 29 18:11:26 cooker kernel: [ 297.924619] [ INFO: inconsistent lock state ] Sep 29 18:11:26 cooker kernel: [ 297.924628] 2.6.27-rc7-1avb #11 Sep 29 18:11:26 cooker kernel: [ 297.924635] --------------------------------- Sep 29 18:11:26 cooker kernel: [ 297.924643] inconsistent {softirq-on-W} -> {in-softirq-W} usage. Sep 29 18:11:26 cooker kernel: [ 297.924653] ksoftirqd/0/3 [HC0[0]:SC1[1]:HE0:SE0] takes: Sep 29 18:11:26 cooker kernel: [ 297.924663] (_minor_lock){-+..}, at: [] _atomic_dec_and_lock+0x18/0x40 Sep 29 18:11:26 cooker kernel: [ 297.924711] {softirq-on-W} state was registered at: Sep 29 18:11:26 cooker kernel: [ 297.924721] [] __lock_acquire+0x51d/0x1640 Sep 29 18:11:26 cooker kernel: [ 297.924761] [] lock_acquire+0x5c/0x80 Sep 29 18:11:26 cooker kernel: [ 297.924774] [] _spin_lock+0x3a/0x70 Sep 29 18:11:26 cooker kernel: [ 297.924803] [] dm_create+0x3d9/0x480 [dm_mod] Sep 29 18:11:26 cooker kernel: [ 297.924895] [] dev_create+0x41/0x280 [dm_mod] Sep 29 18:11:26 cooker kernel: [ 297.924925] [] dm_ctl_ioctl+0x1cb/0x240 [dm_mod] Sep 29 18:11:26 cooker kernel: [ 297.924950] [] vfs_ioctl+0x25/0x80 Sep 29 18:11:26 cooker kernel: [ 297.924966] [] do_vfs_ioctl+0x23a/0x2b0 Sep 29 18:11:26 cooker kernel: [ 297.924977] [] sys_ioctl+0x3e/0x70 Sep 29 18:11:26 cooker kernel: [ 297.924991] [] sysenter_do_call+0x12/0x31 Sep 29 18:11:26 cooker kernel: [ 297.925010] [] 0xffffffff Sep 29 18:11:26 cooker kernel: [ 297.925024] irq event stamp: 117181 Sep 29 18:11:26 cooker kernel: [ 297.925031] hardirqs last enabled at (117180): [] trace_hardirqs_on+0xb/0x10 Sep 29 18:11:26 cooker kernel: [ 297.925048] hardirqs last disabled at (117181): [] trace_hardirqs_off+0xb/0x10 Sep 29 18:11:26 cooker kernel: [ 297.925064] softirqs last enabled at (117160): [] __do_softirq+0xa3/0xc0 Sep 29 18:11:26 cooker kernel: [ 297.925102] softirqs last disabled at (117165): [] do_softirq+0x96/0xd0 Sep 29 18:11:26 cooker kernel: [ 297.925125] Sep 29 18:11:26 cooker kernel: [ 297.925127] other info that might help us debug this: Sep 29 18:11:26 cooker kernel: [ 297.925137] 1 lock held by ksoftirqd/0/3: Sep 29 18:11:26 cooker kernel: [ 297.925144] #0: (&m->lock){.+..}, at: [] fail_path+0x26/0xc0 [dm_multipath] Sep 29 18:11:26 cooker kernel: [ 297.925186] Sep 29 18:11:26 cooker kernel: [ 297.925189] stack backtrace: Sep 29 18:11:26 cooker kernel: [ 297.925206] Pid: 3, comm: ksoftirqd/0 Not tainted 2.6.27-rc7-1avb #11 Sep 29 18:11:26 cooker kernel: [ 297.925222] [] ? printk+0x1d/0x23 Sep 29 18:11:26 cooker kernel: [ 297.925240] [] print_usage_bug+0x16c/0x1b0 Sep 29 18:11:26 cooker kernel: [ 297.925254] [] mark_lock+0x667/0xa20 Sep 29 18:11:26 cooker kernel: [ 297.925267] [] __lock_acquire+0x4d8/0x1640 Sep 29 18:11:26 cooker kernel: [ 297.925279] [] ? __lock_acquire+0x10e5/0x1640 Sep 29 18:11:26 cooker kernel: [ 297.925295] [] ? dm_uevent_add+0x26/0x60 [dm_mod] Sep 29 18:11:26 cooker kernel: [ 297.925325] [] ? trace_hardirqs_off+0xb/0x10 Sep 29 18:11:26 cooker kernel: [ 297.925342] [] ? dm_uevent_add+0x26/0x60 [dm_mod] Sep 29 18:11:26 cooker kernel: [ 297.925369] [] lock_acquire+0x5c/0x80 Sep 29 18:11:26 cooker kernel: [ 297.925383] [] ? _atomic_dec_and_lock+0x18/0x40 Sep 29 18:11:26 cooker kernel: [ 297.925401] [] _spin_lock+0x3a/0x70 Sep 29 18:11:26 cooker kernel: [ 297.925413] [] ? _atomic_dec_and_lock+0x18/0x40 Sep 29 18:11:26 cooker kernel: [ 297.925430] [] _atomic_dec_and_lock+0x18/0x40 Sep 29 18:11:26 cooker kernel: [ 297.925443] [] dm_put+0x2a/0x190 [dm_mod] Sep 29 18:11:26 cooker kernel: [ 297.925469] [] dm_path_uevent+0x138/0x200 [dm_mod] Sep 29 18:11:26 cooker kernel: [ 297.925504] [] fail_path+0x96/0xc0 [dm_multipath] Sep 29 18:11:26 cooker kernel: [ 297.925521] [] multipath_end_io+0xfd/0x1c0 [dm_multipath] Sep 29 18:11:26 cooker kernel: [ 297.925538] [] clone_endio+0x52/0xc0 [dm_mod] Sep 29 18:11:26 cooker kernel: [ 297.925564] [] ? multipath_end_io+0x0/0x1c0 [dm_multipath] Sep 29 18:11:26 cooker kernel: [ 297.925583] [] ? clone_endio+0x0/0xc0 [dm_mod] Sep 29 18:11:26 cooker kernel: [ 297.925609] [] bio_endio+0x21/0x40 Sep 29 18:11:26 cooker kernel: [ 297.925627] [] req_bio_endio+0x79/0xd0 Sep 29 18:11:26 cooker kernel: [ 297.925646] [] __end_that_request_first+0x7a/0x290 Sep 29 18:11:26 cooker kernel: [ 297.925660] [] blk_end_io+0x87/0xc0 Sep 29 18:11:26 cooker kernel: [ 297.925672] [] blk_end_request+0x1f/0x30 Sep 29 18:11:26 cooker kernel: [ 297.925685] [] scsi_end_request+0x2d/0x90 [scsi_mod] Sep 29 18:11:26 cooker kernel: [ 297.925855] [] scsi_io_completion+0x182/0x490 [scsi_mod] Sep 29 18:11:26 cooker kernel: [ 297.925894] [] ? trace_hardirqs_on_caller+0x75/0x140 Sep 29 18:11:26 cooker kernel: [ 297.925913] [] scsi_finish_command+0x83/0xe0 [scsi_mod] Sep 29 18:11:26 cooker kernel: [ 297.925951] [] ? trace_hardirqs_on+0xb/0x10 Sep 29 18:11:26 cooker kernel: [ 297.925967] [] scsi_softirq_done+0x103/0x120 [scsi_mod] Sep 29 18:11:26 cooker kernel: [ 297.926004] [] ? trace_hardirqs_on+0xb/0x10 Sep 29 18:11:26 cooker kernel: [ 297.926020] [] ? trace_hardirqs_on_caller+0x75/0x140 Sep 29 18:11:26 cooker kernel: [ 297.926036] [] blk_done_softirq+0x6d/0x80 Sep 29 18:11:26 cooker kernel: [ 297.926050] [] __do_softirq+0x62/0xc0 Sep 29 18:11:26 cooker kernel: [ 297.926063] [] ? __do_softirq+0x0/0xc0 Sep 29 18:11:26 cooker kernel: [ 297.926077] [] do_softirq+0x96/0xd0 Sep 29 18:11:26 cooker kernel: [ 297.926092] [] ? ksoftirqd+0x0/0xc0 Sep 29 18:11:26 cooker kernel: [ 297.926107] [] ksoftirqd+0x45/0xc0 Sep 29 18:11:26 cooker kernel: [ 297.926119] [] kthread+0x3f/0x80 Sep 29 18:11:26 cooker kernel: [ 297.926141] [] ? kthread+0x0/0x80 Sep 29 18:11:26 cooker kernel: [ 297.926154] [] kernel_thread_helper+0x7/0x10 Sep 29 18:11:26 cooker kernel: [ 297.926168] ======================= Sep 29 18:11:26 cooker kernel: [ 297.942316] sd 0:0:0:0: [sda] Result: hostbyte=0x07 driverbyte=0x00 Sep 29 18:11:26 cooker kernel: [ 297.942347] end_request: I/O error, dev sda, sector 3280 Sep 29 18:11:26 cooker kernel: [ 297.966225] sd 0:0:0:0: [sda] Result: hostbyte=0x07 driverbyte=0x00 Sep 29 18:11:26 cooker kernel: [ 297.966261] end_request: I/O error, dev sda, sector 3296 Sep 29 18:11:26 cooker kernel: [ 297.967662] sd 0:0:0:0: [sda] READ CAPACITY failed Sep 29 18:11:26 cooker kernel: [ 297.967677] sd 0:0:0:0: [sda] Result: hostbyte=0x07 driverbyte=0x00 Sep 29 18:11:26 cooker kernel: [ 297.967690] sd 0:0:0:0: [sda] Sense not available. Sep 29 18:11:26 cooker kernel: [ 297.968125] sd 0:0:0:0: [sda] Write Protect is off Sep 29 18:11:26 cooker kernel: [ 297.968147] sd 0:0:0:0: [sda] Assuming drive cache: write through Sep 29 18:11:26 cooker kernel: [ 297.970045] sd 0:0:0:0: [sda] READ CAPACITY failed Sep 29 18:11:26 cooker kernel: [ 297.970056] sd 0:0:0:0: [sda] Result: hostbyte=0x07 driverbyte=0x00 Sep 29 18:11:26 cooker kernel: [ 297.970069] sd 0:0:0:0: [sda] Sense not available. Sep 29 18:11:26 cooker kernel: [ 297.970526] sd 0:0:0:0: [sda] Write Protect is off Sep 29 18:11:26 cooker kernel: [ 297.970544] sd 0:0:0:0: [sda] Assuming drive cache: write through Sep 29 18:11:26 cooker kernel: [ 298.044247] usb 1-1: USB disconnect, address 2 Sep 29 18:15:55 cooker multipathd: 12345678: Entering recovery mode: max_retries=5 Sep 29 18:15:55 cooker multipathd: 12345678: map in use Sep 29 18:15:55 cooker multipathd: 12345678: unable to flush devmap Sep 29 18:15:55 cooker multipathd: 12345678: Entering recovery mode: max_retries=5 Sep 29 18:15:55 cooker multipathd: 12345678: Entering recovery mode: max_retries=5 Sep 29 18:15:55 cooker multipathd: 12345678: event checker started Sep 29 18:15:55 cooker multipathd: path checkers start up Sep 29 18:15:57 cooker multipathd: process is already running Sep 29 18:15:57 cooker multipathd: process is already running Sep 29 18:16:19 cooker multipathd: 12345678: Disable queueing