Date:	Mon, 31 Mar 2014 14:39:25 +0100
From:	Sitsofe Wheeler <sitsofe@...oo.com>
To:	Kent Overstreet <kmo@...erainc.com>,
	Nicholas Swenson <nks@...erainc.com>
Cc:	Daniel Smedegaard Buus <danielbuus@...il.com>,
	linux-bcache@...r.kernel.org, linux-kernel@...r.kernel.org
Subject: Re: Correct way to remove a cache device?

Any ideas about this oops Kent? I've seen similar problems too...

On Mon, Mar 31, 2014 at 01:47:21PM +0200, Daniel Smedegaard Buus wrote:
> 
> Still having issues with bcache on my AWS EC2 adventure...
> 
> I'm trying to figure out what the correct way of taking down a bcache
> cache device is.
> 
> If I echo 1 to /sys/block/BACKING_DEVICE/bcache/detach, and then to
> /sys/fs/bcache/*/unregister, the system will hang. The detach part
> goes well, but immediately after unregistering, it will crash.
> 
> I only have SSH access to this instance, and get no output from the
> shell, but if I do this in a startup script, what I see from the
> system log in the AWS console is the below output.
> 
> Any ideas?
> 
> Output:
> 
>     [   20.756111] BUG: unable to handle kernel NULL pointer
> dereference at 0000000000000a00
>     [   20.756125] IP: [<ffffffffa0066280>]
> journal_write_unlocked+0x130/0x540 [bcache]
>     [   20.756137] PGD 0
>     [   20.756139] Oops: 0000 [#1] SMP
>     [   20.756143] Modules linked in: dm_crypt isofs raid10 raid456
> async_memcpy async_raid6_recov async_pq async_xor async_tx xor
> raid6_pq raid1 multipath linear bcache raid0 crct10dif_pclmul
> crc32_pclmul ghash_clmulni_intel aesni_intel aes_x86_64 lrw gf128mul
> glue_helper ablk_helper cryptd
>     [   20.756165] CPU: 0 PID: 30 Comm: kworker/0:1 Not tainted
> 3.13.0-19-generic #39-Ubuntu
>     [   20.756173] Workqueue: events journal_write_work [bcache]
>     [   20.756176] task: ffff8800e8eedfc0 ti: ffff8800e8fe4000
> task.ti: ffff8800e8fe4000
>     [   20.756179] RIP: e030:[<ffffffffa0066280>]
> [<ffffffffa0066280>] journal_write_unlocked+0x130/0x540 [bcache]
>     [   20.756187] RSP: e02b:ffff8800e8fe5d90  EFLAGS: 00010202
>     [   20.756189] RAX: 0000000000000000 RBX: 0000000000000001 RCX:
> 0000000000000000
>     [   20.756192] RDX: ffff8800e60c0c48 RSI: ffff8800e60ccad8 RDI:
> ffff8800e60f8040
>     [   20.756194] RBP: ffff8800e8fe5de8 R08: 200398332f400000 R09:
> 5e80000000000000
>     [   20.756197] R10: dffbefcdb6cccbd0 R11: 0000000000000000 R12:
> 0000000000000001
>     [   20.756200] R13: ffff8800e60ccba0 R14: ffff8800e60ccce8 R15:
> ffff8800e60c0000
>     [   20.756206] FS:  00007f65089c7740(0000)
> GS:ffff8800ef600000(0000) knlGS:0000000000000000
>     [   20.756209] CS:  e033 DS: 0000 ES: 0000 CR0: 000000008005003b
>     [   20.756211] CR2: 0000000000000a00 CR3: 00000000e70f6000 CR4:
> 0000000000002660
>     [   20.756214] Stack:
>     [   20.756216]  ffff8800e8fe5db0 ffff8800e60c0000 ffffffff81c15480
> ffffffff81c15480
>     [   20.756221]  ffff8800e8fe5dc8 ffffffff8109cb2d ffff8800e60c0000
> ffff8800e60ccba0
>     [   20.756225]  ffff8800e60ccbd0 0000000000000000 0000000000000000
> ffff8800e8fe5e08
>     [   20.756229] Call Trace:
>     [   20.756237]  [<ffffffff8109cb2d>] ? vtime_common_task_switch+0x3d/0x40
>     [   20.756243]  [<ffffffffa00666e0>] journal_try_write+0x50/0x60 [bcache]
>     [   20.756248]  [<ffffffffa0066712>] journal_write_work+0x22/0x30 [bcache]
>     [   20.756253]  [<ffffffff810824a2>] process_one_work+0x182/0x450
>     [   20.756257]  [<ffffffff81083241>] worker_thread+0x121/0x410
>     [   20.756260]  [<ffffffff81083120>] ? rescuer_thread+0x3e0/0x3e0
>     [   20.756264]  [<ffffffff81089ed2>] kthread+0xd2/0xf0
>     [   20.756267]  [<ffffffff81089e00>] ? kthread_create_on_node+0x190/0x190
>     [   20.756273]  [<ffffffff817219bc>] ret_from_fork+0x7c/0xb0
>     [   20.756276]  [<ffffffff81089e00>] ? kthread_create_on_node+0x190/0x190
>     [   20.756278] Code: 00 00 e8 04 03 30 e1 31 c0 66 41 83 bd 94 38
> ff ff 00 49 8b 8d a0 40 ff ff 49 8d 97 48 0c 00 00 74 3c 66 0f 1f 84
> 00 00 00 00 00 <48> 8b b9 00 0a 00 00 0f b7 89 ce 00 00 00 83 c0 01 49
> 8b 36 48
>     [   20.756310] RIP  [<ffffffffa0066280>]
> journal_write_unlocked+0x130/0x540 [bcache]
>     [   20.756316]  RSP <ffff8800e8fe5d90>
>     [   20.756317] CR2: 0000000000000a00
>     [   20.756320] ---[ end trace 84c8ace3e9ccb27e ]---
>     [   20.756384] BUG: unable to handle kernel paging request at
> ffffffffffffffd8
>     [   20.756390] IP: [<ffffffff8108a570>] kthread_data+0x10/0x20
>     [   20.756396] PGD 1c11067 PUD 1c13067 PMD 0
>     [   20.756401] Oops: 0000 [#2] SMP
>     [   20.756405] Modules linked in: dm_crypt isofs raid10 raid456
> async_memcpy async_raid6_recov async_pq async_xor async_tx xor
> raid6_pq raid1 multipath linear bcache raid0 crct10dif_pclmul
> crc32_pclmul ghash_clmulni_intel aesni_intel aes_x86_64 lrw gf128mul
> glue_helper ablk_helper cryptd
>     [   20.756434] CPU: 0 PID: 30 Comm: kworker/0:1 Tainted: G      D
>     3.13.0-19-generic #39-Ubuntu
>     [   20.756450] task: ffff8800e8eedfc0 ti: ffff8800e8fe4000
> task.ti: ffff8800e8fe4000
>     [   20.756455] RIP: e030:[<ffffffff8108a570>]
> [<ffffffff8108a570>] kthread_data+0x10/0x20
>     [   20.756461] RSP: e02b:ffff8800e8fe59e8  EFLAGS: 00010002
>     [   20.756464] RAX: 0000000000000000 RBX: 0000000000000000 RCX:
> 0000000000000005
>     [   20.756468] RDX: 0000000000000004 RSI: 0000000000000000 RDI:
> ffff8800e8eedfc0
>     [   20.756472] RBP: ffff8800e8fe59e8 R08: 0000000000000000 R09:
> ffff8800ef618580
>     [   20.756476] R10: ffffffff8133443a R11: ffffea0003996900 R12:
> ffff8800ef614440
>     [   20.756481] R13: 0000000000000000 R14: ffff8800e8eedfb0 R15:
> ffff8800e8eedfc0
>     [   20.756487] FS:  00007f65089c7740(0000)
> GS:ffff8800ef600000(0000) knlGS:0000000000000000
>     [   20.756492] CS:  e033 DS: 0000 ES: 0000 CR0: 000000008005003b
>     [   20.756497] CR2: 0000000000000028 CR3: 00000000e70f6000 CR4:
> 0000000000002660
>     [   20.756501] Stack:
>     [   20.756504]  ffff8800e8fe5a00 ffffffff81083951 ffff8800e8eedfc0
> ffff8800e8fe5a60
>     [   20.756511]  ffffffff81715249 ffff8800e8eedfc0 ffff8800e8fe5fd8
> 0000000000014440
>     [   20.756518]  0000000000014440 ffff8800e8eedfc0 ffff8800e8eee5f8
> ffff8800e8eedfb0
>     [   20.756525] Call Trace:
>     [   20.756530]  [<ffffffff81083951>] wq_worker_sleeping+0x11/0x90
>     [   20.756536]  [<ffffffff81715249>] __schedule+0x589/0x7d0
>     [   20.756541]  [<ffffffff817154b9>] schedule+0x29/0x70
>     [   20.756547]  [<ffffffff81068c3f>] do_exit+0x6df/0xa50
>     [   20.756553]  [<ffffffff8171a539>] oops_end+0xa9/0x150
>     [   20.756559]  [<ffffffff81709614>] no_context+0x27e/0x28b
>     [   20.756564]  [<ffffffff81709694>] __bad_area_nosemaphore+0x73/0x1ca
>     [   20.756570]  [<ffffffff817097fe>] bad_area_nosemaphore+0x13/0x15
>     [   20.756576]  [<ffffffff8171cf07>] __do_page_fault+0xa7/0x560
>     [   20.756582]  [<ffffffff81718eb0>] ? _raw_spin_unlock_irqrestore+0x20/0x40
>     [   20.756589]  [<ffffffff810a95f4>] ? __wake_up+0x44/0x50
>     [   20.756595]  [<ffffffff81641479>] ?
> netlink_broadcast_filtered+0x129/0x3b0
>     [   20.756602]  [<ffffffff8135c510>] ? kobj_ns_drop+0x50/0x50
>     [   20.756607]  [<ffffffff8171d3da>] do_page_fault+0x1a/0x70
>     [   20.756611]  [<ffffffff81719848>] page_fault+0x28/0x30
>     [   20.756616]  [<ffffffffa0066280>] ?
> journal_write_unlocked+0x130/0x540 [bcache]
>     [   20.756620]  [<ffffffff8109cb2d>] ? vtime_common_task_switch+0x3d/0x40
>     [   20.756625]  [<ffffffffa00666e0>] journal_try_write+0x50/0x60 [bcache]
>     [   20.756630]  [<ffffffffa0066712>] journal_write_work+0x22/0x30 [bcache]
>     [   20.756634]  [<ffffffff810824a2>] process_one_work+0x182/0x450
>     [   20.756638]  [<ffffffff81083241>] worker_thread+0x121/0x410
>     [   20.756641]  [<ffffffff81083120>] ? rescuer_thread+0x3e0/0x3e0
>     [   20.756644]  [<ffffffff81089ed2>] kthread+0xd2/0xf0
>     [   20.756648]  [<ffffffff81089e00>] ? kthread_create_on_node+0x190/0x190
>     [   20.756651]  [<ffffffff817219bc>] ret_from_fork+0x7c/0xb0
>     [   20.756654]  [<ffffffff81089e00>] ? kthread_create_on_node+0x190/0x190
>     [   20.756657] Code: 00 48 89 e5 5d 48 8b 40 c8 48 c1 e8 02 83 e0
> 01 c3 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 48 8b 87 a8 03 00
> 00 55 48 89 e5 <48> 8b 40 d8 5d c3 66 2e 0f 1f 84 00 00 00 00 00 0f 1f
> 44 00 00
>     [   20.756688] RIP  [<ffffffff8108a570>] kthread_data+0x10/0x20
>     [   20.756691]  RSP <ffff8800e8fe59e8>
>     [   20.756693] CR2: ffffffffffffffd8
>     [   20.756695] ---[ end trace 84c8ace3e9ccb27f ]---
>     [   20.756697] Fixing recursive fault but reboot is needed!

-- 
Sitsofe | http://sucs.org/~sits/
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Powered by blists - more mailing lists