| lists.openwall.net | lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC | |
|
Open Source and information security mailing list archives
| ||
|
Message-ID: <77b734dd-d940-236c-4cef-2f3a10a735ab@oracle.com>
Date: Mon, 23 Aug 2021 15:44:49 +0800
From: Anand Jain <anand.jain@...cle.com>
To: Desmond Cheong Zhi Xi <desmondcheongzx@...il.com>,
linux-btrfs@...r.kernel.org
Cc: linux-kernel@...r.kernel.org, clm@...com,
skhan@...uxfoundation.org, gregkh@...uxfoundation.org,
linux-kernel-mentees@...ts.linuxfoundation.org,
josef@...icpanda.com, dsterba@...e.com
Subject: Re: [PATCH] btrfs: reset replace target device to allocation state on
close
On 21/08/2021 01:50, Desmond Cheong Zhi Xi wrote:
> This crash was observed with a failed assertion on device close:
>
> BTRFS: Transaction aborted (error -28)
> WARNING: CPU: 1 PID: 3902 at fs/btrfs/extent-tree.c:2150 btrfs_run_delayed_refs+0x1d2/0x1e0 [btrfs]
> Modules linked in: btrfs blake2b_generic libcrc32c crc32c_intel xor zstd_decompress zstd_compress xxhash lzo_compress lzo_decompress raid6_pq loop
> CPU: 1 PID: 3902 Comm: kworker/u8:4 Not tainted 5.14.0-rc5-default+ #1532
> Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.12.0-59-gc9ba527-rebuilt.opensuse.org 04/01/2014
> Workqueue: events_unbound btrfs_async_reclaim_metadata_space [btrfs]
> RIP: 0010:btrfs_run_delayed_refs+0x1d2/0x1e0 [btrfs]
> RSP: 0018:ffffb7a5452d7d80 EFLAGS: 00010282
> RAX: 0000000000000000 RBX: 0000000000000003 RCX: 0000000000000000
> RDX: 0000000000000001 RSI: ffffffffabee13c4 RDI: 00000000ffffffff
> RBP: ffff97834176a378 R08: 0000000000000001 R09: 0000000000000001
> R10: 0000000000000000 R11: 0000000000000001 R12: ffff97835195d388
> R13: 0000000005b08000 R14: ffff978385484000 R15: 000000000000016c
> FS: 0000000000000000(0000) GS:ffff9783bd800000(0000) knlGS:0000000000000000
> CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> CR2: 000056190d003fe8 CR3: 000000002a81e005 CR4: 0000000000170ea0
> Call Trace:
> flush_space+0x197/0x2f0 [btrfs]
> btrfs_async_reclaim_metadata_space+0x139/0x300 [btrfs]
> process_one_work+0x262/0x5e0
> worker_thread+0x4c/0x320
> ? process_one_work+0x5e0/0x5e0
> kthread+0x144/0x170
> ? set_kthread_struct+0x40/0x40
> ret_from_fork+0x1f/0x30
> irq event stamp: 19334989
> hardirqs last enabled at (19334997): [<ffffffffab0e0c87>] console_unlock+0x2b7/0x400
> hardirqs last disabled at (19335006): [<ffffffffab0e0d0d>] console_unlock+0x33d/0x400
> softirqs last enabled at (19334900): [<ffffffffaba0030d>] __do_softirq+0x30d/0x574
> softirqs last disabled at (19334893): [<ffffffffab0721ec>] irq_exit_rcu+0x12c/0x140
> ---[ end trace 45939e308e0dd3c7 ]---
> BTRFS: error (device vdd) in btrfs_run_delayed_refs:2150: errno=-28 No space left
> BTRFS info (device vdd): forced readonly
> BTRFS warning (device vdd): failed setting block group ro: -30
> BTRFS info (device vdd): suspending dev_replace for unmount
> assertion failed: !test_bit(BTRFS_DEV_STATE_REPLACE_TGT, &device->dev_state), in fs/btrfs/volumes.c:1150
> ------------[ cut here ]------------
> kernel BUG at fs/btrfs/ctree.h:3431!
> invalid opcode: 0000 [#1] PREEMPT SMP
> CPU: 1 PID: 3982 Comm: umount Tainted: G W 5.14.0-rc5-default+ #1532
> Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.12.0-59-gc9ba527-rebuilt.opensuse.org 04/01/2014
> RIP: 0010:assertfail.constprop.0+0x18/0x1a [btrfs]
> RSP: 0018:ffffb7a5454c7db8 EFLAGS: 00010246
> RAX: 0000000000000068 RBX: ffff978364b91c00 RCX: 0000000000000000
> RDX: 0000000000000000 RSI: ffffffffabee13c4 RDI: 00000000ffffffff
> RBP: ffff9783523a4c00 R08: 0000000000000001 R09: 0000000000000001
> R10: 0000000000000000 R11: 0000000000000001 R12: ffff9783523a4d18
> R13: 0000000000000000 R14: 0000000000000004 R15: 0000000000000003
> FS: 00007f61c8f42800(0000) GS:ffff9783bd800000(0000) knlGS:0000000000000000
> CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> CR2: 000056190cffa810 CR3: 0000000030b96002 CR4: 0000000000170ea0
> Call Trace:
> btrfs_close_one_device.cold+0x11/0x55 [btrfs]
> close_fs_devices+0x44/0xb0 [btrfs]
> btrfs_close_devices+0x48/0x160 [btrfs]
> generic_shutdown_super+0x69/0x100
> kill_anon_super+0x14/0x30
> btrfs_kill_super+0x12/0x20 [btrfs]
> deactivate_locked_super+0x2c/0xa0
> cleanup_mnt+0x144/0x1b0
> task_work_run+0x59/0xa0
> exit_to_user_mode_loop+0xe7/0xf0
> exit_to_user_mode_prepare+0xaf/0xf0
> syscall_exit_to_user_mode+0x19/0x50
> do_syscall_64+0x4a/0x90
> entry_SYSCALL_64_after_hwframe+0x44/0xae
>
> This happens when close_ctree is called while a dev_replace hasn't
> completed. In close_ctree, we suspend the dev_replace, but keep the
> replace target around so that we can resume the dev_replace procedure
> when we mount the root again. This is the call trace:
>
> close_ctree():
> btrfs_dev_replace_suspend_for_unmount();
> btrfs_close_devices():
> btrfs_close_fs_devices():
> btrfs_close_one_device():
> ASSERT(!test_bit(BTRFS_DEV_STATE_REPLACE_TGT,
> &device->dev_state));
>
> However, since the replace target sticks around, there is a device
> with BTRFS_DEV_STATE_REPLACE_TGT set on close, and we fail the
> assertion in btrfs_close_one_device.
>
> To fix this, if we come across the replace target device when
> closing, we should properly reset it back to allocation state. This
> fix also ensures that if a non-target device has a corrupted state and
> has the BTRFS_DEV_STATE_REPLACE_TGT bit set, the assertion will still
> catch the error.
>
> Reported-by: David Sterba <dsterba@...e.com>
> Signed-off-by: Desmond Cheong Zhi Xi <desmondcheongzx@...il.com>
> ---
> fs/btrfs/volumes.c | 3 +++
> 1 file changed, 3 insertions(+)
>
> diff --git a/fs/btrfs/volumes.c b/fs/btrfs/volumes.c
> index 70f94b75f25a..a5afebb78ecf 100644
> --- a/fs/btrfs/volumes.c
> +++ b/fs/btrfs/volumes.c
> @@ -1130,6 +1130,9 @@ static void btrfs_close_one_device(struct btrfs_device *device)
> fs_devices->rw_devices--;
> }
>
> + if (device->devid == BTRFS_DEV_REPLACE_DEVID)
> + clear_bit(BTRFS_DEV_STATE_REPLACE_TGT, &device->dev_state);
> +
This bug is reminiscent of the commit 321f69f86a0f (btrfs: reset the
device back to allocation state when removing).
Before this commit, we freed the btrfs_device at every close. And alloc
a new btrfs_device, so all the dev_state resets back to 0.
Moving on, we still have other dev_state which aren't reset back to 0 at
the time of close. But it can be a separate cleanup patch when needed.
For now, looks good for me.
Reviewed-by: Anand Jain <anand.jain@...cle.com>
Thanks, Anand
> if (test_bit(BTRFS_DEV_STATE_MISSING, &device->dev_state))
> fs_devices->missing_devices--;
>
>
Powered by blists - more mailing lists