| lists.openwall.net | lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC | |
|
Open Source and information security mailing list archives
| ||
|
Message-Id: <20200210204651.21674-4-longman@redhat.com>
Date: Mon, 10 Feb 2020 15:46:51 -0500
From: Waiman Long <longman@...hat.com>
To: Peter Zijlstra <peterz@...radead.org>,
Ingo Molnar <mingo@...hat.com>, Will Deacon <will@...nel.org>,
Christoph Lameter <cl@...ux.com>,
Pekka Enberg <penberg@...nel.org>,
David Rientjes <rientjes@...gle.com>,
Joonsoo Kim <iamjoonsoo.kim@....com>,
Andrew Morton <akpm@...ux-foundation.org>
Cc: linux-kernel@...r.kernel.org, linux-mm@...ck.org,
Waiman Long <longman@...hat.com>
Subject: [PATCH 3/3] mm/slub: Fix potential deadlock problem in slab_attr_store()
The following lockdep splat was seen:
[ 176.241923] ======================================================
[ 176.241924] WARNING: possible circular locking dependency detected
[ 176.241926] 4.18.0-172.rt13.29.el8.x86_64+debug #1 Not tainted
[ 176.241927] ------------------------------------------------------
[ 176.241929] slub_cpu_partia/5371 is trying to acquire lock:
[ 176.241930] ffffffffa0b83718 (slab_mutex){+.+.}, at: slab_attr_store+0x6b/0xe0
[ 176.241941] but task is already holding lock:
[ 176.241942] ffff88bb6d8b83c8 (kn->count#103){++++}, at: kernfs_fop_write+0x1cc/0x400
[ 176.241947] which lock already depends on the new lock.
[ 176.241949] the existing dependency chain (in reverse order) is:
[ 176.241949] -> #1 (kn->count#103){++++}:
[ 176.241955] __kernfs_remove+0x616/0x800
[ 176.241957] kernfs_remove_by_name_ns+0x3e/0x80
[ 176.241959] sysfs_slab_add+0x1c6/0x330
[ 176.241961] __kmem_cache_create+0x15f/0x1b0
[ 176.241964] create_cache+0xe1/0x220
[ 176.241966] kmem_cache_create_usercopy+0x1a3/0x260
[ 176.241967] kmem_cache_create+0x12/0x20
[ 176.242076] mlx5_init_fs+0x18d/0x1a00 [mlx5_core]
[ 176.242100] mlx5_load_one+0x3b4/0x1730 [mlx5_core]
[ 176.242124] init_one+0x901/0x11b0 [mlx5_core]
[ 176.242127] local_pci_probe+0xd4/0x180
[ 176.242131] work_for_cpu_fn+0x51/0xa0
[ 176.242133] process_one_work+0x91a/0x1ac0
[ 176.242134] worker_thread+0x536/0xb40
[ 176.242136] kthread+0x30c/0x3d0
[ 176.242140] ret_from_fork+0x27/0x50
[ 176.242140] -> #0 (slab_mutex){+.+.}:
[ 176.242145] __lock_acquire+0x22cb/0x48c0
[ 176.242146] lock_acquire+0x134/0x4c0
[ 176.242148] _mutex_lock+0x28/0x40
[ 176.242150] slab_attr_store+0x6b/0xe0
[ 176.242151] kernfs_fop_write+0x251/0x400
[ 176.242154] vfs_write+0x157/0x460
[ 176.242155] ksys_write+0xb8/0x170
[ 176.242158] do_syscall_64+0x13c/0x710
[ 176.242160] entry_SYSCALL_64_after_hwframe+0x6a/0xdf
[ 176.242161]
other info that might help us debug this:
[ 176.242161] Possible unsafe locking scenario:
[ 176.242162] CPU0 CPU1
[ 176.242163] ---- ----
[ 176.242163] lock(kn->count#103);
[ 176.242165] lock(slab_mutex);
[ 176.242166] lock(kn->count#103);
[ 176.242167] lock(slab_mutex);
[ 176.242169]
*** DEADLOCK ***
[ 176.242170] 3 locks held by slub_cpu_partia/5371:
[ 176.242170] #0: ffff888705e3a800 (sb_writers#4){.+.+}, at: vfs_write+0x31c/0x460
[ 176.242174] #1: ffff889aeec4d658 (&of->mutex){+.+.}, at: kernfs_fop_write+0x1a9/0x400
[ 176.242177] #2: ffff88bb6d8b83c8 (kn->count#103){++++}, at: kernfs_fop_write+0x1cc/0x400
[ 176.242180]
stack backtrace:
[ 176.242183] CPU: 36 PID: 5371 Comm: slub_cpu_partia Not tainted 4.18.0-172.rt13.29.el8.x86_64+debug #1
[ 176.242184] Hardware name: AMD Corporation DAYTONA_X/DAYTONA_X, BIOS RDY1005C 11/22/2019
[ 176.242185] Call Trace:
[ 176.242190] dump_stack+0x9a/0xf0
[ 176.242193] check_noncircular+0x317/0x3c0
[ 176.242195] ? print_circular_bug+0x1e0/0x1e0
[ 176.242199] ? native_sched_clock+0x32/0x1e0
[ 176.242202] ? sched_clock+0x5/0x10
[ 176.242205] ? sched_clock_cpu+0x238/0x340
[ 176.242208] __lock_acquire+0x22cb/0x48c0
[ 176.242213] ? trace_hardirqs_on+0x10/0x10
[ 176.242215] ? trace_hardirqs_on+0x10/0x10
[ 176.242218] lock_acquire+0x134/0x4c0
[ 176.242220] ? slab_attr_store+0x6b/0xe0
[ 176.242223] _mutex_lock+0x28/0x40
[ 176.242225] ? slab_attr_store+0x6b/0xe0
[ 176.242227] slab_attr_store+0x6b/0xe0
[ 176.242229] ? sysfs_file_ops+0x160/0x160
[ 176.242230] kernfs_fop_write+0x251/0x400
[ 176.242232] ? __sb_start_write+0x26a/0x3f0
[ 176.242234] vfs_write+0x157/0x460
[ 176.242237] ksys_write+0xb8/0x170
[ 176.242239] ? __ia32_sys_read+0xb0/0xb0
[ 176.242242] ? do_syscall_64+0xb9/0x710
[ 176.242245] do_syscall_64+0x13c/0x710
[ 176.242247] entry_SYSCALL_64_after_hwframe+0x6a/0xdf
In order to fix this circular lock dependency problem, we need to do a
mutex_trylock(&slab_mutex) in slab_attr_store() for CPU0 above. A simple
trylock, however, is easy to fail causing user dissatisfaction. So the
new mutex_timed_lock() function is now used to do a trylock with a
100ms timeout.
Signed-off-by: Waiman Long <longman@...hat.com>
---
mm/slub.c | 7 ++++++-
1 file changed, 6 insertions(+), 1 deletion(-)
diff --git a/mm/slub.c b/mm/slub.c
index 17dc00e33115..495bec9b66ab 100644
--- a/mm/slub.c
+++ b/mm/slub.c
@@ -5536,7 +5536,12 @@ static ssize_t slab_attr_store(struct kobject *kobj,
if (slab_state >= FULL && err >= 0 && is_root_cache(s)) {
struct kmem_cache *c;
- mutex_lock(&slab_mutex);
+ /*
+ * Timeout after 100ms
+ */
+ if (mutex_timed_lock(&slab_mutex, 100) < 0)
+ return -EBUSY;
+
if (s->max_attr_size < len)
s->max_attr_size = len;
--
2.18.1
Powered by blists - more mailing lists