linux-kernel - Re: [PATCH] mm: slub: fix a deadlock warning in kmem_cache

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [day] [month] [year] [list]

Message-ID: <CAFL455m6BZW-KVNOkJPo3BSeEK39Vq597F-b5XvQJmU3mrPmZA@mail.gmail.com>
Date:   Wed, 1 Jun 2022 16:36:06 +0200
From:   Maurizio Lombardi <mlombard@...hat.com>
To:     Xin Long <lucien.xin@...il.com>
Cc:     linux-kernel@...r.kernel.org, linux-mm@...ck.org,
        Christoph Lameter <cl@...ux.com>,
        Pekka Enberg <penberg@...nel.org>,
        David Rientjes <rientjes@...gle.com>,
        Joonsoo Kim <iamjoonsoo.kim@....com>,
        Andrew Morton <akpm@...ux-foundation.org>,
        Vlastimil Babka <vbabka@...e.cz>,
        Sebastian Andrzej Siewior <bigeasy@...utronix.de>,
        Antoine Tenart <atenart@...nel.org>
Subject: Re: [PATCH] mm: slub: fix a deadlock warning in kmem_cache_destroy

pá 14. 1. 2022 v 15:23 odesílatel Xin Long <lucien.xin@...il.com> napsal:
>
> cpus_read_lock() is introduced into kmem_cache_destroy() by
> commit 5a836bf6b09f ("mm: slub: move flush_cpu_slab() invocations
> __free_slab() invocations out of IRQ context"), and it could cause
> a deadlock.

FYI,

I received a bug report from one of our customers, he complains that
his system (with nvmefc boot from SAN) hangs when rebooting.
He runs a RHEL-9 kernel based on version 5.14.0.

What is interesting is that, according to him, after reverting commit
5a836bf6b09f
("mm: slub: move flush_cpu_slab() invocations __free_slab()
invocations out of IRQ context")
the reboot operation doesn't hang anymore.

The call trace seems to point to a possible problem due to the fact that
nvme_delete_ctrl_work is allocated with the WQ_MEM_RECLAIM bit set.

[  453.012078] ------------[ cut here ]------------
[  453.016744] workqueue: WQ_MEM_RECLAIM
nvme-delete-wq:nvme_delete_ctrl_work [nvme_core] is flushing
!WQ_MEM_RECLAIM events:flush_cpu_slab
[  453.016789] WARNING: CPU: 37 PID: 410 at kernel/workqueue.c:2637
check_flush_dependency+0x10a/0x120
[...]
[  453.262125] Call Trace:
[  453.264582]  __flush_work.isra.0+0xbf/0x220
[  453.268775]  ? __queue_work+0x1dc/0x420
[  453.272623]  flush_all_cpus_locked+0xfb/0x120
[  453.276992]  __kmem_cache_shutdown+0x2b/0x320
[  453.281361]  kmem_cache_destroy+0x49/0x100
[  453.285465]  bioset_exit+0x143/0x190
[  453.289052]  blk_release_queue+0xb9/0x100
[  453.293075]  kobject_cleanup+0x37/0x130
[  453.296922]  nvme_fc_ctrl_free+0xc6/0x150 [nvme_fc]
[  453.302397]  nvme_free_ctrl+0x1ac/0x2b0 [nvme_core]
[  453.307818]  device_release+0x31/0x90
[  453.312005]  kobject_cleanup+0x37/0x130
[  453.316369]  process_one_work+0x1e5/0x3c0
[  453.320895]  worker_thread+0x50/0x3b0
[  453.325074]  ? rescuer_thread+0x370/0x370
[  453.329592]  kthread+0x146/0x170
[  453.333322]  ? set_kthread_struct+0x40/0x40
[  453.338027]  ret_from_fork+0x1f/0x30
[  453.342082] ---[ end trace 8c9cdd85adbbfc4f ]---

Maurizio Lombardi