linux-kernel - Re: [RFC PATCH] mm: kasan: suppress soft lockup in slub when !CONFIG

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <CACT4Y+aB088z8zBuQC8Ff6Sf-2_QHVNRjfVpVjy7Xu8+G5BriQ@mail.gmail.com>
Date:   Fri, 8 Dec 2017 09:26:37 +0100
From:   Dmitry Vyukov <dvyukov@...gle.com>
To:     Matthew Wilcox <willy@...radead.org>
Cc:     Yang Shi <yang.s@...baba-inc.com>,
        Andrey Ryabinin <aryabinin@...tuozzo.com>,
        Alexander Potapenko <glider@...gle.com>,
        Andrew Morton <akpm@...ux-foundation.org>,
        Linux-MM <linux-mm@...ck.org>,
        kasan-dev <kasan-dev@...glegroups.com>,
        LKML <linux-kernel@...r.kernel.org>
Subject: Re: [RFC PATCH] mm: kasan: suppress soft lockup in slub when !CONFIG_PREEMPT

On Fri, Dec 8, 2017 at 12:40 AM, Matthew Wilcox <willy@...radead.org> wrote:
> On Fri, Dec 08, 2017 at 07:30:07AM +0800, Yang Shi wrote:
>> When running stress test with KASAN enabled, the below softlockup may
>> happen occasionally:
>>
>> NMI watchdog: BUG: soft lockup - CPU#7 stuck for 22s!
>> hardirqs last  enabled at (0): [<          (null)>]      (null)
>> hardirqs last disabled at (0): [] copy_process.part.30+0x5c6/0x1f50
>> softirqs last  enabled at (0): [] copy_process.part.30+0x5c6/0x1f50
>> softirqs last disabled at (0): [<          (null)>]      (null)
>
>> Call Trace:
>>  [] __slab_free+0x19c/0x270
>>  [] ___cache_free+0xa6/0xb0
>>  [] qlist_free_all+0x47/0x80
>>  [] quarantine_reduce+0x159/0x190
>>  [] kasan_kmalloc+0xaf/0xc0
>>  [] kasan_slab_alloc+0x12/0x20
>>  [] kmem_cache_alloc+0xfa/0x360
>>  [] ? getname_flags+0x4f/0x1f0
>>  [] getname_flags+0x4f/0x1f0
>>  [] getname+0x12/0x20
>>  [] do_sys_open+0xf9/0x210
>>  [] SyS_open+0x1e/0x20
>>  [] entry_SYSCALL_64_fastpath+0x1f/0xc2
>
> This feels like papering over a problem.  KASAN only calls
> quarantine_reduce() when it's allowed to block.  Presumably it has
> millions of entries on the free list at this point.  I think the right
> thing to do is for qlist_free_all() to call cond_resched() after freeing
> every N items.


Agree. Adding touch_softlockup_watchdog() to a random low-level
function looks like a wrong thing to do.
quarantine_reduce() already has this logic. Look at
QUARANTINE_BATCHES. It's meant to do exactly this -- limit amount of
work in quarantine_reduce() and in quarantine_remove_cache() to
reasonably-sized batches. We could simply increase number of batches
to make them smaller. But it would be good to understand what exactly
happens in this case. Batches should on a par of ~~1MB. Why freeing
1MB worth of objects (smallest of which is 32b) takes 22 seconds?



>> The code is run in irq disabled or preempt disabled context, so
>> cond_resched() can't be used in this case. Touch softlockup watchdog when
>> KASAN is enabled to suppress the warning.
>>
>> Signed-off-by: Yang Shi <yang.s@...baba-inc.com>
>> ---
>>  mm/slub.c | 5 +++++
>>  1 file changed, 5 insertions(+)
>>
>> diff --git a/mm/slub.c b/mm/slub.c
>> index cfd56e5..4ae435e 100644
>> --- a/mm/slub.c
>> +++ b/mm/slub.c
>> @@ -35,6 +35,7 @@
>>  #include <linux/prefetch.h>
>>  #include <linux/memcontrol.h>
>>  #include <linux/random.h>
>> +#include <linux/nmi.h>
>>
>>  #include <trace/events/kmem.h>
>>
>> @@ -2266,6 +2267,10 @@ static void put_cpu_partial(struct kmem_cache *s, struct page *page, int drain)
>>               page->pobjects = pobjects;
>>               page->next = oldpage;
>>
>> +#ifdef CONFIG_KASAN
>> +             touch_softlockup_watchdog();
>> +#endif
>> +
>>       } while (this_cpu_cmpxchg(s->cpu_slab->partial, oldpage, page)
>>                                                               != oldpage);
>>       if (unlikely(!s->cpu_partial)) {
>> --
>> 1.8.3.1
>>
>> --
>> To unsubscribe, send a message with 'unsubscribe linux-mm' in
>> the body to majordomo@...ck.org.  For more info on Linux MM,
>> see: http://www.linux-mm.org/ .
>> Don't email: <a href=mailto:"dont@...ck.org"> email@...ck.org </a>