linux-kernel - Re: [PATCH 6.10 000/809] 6.10.3-rc3 review

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <75fc1760-7826-49d1-8da5-99c09ddb2aa4@suse.cz>
Date: Tue, 6 Aug 2024 21:31:55 +0200
From: Vlastimil Babka <vbabka@...e.cz>
To: Guenter Roeck <linux@...ck-us.net>,
 Linus Torvalds <torvalds@...ux-foundation.org>
Cc: linux-kernel@...r.kernel.org, Linux-MM <linux-mm@...ck.org>,
 Thomas Gleixner <tglx@...utronix.de>
Subject: Re: [PATCH 6.10 000/809] 6.10.3-rc3 review

On 8/6/24 20:09, Guenter Roeck wrote:
> On 8/6/24 04:02, Vlastimil Babka wrote:
>> On 8/6/24 04:40, Linus Torvalds wrote:
>>> [ Let's drop random people and bring in Vlastimil ]
>> 
>> tglx was reproducing it so I add him back
>> 
>>> Vlastimil,
>>>   it turns out that the "this patch" is entirely a red herring, and the
>>> problem comes and goes randomly with just some code layout issues. See
>>>
>>>     http://server.roeck-us.net/qemu/parisc64-6.10.3/
>>>
>>> for more detail, particularly you'll see the "log.bad.gz" with the full log.
>> 
>> [    0.000000] BUG kmem_cache_node (Not tainted): objects 21 > max 16
>> [    0.000000] Slab 0x0000000041ed0000 objects=21 used=5 fp=0x00000000434003d0 flags=0x200(workingset|section=0|zone=0)
>> 
>> flags tell us this came from the partial list (workingset), there's no head flag so order-0
>> 
>> since the error was detected it basically throws the slab page away and tries another one
>> 
>> [    0.000000] BUG kmem_cache (Tainted: G    B             ): objects 25 > max 16
>> [    0.000000] Slab 0x0000000041ed0080 objects=25 used=6 fp=0x0000000043402790 flags=0x240(workingset|head|section=0|zone=0)

OK so I missed the first was "kmem_cache_node" and the rest is "kmem_cache"
which explains the 21 vs 25 difference, and the difference in order etc.

>> this was also from the partial list but head flag so at least order-1, two things are weird:
>> - max=16 is same as above even though it should be at least double as
>> slab page's order is larger
>> - objects=25 also isn't at least twice than objects=21
>> 
>> All the following are:
>> [    0.000000] BUG kmem_cache (Tainted: G    B             ): objects 25 > max 16
>> [    0.000000] Slab 0x0000000041ed0300 objects=25 used=1 fp=0x000000004340c150 flags=0x40(head|section=0|zone=0)
>> 
>> we depleted the partial list so it's allocating new slab pages, that are
>> also at least order-1
>> 
>> It looks like maxobj calculation is bogus, would be useful to see what values it
>> calculates from. I'm attaching a diff, but maybe it will also hide the issue...
>> 
> 
> If I am really careful I can catch the problem. Adding a parameter to slab_err()
> makes it disappear. Calling slab_order() multiple times makes it disappear. But I can
> assign the result of slab_order() to a variable and go from there, by changing the
> parameters to slab_err() one at a time. Here is what I get by combining multiple
> test runs.
> 
> objects 21 max 16 order 0 size 192
>   with:
>    Slab 0x0000000041ed0000 objects=21 used=5 fp=0x00000000434003d0 flags=0x200(workingset|section=0|zone=0)
> objects 25 max 16 order 1 size 320 (many times)
>   with:
>    Slab 0x0000000041ed0080 objects=25 used=6 fp=0x0000000043402790 flags=0x240(workingset|head|section=0|zone=0)
>    Slab 0x0000000041ed0300 objects=25 used=1 fp=0x000000004340c150 flags=0x40(head|section=0|zone=0)
>    Slab 0x0000000041ed0380 objects=25 used=1 fp=0x000000004340e150 flags=0x40(head|section=0|zone=0)
>    and so on
> 
> If I add a log message into check_slab() and display every check I get:
> 
> objects 21 max 21 (order 0 size 192)
> objects 25 max 25 (order 1 size 320)
> objects 25 max 25 (order 1 size 320)
> objects 25 max 25 (order 1 size 320)
> objects 21 max 21 (order 0 size 192)
> 
> and so on. I guess that confirms that the maxobj calculation is wrong in the failure case.

Agreed.

> That seems weird, though, given that order_objects() should return a constant based on
> order and size. PAGE_SIZE is 4096, meaning order_objects() should return 21 and 25, never
> 16. That does make me really wonder if there is an emulation problem, though I don't
> really understand why that would not be persistent.

Yeah 21 and 25 are expected for these sizes and orders.

> Any idea what else I could test ?
> 
> Thanks,
> Guenter
>