linux-kernel - Re: kswapd0: page allocation failure: order:0, mode:0x820(GFP_ATOMIC), nodemask=(null),cpuset=/,mems

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives
Hash Suite: Windows password security audit tool. GUI, reports in PDF.
[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]
Message-ID: <45fc081c-ee8d-4774-a597-708d2924f812@redhat.com>
Date: Sat, 1 Jun 2024 17:37:02 +0200
From: David Hildenbrand <david@...hat.com>
To: Yu Zhao <yuzhao@...gle.com>, Erhard Furtner <erhard_f@...lbox.org>
Cc: linux-mm@...ck.org, linux-kernel@...r.kernel.org,
 linuxppc-dev@...ts.ozlabs.org
Subject: Re: kswapd0: page allocation failure: order:0,
 mode:0x820(GFP_ATOMIC), nodemask=(null),cpuset=/,mems_allowed=0 (Kernel
 v6.5.9, 32bit ppc)

On 01.06.24 08:01, Yu Zhao wrote:
> On Wed, May 15, 2024 at 4:06 PM Yu Zhao <yuzhao@...gle.com> wrote:
>>
>> On Wed, May 15, 2024 at 2:45 PM Erhard Furtner <erhard_f@...lbox.org> wrote:
>>>
>>> On Wed, 8 May 2024 20:21:11 +0200
>>> Erhard Furtner <erhard_f@...lbox.org> wrote:
>>>
>>>> Greetings!
>>>>
>>>> Got that on my dual CPU PowerMac G4 DP shortly after boot. This does not happen every time at bootup though:
>>>>
>>>> [...]
>>>> kswapd0: page allocation failure: order:0, mode:0x820(GFP_ATOMIC), nodemask=(null),cpuset=/,mems_allowed=0
>>>> CPU: 1 PID: 40 Comm: kswapd0 Not tainted 6.8.9-gentoo-PMacG4 #1
>>>> Hardware name: PowerMac3,6 7455 0x80010303 PowerMac
>>>
>>> Very similar page allocation failure on the same machine on kernel 6.9.0 too. Seems it can easily be provoked by running a memory stressor, e.g. "stress-ng --vm 2 --vm-bytes 1930M --verify -v":
>>>
>>> [...]
>>> kswapd0: page allocation failure: order:0, mode:0xcc0(GFP_KERNEL), nodemask=(null),cpuset=/,mems_allowed=0
>>> CPU: 0 PID: 41 Comm: kswapd0 Not tainted 6.9.0-gentoo-PMacG4 #1
>>> Hardware name: PowerMac3,6 7455 0x80010303 PowerMac
>>> Call Trace:
>>> [c1c65940] [c07926d4] dump_stack_lvl+0x80/0xac (unreliable)
>>> [c1c65960] [c01b6234] warn_alloc+0x100/0x178
>>> [c1c659c0] [c01b661c] __alloc_pages+0x370/0x8d0
>>> [c1c65a80] [c01c4854] __read_swap_cache_async+0xc0/0x1cc
>>> [c1c65ad0] [c01cb580] zswap_writeback_entry+0x50/0x154
>>> [c1c65be0] [c01cb6f4] shrink_memcg_cb+0x70/0xec
>>> [c1c65c10] [c019518c] __list_lru_walk_one+0xa0/0x154
>>> [c1c65c70] [c01952a4] list_lru_walk_one+0x64/0x7c
>>> [c1c65ca0] [c01cad00] zswap_shrinker_scan+0xac/0xc4
>>> [c1c65cd0] [c018052c] do_shrink_slab+0x18c/0x304
>>> [c1c65d20] [c0180a40] shrink_slab+0x174/0x260
>>> [c1c65da0] [c017cb0c] shrink_one+0xbc/0x134
>>> [c1c65dd0] [c017e3e4] shrink_node+0x238/0x84c
>>> [c1c65e50] [c017ed38] balance_pgdat+0x340/0x650
>>> [c1c65f50] [c017f270] kswapd+0x228/0x25c
>>> [c1c65fc0] [c006bbac] kthread+0xe4/0xe8
>>> [c1c65ff0] [c0015304] start_kernel_thread+0x10/0x14
>>> SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
>>>    cache: skbuff_small_head, object size: 480, buffer size: 544, default order: 1, min order: 0
>>>    node 0: slabs: 15, objs: 225, free: 0
>>> SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
>>>    cache: skbuff_small_head, object size: 480, buffer size: 544, default order: 1, min order: 0
>>>    node 0: slabs: 15, objs: 225, free: 0
>>> SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
>>>    cache: skbuff_small_head, object size: 480, buffer size: 544, default order: 1, min order: 0
>>>    node 0: slabs: 15, objs: 225, free: 0
>>> SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
>>>    cache: skbuff_small_head, object size: 480, buffer size: 544, default order: 1, min order: 0
>>>    node 0: slabs: 15, objs: 225, free: 0
>>> SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
>>>    cache: skbuff_small_head, object size: 480, buffer size: 544, default order: 1, min order: 0
>>>    node 0: slabs: 15, objs: 225, free: 0
>>> SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
>>>    cache: kmalloc-rnd-15-2k, object size: 2048, buffer size: 6144, default order: 3, min order: 1
>>>    kmalloc-rnd-15-2k debugging increased min order, use slab_debug=O to disable.
>>>    node 0: slabs: 33, objs: 165, free: 0
>>> SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
>>>    cache: skbuff_small_head, object size: 480, buffer size: 544, default order: 1, min order: 0
>>>    node 0: slabs: 15, objs: 225, free: 0
>>> SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
>>>    cache: kmalloc-rnd-15-2k, object size: 2048, buffer size: 6144, default order: 3, min order: 1
>>>    kmalloc-rnd-15-2k debugging increased min order, use slab_debug=O to disable.
>>>    node 0: slabs: 33, objs: 165, free: 0
>>> SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
>>>    cache: skbuff_small_head, object size: 480, buffer size: 544, default order: 1, min order: 0
>>>    node 0: slabs: 15, objs: 225, free: 0
>>> SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
>>>    cache: kmalloc-rnd-15-2k, object size: 2048, buffer size: 6144, default order: 3, min order: 1
>>>    kmalloc-rnd-15-2k debugging increased min order, use slab_debug=O to disable.
>>>    node 0: slabs: 33, objs: 165, free: 0
>>> Mem-Info:
>>> active_anon:340071 inactive_anon:139179 isolated_anon:0
>>>   active_file:8297 inactive_file:2506 isolated_file:0
>>>   unevictable:4 dirty:1 writeback:18
>>>   slab_reclaimable:1377 slab_unreclaimable:7426
>>>   mapped:6804 shmem:112 pagetables:946
>>>   sec_pagetables:0 bounce:0
>>>   kernel_misc_reclaimable:0
>>>   free:1141 free_pcp:7 free_cma:0
>>> Node 0 active_anon:1360284kB inactive_anon:556716kB active_file:33188kB inactive_file:10024kB unevictable:16kB isolated(anon):0kB isolated(file):0kB mapped:27216kB dirty:4kB writeback:72kB shmem:448kB writeback_tmp:0kB kernel_stack:1560kB pagetables:3784kB sec_pagetables:0kB all_unreclaimable? no
>>> DMA free:56kB boost:7756kB min:11208kB low:12068kB high:12928kB reserved_highatomic:0KB active_anon:635128kB inactive_anon:58260kB active_file:268kB inactive_file:3000kB unevictable:0kB writepending:324kB present:786432kB managed:746644kB mlocked:0kB bounce:0kB free_pcp:28kB local_pcp:28kB free_cma:0kB
>>> lowmem_reserve[]: 0 0 1280 1280
>>> DMA: 0*4kB 0*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 0kB
>>> 63943 total pagecache pages
>>> 53024 pages in swap cache
>>> Free swap  = 8057248kB
>>> Total swap = 8388604kB
>>> 524288 pages RAM
>>> 327680 pages HighMem/MovableOnly
>>> 9947 pages reserved
>>> warn_alloc: 396 callbacks suppressed
>>> kswapd0: page allocation failure: order:0, mode:0xcc0(GFP_KERNEL), nodemask=(null),cpuset=/,mems_allowed=0
>>> CPU: 1 PID: 41 Comm: kswapd0 Not tainted 6.9.0-gentoo-PMacG4 #1
>>> Hardware name: PowerMac3,6 7455 0x80010303 PowerMac
>>> Call Trace:
>>> [c1c65940] [c07926d4] dump_stack_lvl+0x80/0xac (unreliable)
>>> [c1c65960] [c01b6234] warn_alloc+0x100/0x178
>>> [c1c659c0] [c01b661c] __alloc_pages+0x370/0x8d0
>>> [c1c65a80] [c01c4854] __read_swap_cache_async+0xc0/0x1cc
>>> [c1c65ad0] [c01cb580] zswap_writeback_entry+0x50/0x154
>>> [c1c65be0] [c01cb6f4] shrink_memcg_cb+0x70/0xec
>>> [c1c65c10] [c019518c] __list_lru_walk_one+0xa0/0x154
>>> [c1c65c70] [c01952a4] list_lru_walk_one+0x64/0x7c
>>> [c1c65ca0] [c01cad00] zswap_shrinker_scan+0xac/0xc4
>>> [c1c65cd0] [c018052c] do_shrink_slab+0x18c/0x304
>>> [c1c65d20] [c0180a40] shrink_slab+0x174/0x260
>>> [c1c65da0] [c017cb0c] shrink_one+0xbc/0x134
>>> [c1c65dd0] [c017e3e4] shrink_node+0x238/0x84c
>>> [c1c65e50] [c017ed38] balance_pgdat+0x340/0x650
>>> [c1c65f50] [c017f270] kswapd+0x228/0x25c
>>> slab_out_of_memory: 53 callbacks suppressed
>>> SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
>>>    cache: skbuff_small_head, object size: 480, buffer size: 544, default order: 1, min order: 0
>>>    node 0: slabs: 18, objs: 270, free: 0
>>> SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
>>>    cache: skbuff_small_head, object size: 480, buffer size: 544, default order: 1, min order: 0
>>>    node 0: slabs: 18, objs: 270, free: 0
>>> SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
>>>    cache: skbuff_small_head, object size: 480, buffer size: 544, default order: 1, min order: 0
>>>    node 0: slabs: 18, objs: 270, free: 0
>>> SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
>>>    cache: skbuff_small_head, object size: 480, buffer size: 544, default order: 1, min order: 0
>>>    node 0: slabs: 18, objs: 270, free: 0
>>> SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
>>>    cache: skbuff_small_head, object size: 480, buffer size: 544, default order: 1, min order: 0
>>>    node 0: slabs: 18, objs: 270, free: 0
>>> SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
>>>    cache: kmalloc-rnd-15-2k, object size: 2048, buffer size: 6144, default order: 3, min order: 1
>>>    kmalloc-rnd-15-2k debugging increased min order, use slab_debug=O to disable.
>>>    node 0: slabs: 33, objs: 165, free: 0
>>> SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
>>>    cache: skbuff_small_head, object size: 480, buffer size: 544, default order: 1, min order: 0
>>>    node 0: slabs: 18, objs: 270, free: 0
>>> SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
>>>    cache: kmalloc-rnd-15-2k, object size: 2048, buffer size: 6144, default order: 3, min order: 1
>>>    kmalloc-rnd-15-2k debugging increased min order, use slab_debug=O to disable.
>>>    node 0: slabs: 33, objs: 165, free: 0
>>> SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
>>>    cache: skbuff_small_head, object size: 480, buffer size: 544, default order: 1, min order: 0
>>>    node 0: slabs: 18, objs: 270, free: 0
>>> SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
>>>    cache: kmalloc-rnd-15-2k, object size: 2048, buffer size: 6144, default order: 3, min order: 1
>>>    kmalloc-rnd-15-2k debugging increased min order, use slab_debug=O to disable.
>>>    node 0: slabs: 33, objs: 165, free: 0
>>> [c1c65fc0] [c006bbac] kthread+0xe4/0xe8
>>> [c1c65ff0] [c0015304] start_kernel_thread+0x10/0x14
>>> Mem-Info:
>>> active_anon:351976 inactive_anon:123514 isolated_anon:0
>>>   active_file:4648 inactive_file:2081 isolated_file:0
>>>   unevictable:4 dirty:1 writeback:39
>>>   slab_reclaimable:918 slab_unreclaimable:7222
>>>   mapped:5359 shmem:21 pagetables:940
>>>   sec_pagetables:0 bounce:0
>>>   kernel_misc_reclaimable:0
>>>   free:2563 free_pcp:142 free_cma:0
>>> Node 0 active_anon:1407904kB inactive_anon:494056kB active_file:18592kB inactive_file:8324kB unevictable:16kB isolated(anon):0kB isolated(file):0kB mapped:21436kB dirty:4kB writeback:156kB shmem:84kB writeback_tmp:0kB kernel_stack:1552kB pagetables:3760kB sec_pagetables:0kB all_unreclaimable? no
>>> DMA free:0kB boost:7756kB min:11208kB low:12068kB high:12928kB reserved_highatomic:0KB active_anon:199336kB inactive_anon:491432kB active_file:4612kB inactive_file:5980kB unevictable:0kB writepending:660kB present:786432kB managed:746644kB mlocked:0kB bounce:0kB free_pcp:568kB local_pcp:20kB free_cma:0kB
>>> lowmem_reserve[]: 0 0 1280 1280
>>> DMA: 0*4kB 0*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 0kB
>>> 45961 total pagecache pages
>>> 39207 pages in swap cache
>>> Free swap  = 8093096kB
>>> Total swap = 8388604kB
>>> 524288 pages RAM
>>> 327680 pages HighMem/MovableOnly
>>> 9947 pages reserved
>>> warn_alloc: 343 callbacks suppressed
>>> kswapd0: page allocation failure: order:0, mode:0xcc0(GFP_KERNEL), nodemask=(null),cpuset=/,mems_allowed=0
>>> CPU: 0 PID: 41 Comm: kswapd0 Not tainted 6.9.0-gentoo-PMacG4 #1
>>> Hardware name: PowerMac3,6 7455 0x80010303 PowerMac
>>> Call Trace:
>>> [c1c65940] [c07926d4] dump_stack_lvl+0x80/0xac (unreliable)
>>> [c1c65960] [c01b6234] warn_alloc+0x100/0x178
>>> [c1c659c0] [c01b661c] __alloc_pages+0x370/0x8d0
>>> [c1c65a80] [c01c4854] __read_swap_cache_async+0xc0/0x1cc
>>> [c1c65ad0] [c01cb580] zswap_writeback_entry+0x50/0x154
>>> [c1c65be0] [c01cb6f4] shrink_memcg_cb+0x70/0xec
>>> [c1c65c10] [c019518c] __list_lru_walk_one+0xa0/0x154
>>> slab_out_of_memory: 59 callbacks suppressed
>>> SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
>>>    cache: skbuff_small_head, object size: 480, buffer size: 544, default order: 1, min order: 0
>>>    node 0: slabs: 18, objs: 270, free: 0
>>> SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
>>>    cache: skbuff_small_head, object size: 480, buffer size: 544, default order: 1, min order: 0
>>>    node 0: slabs: 18, objs: 270, free: 0
>>> SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
>>>    cache: skbuff_small_head, object size: 480, buffer size: 544, default order: 1, min order: 0
>>>    node 0: slabs: 18, objs: 270, free: 0
>>> SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
>>>    cache: skbuff_small_head, object size: 480, buffer size: 544, default order: 1, min order: 0
>>>    node 0: slabs: 18, objs: 270, free: 0
>>> SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
>>>    cache: skbuff_small_head, object size: 480, buffer size: 544, default order: 1, min order: 0
>>>    node 0: slabs: 18, objs: 270, free: 0
>>> SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
>>>    cache: kmalloc-rnd-15-2k, object size: 2048, buffer size: 6144, default order: 3, min order: 1
>>>    kmalloc-rnd-15-2k debugging increased min order, use slab_debug=O to disable.
>>>    node 0: slabs: 33, objs: 165, free: 0
>>> SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
>>>    cache: skbuff_small_head, object size: 480, buffer size: 544, default order: 1, min order: 0
>>>    node 0: slabs: 18, objs: 270, free: 0
>>> SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
>>>    cache: kmalloc-rnd-15-2k, object size: 2048, buffer size: 6144, default order: 3, min order: 1
>>>    kmalloc-rnd-15-2k debugging increased min order, use slab_debug=O to disable.
>>>    node 0: slabs: 33, objs: 165, free: 0
>>> SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
>>>    cache: skbuff_small_head, object size: 480, buffer size: 544, default order: 1, min order: 0
>>>    node 0: slabs: 18, objs: 270, free: 0
>>> SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
>>>    cache: kmalloc-rnd-15-2k, object size: 2048, buffer size: 6144, default order: 3, min order: 1
>>>    kmalloc-rnd-15-2k debugging increased min order, use slab_debug=O to disable.
>>>    node 0: slabs: 33, objs: 165, free: 0
>>> [c1c65c70] [c01952a4] list_lru_walk_one+0x64/0x7c
>>> [c1c65ca0] [c01cad00] zswap_shrinker_scan+0xac/0xc4
>>> [c1c65cd0] [c018052c] do_shrink_slab+0x18c/0x304
>>> [c1c65d20] [c0180a40] shrink_slab+0x174/0x260
>>> [c1c65da0] [c017cb0c] shrink_one+0xbc/0x134
>>> [c1c65dd0] [c017e3e4] shrink_node+0x238/0x84c
>>> [c1c65e50] [c017ed38] balance_pgdat+0x340/0x650
>>> [c1c65f50] [c017f270] kswapd+0x228/0x25c
>>> [c1c65fc0] [c006bbac] kthread+0xe4/0xe8
>>> [c1c65ff0] [c0015304] start_kernel_thread+0x10/0x14
>>> Mem-Info:
>>> active_anon:235002 inactive_anon:240975 isolated_anon:0
>>>   active_file:4356 inactive_file:2551 isolated_file:0
>>>   unevictable:4 dirty:7 writeback:19
>>>   slab_reclaimable:1008 slab_unreclaimable:7218
>>>   mapped:5601 shmem:21 pagetables:939
>>>   sec_pagetables:0 bounce:0
>>>   kernel_misc_reclaimable:0
>>>   free:1340 free_pcp:23 free_cma:0
>>> Node 0 active_anon:940008kB inactive_anon:963900kB active_file:17424kB inactive_file:10204kB unevictable:16kB isolated(anon):0kB isolated(file):0kB mapped:22404kB dirty:28kB writeback:76kB shmem:84kB writeback_tmp:0kB kernel_stack:1552kB pagetables:3756kB sec_pagetables:0kB all_unreclaimable? no
>>> DMA free:0kB boost:7756kB min:11208kB low:12068kB high:12928kB reserved_highatomic:0KB active_anon:644060kB inactive_anon:36332kB active_file:5276kB inactive_file:5516kB unevictable:0kB writepending:348kB present:786432kB managed:746644kB mlocked:0kB bounce:0kB free_pcp:92kB local_pcp:92kB free_cma:0kB
>>> lowmem_reserve[]: 0 0 1280 1280
>>> DMA: 0*4kB 0*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 0kB
>>> 116345 total pagecache pages
>>> 109413 pages in swap cache
>>> Free swap  = 7819300kB
>>> Total swap = 8388604kB
>>> 524288 pages RAM
>>> 327680 pages HighMem/MovableOnly
>>> 9947 pages reserved
>>>
>>>
>>> I switched from zstd to lzo as zswap default compressor so zstd does not show up on the dmesg. But the rest looks pretty similar.
>>>
>>> Full dmesg and kernel .config attached.
>>>
>>> Regards,
>>> Erhard
>>
>> Hi Erhard,
>>
>> Thanks for the reports. I'll take a look at them and get back to you
>> in a few days.
> 
> Hi Erhard,
> 
> The OOM kills on both kernel versions seem to be reasonable to me.
> 
> Your system has 2GB memory and it uses zswap with zsmalloc (which is
> good since it can allocate from the highmem zone) and zstd/lzo (which
> doesn't matter much). Somehow -- I couldn't figure out why -- it
> splits the 2GB into a 0.25GB DMA zone and a 1.75GB highmem zone:
> 
> [    0.000000] Zone ranges:
> [    0.000000]   DMA      [mem 0x0000000000000000-0x000000002fffffff]
> [    0.000000]   Normal   empty
> [    0.000000]   HighMem  [mem 0x0000000030000000-0x000000007fffffff]

That's really odd. But we are messing with "PowerMac3,6", so I don't 
really know what's right or wrong ...

-- 
Cheers,

David / dhildenb