linux-kernel - Re: kswapd0: page allocation failure: order:0, mode:0x820(GFP_ATOMIC), nodemask=(null),cpuset=/,mems

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives
Hash Suite: Windows password security audit tool. GUI, reports in PDF.
[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]
Message-ID: <CAOUHufZ36rQc8AfLtRv2QrEareysdvbprAEO5XkcG-FeDOxFLA@mail.gmail.com>
Date: Sat, 1 Jun 2024 00:01:48 -0600
From: Yu Zhao <yuzhao@...gle.com>
To: Erhard Furtner <erhard_f@...lbox.org>
Cc: linux-mm@...ck.org, linux-kernel@...r.kernel.org, 
	linuxppc-dev@...ts.ozlabs.org
Subject: Re: kswapd0: page allocation failure: order:0, mode:0x820(GFP_ATOMIC),
 nodemask=(null),cpuset=/,mems_allowed=0 (Kernel v6.5.9, 32bit ppc)

On Wed, May 15, 2024 at 4:06 PM Yu Zhao <yuzhao@...gle.com> wrote:
>
> On Wed, May 15, 2024 at 2:45 PM Erhard Furtner <erhard_f@...lbox.org> wrote:
> >
> > On Wed, 8 May 2024 20:21:11 +0200
> > Erhard Furtner <erhard_f@...lbox.org> wrote:
> >
> > > Greetings!
> > >
> > > Got that on my dual CPU PowerMac G4 DP shortly after boot. This does not happen every time at bootup though:
> > >
> > > [...]
> > > kswapd0: page allocation failure: order:0, mode:0x820(GFP_ATOMIC), nodemask=(null),cpuset=/,mems_allowed=0
> > > CPU: 1 PID: 40 Comm: kswapd0 Not tainted 6.8.9-gentoo-PMacG4 #1
> > > Hardware name: PowerMac3,6 7455 0x80010303 PowerMac
> >
> > Very similar page allocation failure on the same machine on kernel 6.9.0 too. Seems it can easily be provoked by running a memory stressor, e.g. "stress-ng --vm 2 --vm-bytes 1930M --verify -v":
> >
> > [...]
> > kswapd0: page allocation failure: order:0, mode:0xcc0(GFP_KERNEL), nodemask=(null),cpuset=/,mems_allowed=0
> > CPU: 0 PID: 41 Comm: kswapd0 Not tainted 6.9.0-gentoo-PMacG4 #1
> > Hardware name: PowerMac3,6 7455 0x80010303 PowerMac
> > Call Trace:
> > [c1c65940] [c07926d4] dump_stack_lvl+0x80/0xac (unreliable)
> > [c1c65960] [c01b6234] warn_alloc+0x100/0x178
> > [c1c659c0] [c01b661c] __alloc_pages+0x370/0x8d0
> > [c1c65a80] [c01c4854] __read_swap_cache_async+0xc0/0x1cc
> > [c1c65ad0] [c01cb580] zswap_writeback_entry+0x50/0x154
> > [c1c65be0] [c01cb6f4] shrink_memcg_cb+0x70/0xec
> > [c1c65c10] [c019518c] __list_lru_walk_one+0xa0/0x154
> > [c1c65c70] [c01952a4] list_lru_walk_one+0x64/0x7c
> > [c1c65ca0] [c01cad00] zswap_shrinker_scan+0xac/0xc4
> > [c1c65cd0] [c018052c] do_shrink_slab+0x18c/0x304
> > [c1c65d20] [c0180a40] shrink_slab+0x174/0x260
> > [c1c65da0] [c017cb0c] shrink_one+0xbc/0x134
> > [c1c65dd0] [c017e3e4] shrink_node+0x238/0x84c
> > [c1c65e50] [c017ed38] balance_pgdat+0x340/0x650
> > [c1c65f50] [c017f270] kswapd+0x228/0x25c
> > [c1c65fc0] [c006bbac] kthread+0xe4/0xe8
> > [c1c65ff0] [c0015304] start_kernel_thread+0x10/0x14
> > SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
> >   cache: skbuff_small_head, object size: 480, buffer size: 544, default order: 1, min order: 0
> >   node 0: slabs: 15, objs: 225, free: 0
> > SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
> >   cache: skbuff_small_head, object size: 480, buffer size: 544, default order: 1, min order: 0
> >   node 0: slabs: 15, objs: 225, free: 0
> > SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
> >   cache: skbuff_small_head, object size: 480, buffer size: 544, default order: 1, min order: 0
> >   node 0: slabs: 15, objs: 225, free: 0
> > SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
> >   cache: skbuff_small_head, object size: 480, buffer size: 544, default order: 1, min order: 0
> >   node 0: slabs: 15, objs: 225, free: 0
> > SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
> >   cache: skbuff_small_head, object size: 480, buffer size: 544, default order: 1, min order: 0
> >   node 0: slabs: 15, objs: 225, free: 0
> > SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
> >   cache: kmalloc-rnd-15-2k, object size: 2048, buffer size: 6144, default order: 3, min order: 1
> >   kmalloc-rnd-15-2k debugging increased min order, use slab_debug=O to disable.
> >   node 0: slabs: 33, objs: 165, free: 0
> > SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
> >   cache: skbuff_small_head, object size: 480, buffer size: 544, default order: 1, min order: 0
> >   node 0: slabs: 15, objs: 225, free: 0
> > SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
> >   cache: kmalloc-rnd-15-2k, object size: 2048, buffer size: 6144, default order: 3, min order: 1
> >   kmalloc-rnd-15-2k debugging increased min order, use slab_debug=O to disable.
> >   node 0: slabs: 33, objs: 165, free: 0
> > SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
> >   cache: skbuff_small_head, object size: 480, buffer size: 544, default order: 1, min order: 0
> >   node 0: slabs: 15, objs: 225, free: 0
> > SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
> >   cache: kmalloc-rnd-15-2k, object size: 2048, buffer size: 6144, default order: 3, min order: 1
> >   kmalloc-rnd-15-2k debugging increased min order, use slab_debug=O to disable.
> >   node 0: slabs: 33, objs: 165, free: 0
> > Mem-Info:
> > active_anon:340071 inactive_anon:139179 isolated_anon:0
> >  active_file:8297 inactive_file:2506 isolated_file:0
> >  unevictable:4 dirty:1 writeback:18
> >  slab_reclaimable:1377 slab_unreclaimable:7426
> >  mapped:6804 shmem:112 pagetables:946
> >  sec_pagetables:0 bounce:0
> >  kernel_misc_reclaimable:0
> >  free:1141 free_pcp:7 free_cma:0
> > Node 0 active_anon:1360284kB inactive_anon:556716kB active_file:33188kB inactive_file:10024kB unevictable:16kB isolated(anon):0kB isolated(file):0kB mapped:27216kB dirty:4kB writeback:72kB shmem:448kB writeback_tmp:0kB kernel_stack:1560kB pagetables:3784kB sec_pagetables:0kB all_unreclaimable? no
> > DMA free:56kB boost:7756kB min:11208kB low:12068kB high:12928kB reserved_highatomic:0KB active_anon:635128kB inactive_anon:58260kB active_file:268kB inactive_file:3000kB unevictable:0kB writepending:324kB present:786432kB managed:746644kB mlocked:0kB bounce:0kB free_pcp:28kB local_pcp:28kB free_cma:0kB
> > lowmem_reserve[]: 0 0 1280 1280
> > DMA: 0*4kB 0*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 0kB
> > 63943 total pagecache pages
> > 53024 pages in swap cache
> > Free swap  = 8057248kB
> > Total swap = 8388604kB
> > 524288 pages RAM
> > 327680 pages HighMem/MovableOnly
> > 9947 pages reserved
> > warn_alloc: 396 callbacks suppressed
> > kswapd0: page allocation failure: order:0, mode:0xcc0(GFP_KERNEL), nodemask=(null),cpuset=/,mems_allowed=0
> > CPU: 1 PID: 41 Comm: kswapd0 Not tainted 6.9.0-gentoo-PMacG4 #1
> > Hardware name: PowerMac3,6 7455 0x80010303 PowerMac
> > Call Trace:
> > [c1c65940] [c07926d4] dump_stack_lvl+0x80/0xac (unreliable)
> > [c1c65960] [c01b6234] warn_alloc+0x100/0x178
> > [c1c659c0] [c01b661c] __alloc_pages+0x370/0x8d0
> > [c1c65a80] [c01c4854] __read_swap_cache_async+0xc0/0x1cc
> > [c1c65ad0] [c01cb580] zswap_writeback_entry+0x50/0x154
> > [c1c65be0] [c01cb6f4] shrink_memcg_cb+0x70/0xec
> > [c1c65c10] [c019518c] __list_lru_walk_one+0xa0/0x154
> > [c1c65c70] [c01952a4] list_lru_walk_one+0x64/0x7c
> > [c1c65ca0] [c01cad00] zswap_shrinker_scan+0xac/0xc4
> > [c1c65cd0] [c018052c] do_shrink_slab+0x18c/0x304
> > [c1c65d20] [c0180a40] shrink_slab+0x174/0x260
> > [c1c65da0] [c017cb0c] shrink_one+0xbc/0x134
> > [c1c65dd0] [c017e3e4] shrink_node+0x238/0x84c
> > [c1c65e50] [c017ed38] balance_pgdat+0x340/0x650
> > [c1c65f50] [c017f270] kswapd+0x228/0x25c
> > slab_out_of_memory: 53 callbacks suppressed
> > SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
> >   cache: skbuff_small_head, object size: 480, buffer size: 544, default order: 1, min order: 0
> >   node 0: slabs: 18, objs: 270, free: 0
> > SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
> >   cache: skbuff_small_head, object size: 480, buffer size: 544, default order: 1, min order: 0
> >   node 0: slabs: 18, objs: 270, free: 0
> > SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
> >   cache: skbuff_small_head, object size: 480, buffer size: 544, default order: 1, min order: 0
> >   node 0: slabs: 18, objs: 270, free: 0
> > SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
> >   cache: skbuff_small_head, object size: 480, buffer size: 544, default order: 1, min order: 0
> >   node 0: slabs: 18, objs: 270, free: 0
> > SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
> >   cache: skbuff_small_head, object size: 480, buffer size: 544, default order: 1, min order: 0
> >   node 0: slabs: 18, objs: 270, free: 0
> > SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
> >   cache: kmalloc-rnd-15-2k, object size: 2048, buffer size: 6144, default order: 3, min order: 1
> >   kmalloc-rnd-15-2k debugging increased min order, use slab_debug=O to disable.
> >   node 0: slabs: 33, objs: 165, free: 0
> > SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
> >   cache: skbuff_small_head, object size: 480, buffer size: 544, default order: 1, min order: 0
> >   node 0: slabs: 18, objs: 270, free: 0
> > SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
> >   cache: kmalloc-rnd-15-2k, object size: 2048, buffer size: 6144, default order: 3, min order: 1
> >   kmalloc-rnd-15-2k debugging increased min order, use slab_debug=O to disable.
> >   node 0: slabs: 33, objs: 165, free: 0
> > SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
> >   cache: skbuff_small_head, object size: 480, buffer size: 544, default order: 1, min order: 0
> >   node 0: slabs: 18, objs: 270, free: 0
> > SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
> >   cache: kmalloc-rnd-15-2k, object size: 2048, buffer size: 6144, default order: 3, min order: 1
> >   kmalloc-rnd-15-2k debugging increased min order, use slab_debug=O to disable.
> >   node 0: slabs: 33, objs: 165, free: 0
> > [c1c65fc0] [c006bbac] kthread+0xe4/0xe8
> > [c1c65ff0] [c0015304] start_kernel_thread+0x10/0x14
> > Mem-Info:
> > active_anon:351976 inactive_anon:123514 isolated_anon:0
> >  active_file:4648 inactive_file:2081 isolated_file:0
> >  unevictable:4 dirty:1 writeback:39
> >  slab_reclaimable:918 slab_unreclaimable:7222
> >  mapped:5359 shmem:21 pagetables:940
> >  sec_pagetables:0 bounce:0
> >  kernel_misc_reclaimable:0
> >  free:2563 free_pcp:142 free_cma:0
> > Node 0 active_anon:1407904kB inactive_anon:494056kB active_file:18592kB inactive_file:8324kB unevictable:16kB isolated(anon):0kB isolated(file):0kB mapped:21436kB dirty:4kB writeback:156kB shmem:84kB writeback_tmp:0kB kernel_stack:1552kB pagetables:3760kB sec_pagetables:0kB all_unreclaimable? no
> > DMA free:0kB boost:7756kB min:11208kB low:12068kB high:12928kB reserved_highatomic:0KB active_anon:199336kB inactive_anon:491432kB active_file:4612kB inactive_file:5980kB unevictable:0kB writepending:660kB present:786432kB managed:746644kB mlocked:0kB bounce:0kB free_pcp:568kB local_pcp:20kB free_cma:0kB
> > lowmem_reserve[]: 0 0 1280 1280
> > DMA: 0*4kB 0*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 0kB
> > 45961 total pagecache pages
> > 39207 pages in swap cache
> > Free swap  = 8093096kB
> > Total swap = 8388604kB
> > 524288 pages RAM
> > 327680 pages HighMem/MovableOnly
> > 9947 pages reserved
> > warn_alloc: 343 callbacks suppressed
> > kswapd0: page allocation failure: order:0, mode:0xcc0(GFP_KERNEL), nodemask=(null),cpuset=/,mems_allowed=0
> > CPU: 0 PID: 41 Comm: kswapd0 Not tainted 6.9.0-gentoo-PMacG4 #1
> > Hardware name: PowerMac3,6 7455 0x80010303 PowerMac
> > Call Trace:
> > [c1c65940] [c07926d4] dump_stack_lvl+0x80/0xac (unreliable)
> > [c1c65960] [c01b6234] warn_alloc+0x100/0x178
> > [c1c659c0] [c01b661c] __alloc_pages+0x370/0x8d0
> > [c1c65a80] [c01c4854] __read_swap_cache_async+0xc0/0x1cc
> > [c1c65ad0] [c01cb580] zswap_writeback_entry+0x50/0x154
> > [c1c65be0] [c01cb6f4] shrink_memcg_cb+0x70/0xec
> > [c1c65c10] [c019518c] __list_lru_walk_one+0xa0/0x154
> > slab_out_of_memory: 59 callbacks suppressed
> > SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
> >   cache: skbuff_small_head, object size: 480, buffer size: 544, default order: 1, min order: 0
> >   node 0: slabs: 18, objs: 270, free: 0
> > SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
> >   cache: skbuff_small_head, object size: 480, buffer size: 544, default order: 1, min order: 0
> >   node 0: slabs: 18, objs: 270, free: 0
> > SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
> >   cache: skbuff_small_head, object size: 480, buffer size: 544, default order: 1, min order: 0
> >   node 0: slabs: 18, objs: 270, free: 0
> > SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
> >   cache: skbuff_small_head, object size: 480, buffer size: 544, default order: 1, min order: 0
> >   node 0: slabs: 18, objs: 270, free: 0
> > SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
> >   cache: skbuff_small_head, object size: 480, buffer size: 544, default order: 1, min order: 0
> >   node 0: slabs: 18, objs: 270, free: 0
> > SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
> >   cache: kmalloc-rnd-15-2k, object size: 2048, buffer size: 6144, default order: 3, min order: 1
> >   kmalloc-rnd-15-2k debugging increased min order, use slab_debug=O to disable.
> >   node 0: slabs: 33, objs: 165, free: 0
> > SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
> >   cache: skbuff_small_head, object size: 480, buffer size: 544, default order: 1, min order: 0
> >   node 0: slabs: 18, objs: 270, free: 0
> > SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
> >   cache: kmalloc-rnd-15-2k, object size: 2048, buffer size: 6144, default order: 3, min order: 1
> >   kmalloc-rnd-15-2k debugging increased min order, use slab_debug=O to disable.
> >   node 0: slabs: 33, objs: 165, free: 0
> > SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
> >   cache: skbuff_small_head, object size: 480, buffer size: 544, default order: 1, min order: 0
> >   node 0: slabs: 18, objs: 270, free: 0
> > SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
> >   cache: kmalloc-rnd-15-2k, object size: 2048, buffer size: 6144, default order: 3, min order: 1
> >   kmalloc-rnd-15-2k debugging increased min order, use slab_debug=O to disable.
> >   node 0: slabs: 33, objs: 165, free: 0
> > [c1c65c70] [c01952a4] list_lru_walk_one+0x64/0x7c
> > [c1c65ca0] [c01cad00] zswap_shrinker_scan+0xac/0xc4
> > [c1c65cd0] [c018052c] do_shrink_slab+0x18c/0x304
> > [c1c65d20] [c0180a40] shrink_slab+0x174/0x260
> > [c1c65da0] [c017cb0c] shrink_one+0xbc/0x134
> > [c1c65dd0] [c017e3e4] shrink_node+0x238/0x84c
> > [c1c65e50] [c017ed38] balance_pgdat+0x340/0x650
> > [c1c65f50] [c017f270] kswapd+0x228/0x25c
> > [c1c65fc0] [c006bbac] kthread+0xe4/0xe8
> > [c1c65ff0] [c0015304] start_kernel_thread+0x10/0x14
> > Mem-Info:
> > active_anon:235002 inactive_anon:240975 isolated_anon:0
> >  active_file:4356 inactive_file:2551 isolated_file:0
> >  unevictable:4 dirty:7 writeback:19
> >  slab_reclaimable:1008 slab_unreclaimable:7218
> >  mapped:5601 shmem:21 pagetables:939
> >  sec_pagetables:0 bounce:0
> >  kernel_misc_reclaimable:0
> >  free:1340 free_pcp:23 free_cma:0
> > Node 0 active_anon:940008kB inactive_anon:963900kB active_file:17424kB inactive_file:10204kB unevictable:16kB isolated(anon):0kB isolated(file):0kB mapped:22404kB dirty:28kB writeback:76kB shmem:84kB writeback_tmp:0kB kernel_stack:1552kB pagetables:3756kB sec_pagetables:0kB all_unreclaimable? no
> > DMA free:0kB boost:7756kB min:11208kB low:12068kB high:12928kB reserved_highatomic:0KB active_anon:644060kB inactive_anon:36332kB active_file:5276kB inactive_file:5516kB unevictable:0kB writepending:348kB present:786432kB managed:746644kB mlocked:0kB bounce:0kB free_pcp:92kB local_pcp:92kB free_cma:0kB
> > lowmem_reserve[]: 0 0 1280 1280
> > DMA: 0*4kB 0*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 0kB
> > 116345 total pagecache pages
> > 109413 pages in swap cache
> > Free swap  = 7819300kB
> > Total swap = 8388604kB
> > 524288 pages RAM
> > 327680 pages HighMem/MovableOnly
> > 9947 pages reserved
> >
> >
> > I switched from zstd to lzo as zswap default compressor so zstd does not show up on the dmesg. But the rest looks pretty similar.
> >
> > Full dmesg and kernel .config attached.
> >
> > Regards,
> > Erhard
>
> Hi Erhard,
>
> Thanks for the reports. I'll take a look at them and get back to you
> in a few days.

Hi Erhard,

The OOM kills on both kernel versions seem to be reasonable to me.

Your system has 2GB memory and it uses zswap with zsmalloc (which is
good since it can allocate from the highmem zone) and zstd/lzo (which
doesn't matter much). Somehow -- I couldn't figure out why -- it
splits the 2GB into a 0.25GB DMA zone and a 1.75GB highmem zone:

[    0.000000] Zone ranges:
[    0.000000]   DMA      [mem 0x0000000000000000-0x000000002fffffff]
[    0.000000]   Normal   empty
[    0.000000]   HighMem  [mem 0x0000000030000000-0x000000007fffffff]

The kernel can't allocate from the highmem zone -- only userspace and
zsmalloc can. OOM kills were due to the low memory conditions in the
DMA zone where the kernel itself failed to allocate from.

Do you know a kernel version that doesn't have OOM kills while running
the same workload? If so, could you send that .config to me? If not,
could you try disabling CONFIG_HIGHMEM? (It might not help but I'm out
of ideas at the moment.)

Thanks!