Date:   Tue, 27 Jun 2017 15:32:19 +0200
From:   Paul Menzel <pmenzel@...gen.mpg.de>
To:     netdev@...r.kernel.org, linux-mm@...ck.org,
        it+linux-netdev@...gen.mpg.de
Subject: [bnx2x] swapper/0: page allocation failure: order:0,
 mode:0x2080020(GFP_ATOMIC)

Dear Linux folks,


With Linux 4.9.32 (and older versions), the following messages are 
reported under high network load.

> [34615.695622] swapper/0: page allocation failure: order:0, mode:0x2080020(GFP_ATOMIC)
> [34615.695631] CPU: 0 PID: 0 Comm: swapper/0 Not tainted 4.9.32.mx64.159 #1
> [34615.695633] Hardware name: Dell Inc. PowerEdge R910/0KYD3D, BIOS 2.10.0 08/29/2013
> [34615.695634]  ffff883f7f803b98 ffffffff813e2ac6 ffffffff81f33ce0 0000000000000000
> [34615.695636]  ffff883f7f803c18 ffffffff8113fdac 0208002000000011 ffffffff81f33ce0
> [34615.695638]  ffff883f7f803bc0 ffff884000000010 ffff883f7f803c28 ffff883f7f803bd8
> [34615.695640] Call Trace:
> [34615.695642]  <IRQ>
> [34615.695651]  [<ffffffff813e2ac6>] dump_stack+0x4d/0x67
> [34615.695655]  [<ffffffff8113fdac>] warn_alloc+0x11c/0x140
> [34615.695657]  [<ffffffff811402ab>] __alloc_pages_nodemask+0x45b/0xe70
> [34615.695661]  [<ffffffff81183cec>] alloc_pages_current+0xac/0x160
> [34615.695667]  [<ffffffff816edfc0>] bnx2x_alloc_rx_sge+0x170/0x220
> [34615.695669]  [<ffffffff816f0c8b>] bnx2x_poll+0x5bb/0x19d0
> [34615.695670]  [<ffffffff816f083f>] ? bnx2x_poll+0x16f/0x19d0
> [34615.695676]  [<ffffffff819eb089>] net_rx_action+0x1f9/0x340
> [34615.695680]  [<ffffffff81a9c228>] __do_softirq+0x88/0x29b
> [34615.695684]  [<ffffffff81064206>] irq_exit+0x76/0x80
> [34615.695686]  [<ffffffff81a9bfd3>] do_IRQ+0x63/0xf0
> [34615.695689]  [<ffffffff81a9a4bf>] common_interrupt+0x7f/0x7f
> [34615.695689]  <EOI>
> [34615.695696]  [<ffffffff81903574>] ? cpuidle_enter_state+0x144/0x2e0
> [34615.695697]  [<ffffffff81903569>] ? cpuidle_enter_state+0x139/0x2e0
> [34615.695699]  [<ffffffff81903747>] cpuidle_enter+0x17/0x20
> [34615.695703]  [<ffffffff8109e103>] call_cpuidle+0x23/0x40
> [34615.695704]  [<ffffffff8109e324>] cpu_startup_entry+0x104/0x1e0
> [34615.695706]  [<ffffffff81a93ef7>] rest_init+0x77/0x80
> [34615.695708]  [<ffffffff8235fec3>] start_kernel+0x3b3/0x3c0
> [34615.695710]  [<ffffffff8235f28f>] x86_64_start_reservations+0x2a/0x2c
> [34615.695711]  [<ffffffff8235f3f9>] x86_64_start_kernel+0x168/0x176
> [34615.695712] Mem-Info:
> [34615.695747] active_anon:275374 inactive_anon:304826 isolated_anon:0
> [34615.695747]  active_file:252290929 inactive_file:8578985 isolated_file:128
> [34615.695747]  unevictable:0 dirty:4122065 writeback:166205 unstable:0
> [34615.695747]  slab_reclaimable:1643416 slab_unreclaimable:57412
> [34615.695747]  mapped:3551 shmem:8 pagetables:2275 bounce:0
> [34615.695747]  free:265929 free_pcp:42297 free_cma:0
> [34615.695757] Node 0 active_anon:276944kB inactive_anon:277380kB active_file:249203496kB inactive_file:9551688kB unevictable:0kB isolated(anon):0kB isolated(file):128kB mapped:1276kB dirty:5052260kB writeback:111676kB shmem:8kB writeback_tmp:0kB unstable:0kB pages_scanned:128 all_unreclaimable? no
> [34615.695769] Node 1 active_anon:144016kB inactive_anon:144900kB active_file:253005028kB inactive_file:7623120kB unevictable:0kB isolated(anon):0kB isolated(file):128kB mapped:2560kB dirty:3717196kB writeback:93784kB shmem:12kB writeback_tmp:0kB unstable:0kB pages_scanned:34 all_unreclaimable? no
> [34615.695782] Node 2 active_anon:164000kB inactive_anon:164004kB active_file:254373252kB inactive_file:7768156kB unevictable:0kB isolated(anon):0kB isolated(file):128kB mapped:3164kB dirty:3754752kB writeback:157076kB shmem:4kB writeback_tmp:0kB unstable:0kB pages_scanned:32 all_unreclaimable? no
> [34615.695795] Node 3 active_anon:516536kB inactive_anon:633020kB active_file:252581940kB inactive_file:9372976kB unevictable:0kB isolated(anon):0kB isolated(file):128kB mapped:7204kB dirty:3964052kB writeback:302284kB shmem:8kB writeback_tmp:0kB unstable:0kB pages_scanned:29 all_unreclaimable? no
> [34615.695797] Node 0 DMA free:15904kB min:0kB low:12kB high:24kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:15988kB managed:15904kB mlocked:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
> [34615.695800] lowmem_reserve[]: 0 1672 257669 257669
> [34615.695803] Node 0 DMA32 free:1023928kB min:104kB low:1816kB high:3528kB active_anon:12kB inactive_anon:156kB active_file:639616kB inactive_file:37904kB unevictable:0kB writepending:9860kB present:2067680kB managed:1722564kB mlocked:0kB slab_reclaimable:4572kB slab_unreclaimable:140kB kernel_stack:416kB pagetables:0kB bounce:0kB free_pcp:13752kB local_pcp:188kB free_cma:0kB
> [34615.695807] lowmem_reserve[]: 0 0 255997 255997
> [34615.695810] Node 0 Normal free:6036kB min:16256kB low:278396kB high:540536kB active_anon:276932kB inactive_anon:277224kB active_file:248563880kB inactive_file:9513784kB unevictable:0kB writepending:5154076kB present:266338304kB managed:262141028kB mlocked:0kB slab_reclaimable:2858988kB slab_unreclaimable:63008kB kernel_stack:15936kB pagetables:2592kB bounce:0kB free_pcp:38680kB local_pcp:132kB free_cma:0kB
> [34615.695813] lowmem_reserve[]: 0 0 0 0
> [34615.695816] Node 1 Normal free:5840kB min:16388kB low:280624kB high:544860kB active_anon:144016kB inactive_anon:144900kB active_file:253005028kB inactive_file:7623120kB unevictable:0kB writepending:3810980kB present:268435456kB managed:264238448kB mlocked:0kB slab_reclaimable:2012588kB slab_unreclaimable:60932kB kernel_stack:16712kB pagetables:1736kB bounce:0kB free_pcp:40480kB local_pcp:768kB free_cma:0kB
> [34615.695819] lowmem_reserve[]: 0 0 0 0
> [34615.695821] Node 2 Normal free:6020kB min:16388kB low:280624kB high:544860kB active_anon:164000kB inactive_anon:164004kB active_file:254373252kB inactive_file:7768156kB unevictable:0kB writepending:3911828kB present:268435456kB managed:264238448kB mlocked:0kB slab_reclaimable:1196052kB slab_unreclaimable:43876kB kernel_stack:21056kB pagetables:1716kB bounce:0kB free_pcp:39772kB local_pcp:740kB free_cma:0kB
> [34615.695825] lowmem_reserve[]: 0 0 0 0
> [34615.695827] Node 3 Normal free:5988kB min:16388kB low:280624kB high:544860kB active_anon:516536kB inactive_anon:633020kB active_file:252581940kB inactive_file:9372976kB unevictable:0kB writepending:4266336kB present:268435456kB managed:264237500kB mlocked:0kB slab_reclaimable:501464kB slab_unreclaimable:61692kB kernel_stack:37208kB pagetables:3056kB bounce:0kB free_pcp:36504kB local_pcp:728kB free_cma:0kB
> [34615.695830] lowmem_reserve[]: 0 0 0 0
> [34615.695832] Node 0 DMA: 0*4kB 0*8kB 0*16kB 1*32kB (U) 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15904kB
> [34615.695839] Node 0 DMA32: 174*4kB (UME) 564*8kB (UME) 504*16kB (UME) 247*32kB (UME) 102*64kB (UME) 23*128kB (UE) 12*256kB (UME) 24*512kB (UME) 15*1024kB (UME) 2*2048kB (U) 234*4096kB (M) = 1023928kB
> [34615.695848] Node 0 Normal: 241*4kB (UME) 38*8kB (UME) 24*16kB (UME) 52*32kB (UM) 11*64kB (UM) 0*128kB 0*256kB 0*512kB 0*1024kB 1*2048kB (H) 0*4096kB = 6068kB
> [34615.695855] Node 1 Normal: 484*4kB (MEH) 36*8kB (UMEH) 38*16kB (UMEH) 64*32kB (UMH) 7*64kB (H) 4*128kB (H) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 5840kB
> [34615.695862] Node 2 Normal: 305*4kB (UMEH) 423*8kB (MEH) 12*16kB (MH) 8*32kB (H) 5*64kB (H) 3*128kB (H) 2*256kB (H) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 6268kB
> [34615.695870] Node 3 Normal: 105*4kB (UME) 758*8kB (M) 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 6484kB
> [34615.695876] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
> [34615.695877] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
> [34615.695878] Node 2 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
> [34615.695878] Node 3 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
> [34615.695879] 260870776 total pagecache pages
> [34615.695880] 405 pages in swap cache
> [34615.695881] Swap cache stats: add 2291, delete 1886, find 8/9
> [34615.695881] Free swap  = 993530660kB
> [34615.695882] Total swap = 993539792kB
> [34615.695882] 268432085 pages RAM
> [34615.695883] 0 pages HighMem/MovableOnly
> [34615.695883] 4283612 pages reserved

In Linux kernel bug #12135 [1], Andrew comments [2]:

> yes, it's just a diagnostic warning that the network is allocating
> memory faster than the VM system is reclaiming it.  Everything should
> recover and run to completion.
> 
> You should be able to make the warning go away by increasing /proc/sys/vm/min_free_kbytes.
On that system with 1 TB of memory, the value is 64 MB.

```
$ more /proc/sys/vm/min_free_kbytes
65536
```

Do you have suggestions on how to tune the system to get rid of the 
messages? Could the Linux kernel be improved?


Kind regards,

Paul


[1] https://bugzilla.kernel.org/show_bug.cgi?id=12135
[2] https://bugzilla.kernel.org/show_bug.cgi?id=12135#c2

View attachment "linux_4.9.32-bnx-page_allocation_errors.txt" of type "text/plain" (112695 bytes)

Powered by blists - more mailing lists