linux-kernel - Re: Kernel falls apart under light memory pressure (i.e. linking vmlinux)

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives
Hash Suite for Android: free password hash cracker in your pocket
[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]
Date:	Fri, 20 May 2011 10:11:47 -0400
From:	Andrew Lutomirski <luto@....edu>
To:	Andrea Arcangeli <aarcange@...hat.com>
Cc:	KAMEZAWA Hiroyuki <kamezawa.hiroyu@...fujitsu.com>,
	Minchan Kim <minchan.kim@...il.com>,
	KOSAKI Motohiro <kosaki.motohiro@...fujitsu.com>,
	fengguang.wu@...el.com, andi@...stfloor.org, linux-mm@...ck.org,
	linux-kernel@...r.kernel.org, mgorman@...e.de, hannes@...xchg.org,
	riel@...hat.com
Subject: Re: Kernel falls apart under light memory pressure (i.e. linking vmlinux)

On Fri, May 20, 2011 at 6:11 AM, Andrea Arcangeli <aarcange@...hat.com> wrote:
> I figure it's not easily reproducible but you can easily rule out THP
> issues by reproducing at least once after booting with
> transparent_hugepage=never or by building the kernel with
> CONFIG_TRANSPARENT_HUGEPAGE=n.

Reproduced with CONFIG_TRANSPARENT_HUGEPAGE=n with and without
compaction and migration.

I applied the attached patch (which includes Minchan's !pgdat_balanced
and need_resched changes).  I see:

[  121.468339] firefox shrink_page_list+0x4f3/0x5ca:
SetPageActive(ffffea00019217a8) w/ prev = 100000000002000D
[  121.469236] firefox shrink_page_list+0x4f3/0x5ca:
SetPageActive(ffffea00016596b8) w/ prev = 100000000002000D
[  121.470207] firefox: shrink_page_list (nr_scanned=94
nr_reclaimed=19 nr_to_reclaim=32 gfp_mask=201DA) found inactive page
ffffea00019217a8 with flags=100000000002004D
[  121.472451] firefox: shrink_page_list (nr_scanned=94
nr_reclaimed=19 nr_to_reclaim=32 gfp_mask=201DA) found inactive page
ffffea00016596b8 with flags=100000000002004D
[  121.482782] dd shrink_page_list+0x4f3/0x5ca:
SetPageActive(ffffea00013a8938) w/ prev = 100000000002000D
[  121.489820] dd shrink_page_list+0x4f3/0x5ca:
SetPageActive(ffffea00017a4e88) w/ prev = 1000000000000801
[  121.490626] dd shrink_page_list+0x4f3/0x5ca:
SetPageActive(ffffea000005edb0) w/ prev = 1000000000000801
[  121.491499] dd: shrink_page_list (nr_scanned=62 nr_reclaimed=0
nr_to_reclaim=32 gfp_mask=200D2) found inactive page ffffea00017a4e88
with flags=1000000000000841
[  121.494337] dd: shrink_page_list (nr_scanned=62 nr_reclaimed=0
nr_to_reclaim=32 gfp_mask=200D2) found inactive page ffffea000005edb0
with flags=1000000000000841
[  121.499219] dd shrink_page_list+0x4f3/0x5ca:
SetPageActive(ffffea000129c788) w/ prev = 1000000000080009
[  121.500363] dd shrink_page_list+0x4f3/0x5ca:
SetPageActive(ffffea000129c830) w/ prev = 1000000000080009
[  121.502270] kswapd0 shrink_page_list+0x4f3/0x5ca:
SetPageActive(ffffea0001146470) w/ prev = 100000000008001D
[  121.661545] kworker/1:1 shrink_page_list+0x4f3/0x5ca:
SetPageActive(ffffea0000058168) w/ prev = 1000000000000801
[  121.662791] kworker/1:1 shrink_page_list+0x4f3/0x5ca:
SetPageActive(ffffea000166f288) w/ prev = 1000000000000801
[  121.665727] kworker/1:1 shrink_page_list+0x4f3/0x5ca:
SetPageActive(ffffea0001681c40) w/ prev = 1000000000000801
[  121.666857] kworker/1:1 shrink_page_list+0x4f3/0x5ca:
SetPageActive(ffffea0001693130) w/ prev = 1000000000000801
[  121.667988] kworker/1:1 shrink_page_list+0x4f3/0x5ca:
SetPageActive(ffffea0000c790d8) w/ prev = 1000000000000801
[  121.669105] kworker/1:1 shrink_page_list+0x4f3/0x5ca:
SetPageActive(ffffea000113fe48) w/ prev = 1000000000000801
[  121.670238] kworker/1:1: shrink_page_list (nr_scanned=102
nr_reclaimed=20 nr_to_reclaim=32 gfp_mask=11212) found inactive page
ffffea0000058168 with flags=1000000000000841
[  121.674061] kworker/1:1: shrink_page_list (nr_scanned=102
nr_reclaimed=20 nr_to_reclaim=32 gfp_mask=11212) found inactive page
ffffea000166f288 with flags=1000000000000841
[  121.678054] kworker/1:1: shrink_page_list (nr_scanned=102
nr_reclaimed=20 nr_to_reclaim=32 gfp_mask=11212) found inactive page
ffffea0001681c40 with flags=1000000000000841
[  121.682069] kworker/1:1: shrink_page_list (nr_scanned=102
nr_reclaimed=20 nr_to_reclaim=32 gfp_mask=11212) found inactive page
ffffea0001693130 with flags=1000000000000841
[  121.686074] kworker/1:1: shrink_page_list (nr_scanned=102
nr_reclaimed=20 nr_to_reclaim=32 gfp_mask=11212) found inactive page
ffffea0000c790d8 with flags=1000000000000841
[  121.690045] kworker/1:1: shrink_page_list (nr_scanned=102
nr_reclaimed=20 nr_to_reclaim=32 gfp_mask=11212) found inactive page
ffffea000113fe48 with flags=1000000000000841
[  121.866205] test_mempressur shrink_page_list+0x4f3/0x5ca:
SetPageActive(ffffea000165d5b8) w/ prev = 100000000002000D
[  121.868204] test_mempressur shrink_page_list+0x4f3/0x5ca:
SetPageActive(ffffea0001661288) w/ prev = 100000000002000D
[  121.870203] test_mempressur shrink_page_list+0x4f3/0x5ca:
SetPageActive(ffffea0001661250) w/ prev = 100000000002000D
[  121.872195] test_mempressur shrink_page_list+0x4f3/0x5ca:
SetPageActive(ffffea000100cee8) w/ prev = 100000000002000D
[  121.873486] test_mempressur shrink_page_list+0x4f3/0x5ca:
SetPageActive(ffffea0000eafab8) w/ prev = 100000000002000D
[  121.874718] test_mempressur shrink_page_list+0x4f3/0x5ca:
SetPageActive(ffffea0000eafaf0) w/ prev = 100000000002000D

This is interesting: it looks like shrink_page_list is making its way
through the list more than once.  It could be reentering itself
somehow or it could have something screwed up with the linked list.

I'll keep slowly debugging, but maybe this is enough for someone
familiar with this code to beat me to it.

Minchan, I think this means that your fixes are just hiding and not
fixing the underlying problem.

Download attachment "vm_tests.patch" of type "application/octet-stream" (3074 bytes)