linux-kernel - Re: Kernel falls apart under light memory pressure (i.e. linking vmlinux)

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite for Android: free password hash cracker in your pocket

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <BANLkTikHMUru=w4zzRmosrg2bDbsFWrkTQ@mail.gmail.com>
Date:	Wed, 18 May 2011 22:15:53 -0400
From:	Andrew Lutomirski <luto@....edu>
To:	Minchan Kim <minchan.kim@...il.com>
Cc:	Wu Fengguang <fengguang.wu@...el.com>,
	Andi Kleen <andi@...stfloor.org>,
	"linux-mm@...ck.org" <linux-mm@...ck.org>,
	LKML <linux-kernel@...r.kernel.org>,
	KOSAKI Motohiro <kosaki.motohiro@...fujitsu.com>,
	KAMEZAWA Hiroyuki <kamezawa.hiroyu@...fujitsu.com>,
	Mel Gorman <mgorman@...e.de>,
	Johannes Weiner <hannes@...xchg.org>,
	Rik van Riel <riel@...hat.com>
Subject: Re: Kernel falls apart under light memory pressure (i.e. linking vmlinux)

On Wed, May 18, 2011 at 1:17 AM, Minchan Kim <minchan.kim@...il.com> wrote:
> On Wed, May 18, 2011 at 4:22 AM, Andrew Lutomirski <luto@....edu> wrote:
>>> No, thanks. However it would be valuable if you can retry with this
>>> patch _alone_ (without the "if (need_resched()) return false;" change,
>>> as I don't see how it helps your case).
>>>
>>> @@ -2286,7 +2290,7 @@ static bool sleeping_prematurely(pg_data_t
>>> *pgdat, int order, long remaining,
>>>        * must be balanced
>>>        */
>>>       if (order)
>>> -               return pgdat_balanced(pgdat, balanced, classzone_idx);
>>> +               return !pgdat_balanced(pgdat, balanced, classzone_idx);
>>>       else
>>>               return !all_zones_ok;
>>>  }
>>
>> Done.
>>
>> I logged in, added swap, and ran a program that allocated 1900MB of
>> RAM and memset it.  The system lagged a bit but survived.  kswapd
>> showed 10% CPU (which is odd, IMO, since I'm using aesni-intel and I
>> think that all the crypt happens in kworker when aesni-intel is in
>> use).
>
> I think kswapd could use 10% enough for reclaim.
>
>>
>> Then I started Firefox, loaded gmail, and ran test_mempressure.sh.
>> Kaboom!  (I.e. system was hung)  SysRq-F saved the system and produced
>
> Hang?
> It means you see softhangup of kswapd? or mouse/keyboard doesn't move?

Mouse and keyboard dead.

> Andrew, Could you test this patch with !pgdat_balanced patch?
> I think we shouldn't see OOM message if we have lots of free swap space.
>
> == CUT_HERE ==
> diff --git a/mm/vmscan.c b/mm/vmscan.c
> index f73b865..cc23f04 100644
> --- a/mm/vmscan.c
> +++ b/mm/vmscan.c
> @@ -1341,10 +1341,6 @@ static inline bool
> should_reclaim_stall(unsigned long nr_taken,
>        if (current_is_kswapd())
>                return false;
>
> -       /* Only stall on lumpy reclaim */
> -       if (sc->reclaim_mode & RECLAIM_MODE_SINGLE)
> -               return false;
> -
>        /* If we have relaimed everything on the isolated list, no stall */
>        if (nr_freed == nr_taken)
>                return false;
>
>
>
> Then, if you don't see any unnecessary OOM but still see the hangup,
> could you apply this patch based on previous?

With this patch, I started GNOME and Firefox, turned on swap, and ran
test_mempressure.sh 1500 1400 1.  Instant panic (or OOPS and hang or
something -- didn't get the top part).  Picture attached -- it looks
like memcg might be involved.  I'm running F15, so it might even be
doing something.

I won't be able to get netconsole dumps until next week because I'm
out of town and only have this one computer here.

I haven't tried the other patch.

Also, the !pgdat_balanced fix plus the if (need_resched()) return
false patch just hung once on 2.6.37-rc9.  I don't know what triggered
it.  Maybe yum.

--Andy

Download attachment "IMG_20110518_184222.jpg" of type "image/jpeg" (94535 bytes)