lists.openwall.net   lists  /  announce  owl-users  owl-dev  john-users  john-dev  passwdqc-users  yescrypt  popa3d-users  /  oss-security  kernel-hardening  musl  sabotage  tlsify  passwords  /  crypt-dev  xvendor  /  Bugtraq  Full-Disclosure  linux-kernel  linux-netdev  linux-ext4  linux-hardening  linux-cve-announce  PHC 
Open Source and information security mailing list archives
 
Hash Suite for Android: free password hash cracker in your pocket
[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]
Date:	Thu, 19 May 2011 11:54:05 +0900
From:	Minchan Kim <minchan.kim@...il.com>
To:	Andrew Lutomirski <luto@....edu>,
	KAMEZAWA Hiroyuki <kamezawa.hiroyu@...fujitsu.com>
Cc:	Wu Fengguang <fengguang.wu@...el.com>,
	Andi Kleen <andi@...stfloor.org>,
	"linux-mm@...ck.org" <linux-mm@...ck.org>,
	LKML <linux-kernel@...r.kernel.org>,
	KOSAKI Motohiro <kosaki.motohiro@...fujitsu.com>,
	Mel Gorman <mgorman@...e.de>,
	Johannes Weiner <hannes@...xchg.org>,
	Rik van Riel <riel@...hat.com>
Subject: Re: Kernel falls apart under light memory pressure (i.e. linking vmlinux)

On Thu, May 19, 2011 at 11:15 AM, Andrew Lutomirski <luto@....edu> wrote:
> On Wed, May 18, 2011 at 1:17 AM, Minchan Kim <minchan.kim@...il.com> wrote:
>> On Wed, May 18, 2011 at 4:22 AM, Andrew Lutomirski <luto@....edu> wrote:
>>>> No, thanks. However it would be valuable if you can retry with this
>>>> patch _alone_ (without the "if (need_resched()) return false;" change,
>>>> as I don't see how it helps your case).
>>>>
>>>> @@ -2286,7 +2290,7 @@ static bool sleeping_prematurely(pg_data_t
>>>> *pgdat, int order, long remaining,
>>>>        * must be balanced
>>>>        */
>>>>       if (order)
>>>> -               return pgdat_balanced(pgdat, balanced, classzone_idx);
>>>> +               return !pgdat_balanced(pgdat, balanced, classzone_idx);
>>>>       else
>>>>               return !all_zones_ok;
>>>>  }
>>>
>>> Done.
>>>
>>> I logged in, added swap, and ran a program that allocated 1900MB of
>>> RAM and memset it.  The system lagged a bit but survived.  kswapd
>>> showed 10% CPU (which is odd, IMO, since I'm using aesni-intel and I
>>> think that all the crypt happens in kworker when aesni-intel is in
>>> use).
>>
>> I think kswapd could use 10% enough for reclaim.
>>
>>>
>>> Then I started Firefox, loaded gmail, and ran test_mempressure.sh.
>>> Kaboom!  (I.e. system was hung)  SysRq-F saved the system and produced
>>
>> Hang?
>> It means you see softhangup of kswapd? or mouse/keyboard doesn't move?
>
> Mouse and keyboard dead.
>
>> Andrew, Could you test this patch with !pgdat_balanced patch?
>> I think we shouldn't see OOM message if we have lots of free swap space.
>>
>> == CUT_HERE ==
>> diff --git a/mm/vmscan.c b/mm/vmscan.c
>> index f73b865..cc23f04 100644
>> --- a/mm/vmscan.c
>> +++ b/mm/vmscan.c
>> @@ -1341,10 +1341,6 @@ static inline bool
>> should_reclaim_stall(unsigned long nr_taken,
>>        if (current_is_kswapd())
>>                return false;
>>
>> -       /* Only stall on lumpy reclaim */
>> -       if (sc->reclaim_mode & RECLAIM_MODE_SINGLE)
>> -               return false;
>> -
>>        /* If we have relaimed everything on the isolated list, no stall */
>>        if (nr_freed == nr_taken)
>>                return false;
>>
>>
>>
>> Then, if you don't see any unnecessary OOM but still see the hangup,
>> could you apply this patch based on previous?
>
> With this patch, I started GNOME and Firefox, turned on swap, and ran
> test_mempressure.sh 1500 1400 1.  Instant panic (or OOPS and hang or
> something -- didn't get the top part).  Picture attached -- it looks
> like memcg might be involved.  I'm running F15, so it might even be
> doing something.

I cannot figure out why happens OOPS.
Let me know your kernel version and config.
Kame. Is there anything related to memcg you guess?

In addition, the patch I give was utterly stupid.
The goal is that we wait dirty page writeback in (order-0 | high
priority) reclaim.
(But I don't think it's ideal solution in this problem but just for
proving the problem)
But although we pass sync with 1 in set_reclaim_mode, it ignores.
So fix is following as. (NOTICE: It doesn't related to your OOPS. )
But before further experiment, let's fix your oops.

diff --git a/mm/vmscan.c b/mm/vmscan.c
index 292582c..69d317e 100644
--- a/mm/vmscan.c
+++ b/mm/vmscan.c
@@ -311,7 +311,8 @@ static void set_reclaim_mode(int priority, struct
scan_control *sc,
         */
        if (sc->order > PAGE_ALLOC_COSTLY_ORDER)
                sc->reclaim_mode |= syncmode;
-       else if (sc->order && priority < DEF_PRIORITY - 2)
+       else if ((sc->order && priority < DEF_PRIORITY - 2) ||
+                               prioiry <= DEF_PRIORITY / 3)
                sc->reclaim_mode |= syncmode;
        else
                sc->reclaim_mode = RECLAIM_MODE_SINGLE | RECLAIM_MODE_ASYNC;
@@ -1349,10 +1350,6 @@ static inline bool
should_reclaim_stall(unsigned long nr_taken,
        if (current_is_kswapd())
                return false;

-       /* Only stall on lumpy reclaim */
-       if (sc->reclaim_mode & RECLAIM_MODE_SINGLE)
-               return false;
-
        /* If we have relaimed everything on the isolated list, no stall */
        if (nr_freed == nr_taken)
                return false;



>
> I won't be able to get netconsole dumps until next week because I'm
> out of town and only have this one computer here.

No problem. :)
We should avoid OOPS for the experiment.


>
> I haven't tried the other patch.
>
> Also, the !pgdat_balanced fix plus the if (need_resched()) return
> false patch just hung once on 2.6.37-rc9.  I don't know what triggered

Thanks for the good information.
It seems need_resched patch isn't good candidate to fix current problem.
We already weeded it out.

Thank you very much for the testing!

> it.  Maybe yum.
>
> --Andy
>



-- 
Kind regards,
Minchan Kim
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Powered by blists - more mailing lists