linux-kernel - Re: s2disk hang update

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives
Hash Suite: Windows password security audit tool. GUI, reports in PDF.
[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]
Message-ID: <4B82A440.6010701@googlemail.com>
Date:	Mon, 22 Feb 2010 15:35:28 +0000
From:	Alan Jenkins <sourcejedi.lkml@...glemail.com>
To:	"Rafael J. Wysocki" <rjw@...k.pl>
CC:	Mel Gorman <mel@....ul.ie>, hugh.dickins@...cali.co.uk,
	Pavel Machek <pavel@....cz>,
	pm list <linux-pm@...ts.linux-foundation.org>,
	linux-kernel <linux-kernel@...r.kernel.org>,
	Kernel Testers List <kernel-testers@...r.kernel.org>
Subject: Re: s2disk hang update

Rafael J. Wysocki wrote:
> On Friday 19 February 2010, Alan Jenkins wrote:
>   
>> On 2/18/10, Rafael J. Wysocki <rjw@...k.pl> wrote:
>>     
>>> On Thursday 18 February 2010, Alan Jenkins wrote:
>>>       
>>>> On 2/17/10, Rafael J. Wysocki <rjw@...k.pl> wrote:
>>>>         
>>>>> On Wednesday 17 February 2010, Alan Jenkins wrote:
>>>>>           
>>>>>> On 2/16/10, Rafael J. Wysocki <rjw@...k.pl> wrote:
>>>>>>             
>>>>>>> On Tuesday 16 February 2010, Alan Jenkins wrote:
>>>>>>>               
>>>>>>>> On 2/16/10, Alan Jenkins <sourcejedi.lkml@...glemail.com> wrote:
>>>>>>>>                 
>>>>>>>>> On 2/15/10, Rafael J. Wysocki <rjw@...k.pl> wrote:
>>>>>>>>>                   
>>>>>>>>>> On Tuesday 09 February 2010, Alan Jenkins wrote:
>>>>>>>>>>                     
>>>>>>>>>>> Perhaps I spoke too soon.  I see the same hang if I run too many
>>>>>>>>>>> applications.  The first hibernation fails with "not enough
>>>>>>>>>>> swap"
>>>>>>>>>>> as
>>>>>>>>>>> expected, but the second or third attempt hangs (with the same
>>>>>>>>>>> backtrace
>>>>>>>>>>> as before).
>>>>>>>>>>>
>>>>>>>>>>> The patch definitely helps though.  Without the patch, I see a
>>>>>>>>>>> hang
>>>>>>>>>>> the
>>>>>>>>>>> first time I try to hibernate with too many applications
>>>>>>>>>>> running.
>>>>>>>>>>>                       
>>>>>>>>>> Well, I have an idea.
>>>>>>>>>>
>>>>>>>>>> Can you try to apply the appended patch in addition and see if
>>>>>>>>>> that
>>>>>>>>>> helps?
>>>>>>>>>>
>>>>>>>>>> Rafael
>>>>>>>>>>                     
>>>>>>>>> It doesn't seem to help.
>>>>>>>>>                   
>>>>>>>> To be clear: It doesn't stop the hang when I hibernate with too many
>>>>>>>> applications.
>>>>>>>>
>>>>>>>> It does stop the same hang in a different case though.
>>>>>>>>
>>>>>>>> 1. boot with init=/bin/bash
>>>>>>>> 2. run s2disk
>>>>>>>> 3. cancel the s2disk
>>>>>>>> 4. repeat steps 2&3
>>>>>>>>
>>>>>>>> With the patch, I can run 10s of iterations, with no hang.
>>>>>>>> Without the patch, it soon hangs, (in disable_nonboot_cpus(), as
>>>>>>>> always).
>>>>>>>>
>>>>>>>> That's what happens on 2.6.33-rc7.  On 2.6.30, there is no problem.
>>>>>>>> On 2.6.31 and 2.6.32 I don't get a hang, but dmesg shows an
>>>>>>>> allocation
>>>>>>>> failure after a couple of iterations ("kthreadd: page allocation
>>>>>>>> failure. order:1, mode:0xd0").  It looks like it might be the same
>>>>>>>> stop_machine thread allocation failure that causes the hang.
>>>>>>>>                 
>>>>>>> Have you tested it alone or on top of the previous one?  If you've
>>>>>>> tested it
>>>>>>> alone, please apply the appended one in addition to it and retest.
>>>>>>>
>>>>>>> Rafael
>>>>>>>               
>>>>>> I did test with both patches applied together -
>>>>>>
>>>>>> 1. [Update] MM / PM: Force GFP_NOIO during suspend/hibernation and
>>>>>> resume
>>>>>> 2. "reducing the number of pages that we're going to keep preallocated
>>>>>> by
>>>>>> 20%"
>>>>>>             
>>>>> In that case you can try to reduce the number of preallocated pages even
>>>>> more,
>>>>> ie. change "/ 5" to "/ 2" (for example) in the second patch.
>>>>>           
>>>> It still hangs if I try to hibernate a couple of times with too many
>>>> applications.
>>>>         
>>> Hmm.  I guess I asked that before, but is this a 32-bit or 64-bit system and
>>> how much RAM is there in the box?
>>>
>>> Rafael
>>>       
>> EeePC 701.  32 bit.  512Mb RAM.  350Mb swap file, on a "first-gen" SSD.
>>     
>
> Hmm.  I'd try to make  free_unnecessary_pages() free all of the preallocated
> pages and see what happens.
>   

It still hangs in hibernation_snapshot() / disable_nonboot_cpus().  
After apparently freeing over 400Mb / 100,000 pages of preallocated ram.



There is a change which I missed before.  When I applied your first 
patch ("Force GFP_NOIO during suspend" etc.), it did change the hung 
task backtraces a bit.  I don't know if it tells us anything.

Without the patch, there were two backtraces.  The first backtrace 
suggested a problem allocating pages for a kernel thread (at 
copy_process() / try_to_free_pages()).  The second showed that this 
problem was blocking s2disk (at hibernation_snapshot() / 
disable_nonboot_cpus() / stop_machine_create()). 

With the GFP_NOIO patch, I see only the s2disk backtrace.


Thanks
Alan

View attachment "free-all-prealloc.patch" of type "text/x-diff" (576 bytes)