| lists.openwall.net | lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC | |
|
Open Source and information security mailing list archives
| ||
|
Message-ID: <9b2b86521002230624g20661564mc35093ee0423ff77@mail.gmail.com>
Date: Tue, 23 Feb 2010 14:24:25 +0000
From: Alan Jenkins <sourcejedi.lkml@...glemail.com>
To: "Rafael J. Wysocki" <rjw@...k.pl>
Cc: Mel Gorman <mel@....ul.ie>, hugh.dickins@...cali.co.uk,
Pavel Machek <pavel@....cz>,
pm list <linux-pm@...ts.linux-foundation.org>,
linux-kernel <linux-kernel@...r.kernel.org>,
Kernel Testers List <kernel-testers@...r.kernel.org>
Subject: Re: s2disk hang update
On 2/22/10, Rafael J. Wysocki <rjw@...k.pl> wrote:
> On Monday 22 February 2010, Alan Jenkins wrote:
>> Rafael J. Wysocki wrote:
>> > On Friday 19 February 2010, Alan Jenkins wrote:
>> >
>> >> On 2/18/10, Rafael J. Wysocki <rjw@...k.pl> wrote:
>> >>
>> >>> On Thursday 18 February 2010, Alan Jenkins wrote:
>> >>>
>> >>>> On 2/17/10, Rafael J. Wysocki <rjw@...k.pl> wrote:
>> >>>>
>> >>>>> On Wednesday 17 February 2010, Alan Jenkins wrote:
>> >>>>>
>> >>>>>> On 2/16/10, Rafael J. Wysocki <rjw@...k.pl> wrote:
>> >>>>>>
>> >>>>>>> On Tuesday 16 February 2010, Alan Jenkins wrote:
>> >>>>>>>
>> >>>>>>>> On 2/16/10, Alan Jenkins <sourcejedi.lkml@...glemail.com> wrote:
>> >>>>>>>>
>> >>>>>>>>> On 2/15/10, Rafael J. Wysocki <rjw@...k.pl> wrote:
>> >>>>>>>>>
>> >>>>>>>>>> On Tuesday 09 February 2010, Alan Jenkins wrote:
>> >>>>>>>>>>
>> >>>>>>>>>>> Perhaps I spoke too soon. I see the same hang if I run too
>> >>>>>>>>>>> many
>> >>>>>>>>>>> applications. The first hibernation fails with "not enough
>> >>>>>>>>>>> swap"
>> >>>>>>>>>>> as
>> >>>>>>>>>>> expected, but the second or third attempt hangs (with the same
>> >>>>>>>>>>> backtrace
>> >>>>>>>>>>> as before).
>> >>>>>>>>>>>
>> >>>>>>>>>>> The patch definitely helps though. Without the patch, I see a
>> >>>>>>>>>>> hang
>> >>>>>>>>>>> the
>> >>>>>>>>>>> first time I try to hibernate with too many applications
>> >>>>>>>>>>> running.
>> >>>>>>>>>>>
>> >>>>>>>>>> Well, I have an idea.
>> >>>>>>>>>>
>> >>>>>>>>>> Can you try to apply the appended patch in addition and see if
>> >>>>>>>>>> that
>> >>>>>>>>>> helps?
>> >>>>>>>>>>
>> >>>>>>>>>> Rafael
>> >>>>>>>>>>
>> >>>>>>>>> It doesn't seem to help.
>> >>>>>>>>>
>> >>>>>>>> To be clear: It doesn't stop the hang when I hibernate with too
>> >>>>>>>> many
>> >>>>>>>> applications.
>> >>>>>>>>
>> >>>>>>>> It does stop the same hang in a different case though.
>> >>>>>>>>
>> >>>>>>>> 1. boot with init=/bin/bash
>> >>>>>>>> 2. run s2disk
>> >>>>>>>> 3. cancel the s2disk
>> >>>>>>>> 4. repeat steps 2&3
>> >>>>>>>>
>> >>>>>>>> With the patch, I can run 10s of iterations, with no hang.
>> >>>>>>>> Without the patch, it soon hangs, (in disable_nonboot_cpus(), as
>> >>>>>>>> always).
>> >>>>>>>>
>> >>>>>>>> That's what happens on 2.6.33-rc7. On 2.6.30, there is no
>> >>>>>>>> problem.
>> >>>>>>>> On 2.6.31 and 2.6.32 I don't get a hang, but dmesg shows an
>> >>>>>>>> allocation
>> >>>>>>>> failure after a couple of iterations ("kthreadd: page allocation
>> >>>>>>>> failure. order:1, mode:0xd0"). It looks like it might be the
>> >>>>>>>> same
>> >>>>>>>> stop_machine thread allocation failure that causes the hang.
>> >>>>>>>>
>> >>>>>>> Have you tested it alone or on top of the previous one? If you've
>> >>>>>>> tested it
>> >>>>>>> alone, please apply the appended one in addition to it and retest.
>> >>>>>>>
>> >>>>>>> Rafael
>> >>>>>>>
>> >>>>>> I did test with both patches applied together -
>> >>>>>>
>> >>>>>> 1. [Update] MM / PM: Force GFP_NOIO during suspend/hibernation and
>> >>>>>> resume
>> >>>>>> 2. "reducing the number of pages that we're going to keep
>> >>>>>> preallocated
>> >>>>>> by
>> >>>>>> 20%"
>> >>>>>>
>> >>>>> In that case you can try to reduce the number of preallocated pages
>> >>>>> even
>> >>>>> more,
>> >>>>> ie. change "/ 5" to "/ 2" (for example) in the second patch.
>> >>>>>
>> >>>> It still hangs if I try to hibernate a couple of times with too many
>> >>>> applications.
>> >>>>
>> >>> Hmm. I guess I asked that before, but is this a 32-bit or 64-bit
>> >>> system and
>> >>> how much RAM is there in the box?
>> >>>
>> >>> Rafael
>> >>>
>> >> EeePC 701. 32 bit. 512Mb RAM. 350Mb swap file, on a "first-gen" SSD.
>> >>
>> >
>> > Hmm. I'd try to make free_unnecessary_pages() free all of the
>> > preallocated
>> > pages and see what happens.
>> >
>>
>> It still hangs in hibernation_snapshot() / disable_nonboot_cpus().
>> After apparently freeing over 400Mb / 100,000 pages of preallocated ram.
>>
>>
>>
>> There is a change which I missed before. When I applied your first
>> patch ("Force GFP_NOIO during suspend" etc.), it did change the hung
>> task backtraces a bit. I don't know if it tells us anything.
>>
>> Without the patch, there were two backtraces. The first backtrace
>> suggested a problem allocating pages for a kernel thread (at
>> copy_process() / try_to_free_pages()). The second showed that this
>> problem was blocking s2disk (at hibernation_snapshot() /
>> disable_nonboot_cpus() / stop_machine_create()).
>>
>> With the GFP_NOIO patch, I see only the s2disk backtrace.
>
> Can you please post this backtrace?
Sure. It's rather like the one I posted before, except
a) it only shows the one hung task (s2disk)
b) this time I had lockdep enabled
c) this time most of the lines don't have question marks.
Kernel verson:
- mainline v2.6.33-rc8-164-gaea187c, with the one patch "Force GFP_NOIO..."
Image:
http://picasaweb.google.com/lh/photo/f9KRZT2l9wCmVt-Jdggd9g?feat=directlink
INFO: task s2disk:1916 blocked for more than 120 seconds
...
Call Trace:
? _raw_spin_unlock
schedule_timeout+0x22 (timer.c:1366)
? mark_held_locks
? _raw_spin_unlock_irq
? trace_hardirqs_on_caller
? trace_hardirqs_on
wait_for_common+0xb8 (sched.c:5844)
? default_wake_function
wait_for_completion+0x12 (sched.c:5879)
kthread_create+0x75 (kthread.c:133)
? worker_thread+0x0
create_workqueue_thread+0x38 (workqueue.c:921)
? worker_thread+0x0
__create_workqueue_key+0x156 (workqueue.c:1006)
stop_machine_create+0x32 (stop_machine.c:121)
disable_nonboot_cpus+0xe (cpu.c:370)
hibernation_snapshot+0x94 (hibernate.c:266)
snapshot_ioctl+0x21b (user.c:256)
...
sys_ioctl+0x41 (ioctl.c:624)
Thanks
Alan
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
Powered by blists - more mailing lists