linux-kernel - Re: khugepaged eating 100%CPU

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [day] [month] [year] [list]

Message-ID: <20110208084557.GC28138@tiehlicka.suse.cz>
Date:	Tue, 8 Feb 2011 09:45:57 +0100
From:	Michal Hocko <mhocko@...e.cz>
To:	Andrea Arcangeli <aarcange@...hat.com>
Cc:	LKML <linux-kernel@...r.kernel.org>, linux-mm@...ck.org
Subject: Re: khugepaged eating 100%CPU

On Tue 08-02-11 00:12:28, Andrea Arcangeli wrote:
> Hello Michal,
> 
> On Mon, Feb 07, 2011 at 10:16:01PM +0100, Michal Hocko wrote:
> > On Mon 07-02-11 22:06:54, Michal Hocko wrote:
> > > Hi Andrea,
> > > 
> > > I am currently running into an issue when khugepaged is running 100% on
> > > one of my CPUs for a long time (at least one hour as I am writing the
> > > email). The kernel is the clean 2.6.38-rc3 (i386) vanilla kernel.
> > > 
> > > I have tried to disable defrag but it didn't help (I haven't rebooted
> > > after setting the value). I am not sure what information is helpful and
> > > also not sure whether I am able to reproduce it after restart (it is the
> > > first time I can see this problem) so sorry for the poor report.
> > > 
> > > Here is some basic info which might be useful (config and sysrq+t are
> > > attached):
> > > =========
> > 
> > And I have just realized that I forgot about the daemon stack:
> > # cat /proc/573/stack 
> > [<c019c981>] shrink_zone+0x1b9/0x455
> > [<c019d462>] do_try_to_free_pages+0x9d/0x301
> > [<c019d803>] try_to_free_pages+0xb3/0x104
> > [<c01966d7>] __alloc_pages_nodemask+0x358/0x589
> > [<c01bf314>] khugepaged+0x13f/0xc60
> > [<c014c301>] kthread+0x67/0x6c
> > [<c0102db6>] kernel_thread_helper+0x6/0x10
> > [<ffffffff>] 0xffffffff
> 
> It would be great to know if __alloc_pages_nodemask returned or if it
> was calling it in a loop.
> 
> When __alloc_pages_nodemask fails in collapse_huge_page, hpage is set
> to ERR_PTR(-ENOMEM), then khugepaged_scan_pmd returns 1, then
> khugepaged_scan_mm_slot goto breakouterloop_mmap_sem and return
> progress, then the khugepaged_do_scan main loop should notice that
> IS_ERR(*hpage) is set and break out of the loop and return void, then
> khugepaged_loop should notice that IS_ERR(hpage) is set and it should
> throttle for alloc_sleep_millisecs inside khugepaged_alloc_sleep
> before setting hpage to NULL and trying again to allocate. I wonder
> what could be going wrong in khugepaged.. I wonder if it's a bug inside
> __alloc_pages_nodemask and not a khugepaged issue. Best would be if
> you run SYSRQ+l several times.

OK, I will try if I see it again.

> 
> I hope you can reproduce, if it's an allocator issue you should notice
> it again by keeping the same workload on that same system. I doubt I
> can reproduce at the moment as I don't know what's going on to
> simulate your load.

My workload is rather "normal", I would say. Firefox with couple of
tabs, skype, mutt, xine (wathing the stream television), kernel builds)
and repeated suspend/wakeup cycles (I am rebooting only when installing
a new kernel - aka new rc is released or I need to test something). So
it is hard to find out what triggered this situation.

I will let you know when I get into the same situation and provide the
sysrq+l.

Thanks
-- 
Michal Hocko
SUSE Labs
SUSE LINUX s.r.o.
Lihovarska 1060/12
190 00 Praha 9    
Czech Republic
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/