linux-kernel - Re: hugetlb oops on 3.1.0-rc8-devel

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [day] [month] [year] [list]

Date:	Tue, 6 Dec 2011 12:43:34 -0800
From:	Andy Lutomirski <luto@...capital.net>
To:	David Rientjes <rientjes@...gle.com>
Cc:	Hillf Danton <dhillf@...il.com>, linux-kernel@...r.kernel.org,
	linux-mm@...ck.org
Subject: Re: hugetlb oops on 3.1.0-rc8-devel

On Tue, Dec 6, 2011 at 12:16 PM, David Rientjes <rientjes@...gle.com> wrote:
> On Wed, 2 Nov 2011, Andy Lutomirski wrote:
>
>> > --- a/mm/hugetlb.c      Sat Aug 13 11:45:14 2011
>> > +++ b/mm/hugetlb.c      Wed Nov  2 20:12:00 2011
>> > @@ -2422,6 +2422,8 @@ retry_avoidcopy:
>> >         * anon_vma prepared.
>> >         */
>> >        if (unlikely(anon_vma_prepare(vma))) {
>> > +               page_cache_release(new_page);
>> > +               page_cache_release(old_page);
>> >                /* Caller expects lock to be held */
>> >                spin_lock(&mm->page_table_lock);
>> >                return VM_FAULT_OOM;
>> >
>>
>> I'll patch it in.  My test case took over a week to hit it once, so I
>> can't guarantee I'll spot it.
>>
>
> This patch was merged and released in 3.2-rc3 as ea4039a34c4c ("hugetlb:
> release pages in the error path of hugetlb_cow()"), Andy is this issue
> fixed for you?

I haven't seen it again with or without the patch.  I suspect that to
trigger it again I'd have to set up an old, buggy version of my
software to hammer on it for awhile, which I won't have a chance to do
any time soon.  Sorry.

If you're interested, the workload that triggered the problem was,
roughly, two programs.  Both were set up to use libhugetlbfs for
everything, and the first program spawned (presumably via fork as
opposed to any clone magic) copies of the second program frequently.
The second program was very memory intensive.  The result was that,
occasionally, fork had issues because it couldn't find free huge pages
in the pool.

--Andy

-- 
Andy Lutomirski
AMA Capital Management, LLC
Office: (310) 553-5322
Mobile: (650) 906-0647
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/