linux-kernel - Re: hugetlb page migration vs. overcommit

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite for Android: free password hash cracker in your pocket

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Date:   Mon, 27 Nov 2017 06:27:36 +0000
From:   Naoya Horiguchi <n-horiguchi@...jp.nec.com>
To:     Mike Kravetz <mike.kravetz@...cle.com>,
        Michal Hocko <mhocko@...nel.org>,
        "linux-mm@...ck.org" <linux-mm@...ck.org>
CC:     Naoya Horiguchi <n-horiguchi@...jp.nec.com>,
        LKML <linux-kernel@...r.kernel.org>
Subject: Re: hugetlb page migration vs. overcommit

On 11/23/2017 04:11 AM, Mike Kravetz wrote:
> On 11/22/2017 07:28 AM, Michal Hocko wrote:
>> Hi,
>> is there any reason why we enforce the overcommit limit during hugetlb
>> pages migration? It's in alloc_huge_page_node->__alloc_buddy_huge_page
>> path. I am wondering whether this is really an intentional behavior.
> 
> I do not think it was intentional.  But, I was not around when that
> code was added.
> 
>> The page migration allocates a page just temporarily so we should be
>> able to go over the overcommit limit for the migration duration. The
>> reason I am asking is that hugetlb pages tend to be utilized usually
>> (otherwise the memory would be just wasted and pool shrunk) but then
>> the migration simply fails which breaks memory hotplug and other
>> migration dependent functionality which is quite suboptimal. You can
>> workaround that by increasing the overcommit limit.
> 
> Yes.  In an environment making optimal use of huge pages, you are unlikely
> to have 'spare pages' set aside for a potential migration operation.  So
> I agree that it would make sense to try and allocate overcommit pages for
> this purpose.

Thank you for pointing this out, Michal, Mike.
Doing overcommitting in hugepage migration is totally right to me,
I just didn't notice it when I wrote the code.

> 
>> Why don't we simply migrate as long as we are able to allocate the
>> target hugetlb page? I have a half baked patch to remove this
>> restriction, would there be an opposition to do something like that?
> 
> I would not be opposed and would help with this effort.  My concern would
> be any subtle hugetlb accounting issues once you start messing with
> additional overcommit pages.

Yes, hugetlb accounting always needs care when touching related code.
I can help testing.

Thanks,
Naoya Horiguchi