linux-kernel - Re: scalability regressions related to hugetlb

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <1883d31a-639e-8717-39b1-426628cb0d56@oracle.com>
Date:   Thu, 24 Mar 2022 21:40:05 -0700
From:   Mike Kravetz <mike.kravetz@...cle.com>
To:     Ray Fucillo <Ray.Fucillo@...ersystems.com>
Cc:     "linux-kernel@...r.kernel.org" <linux-kernel@...r.kernel.org>,
        linux-mm <linux-mm@...ck.org>
Subject: Re: scalability regressions related to hugetlb_fault() changes

On 3/24/22 17:02, Ray Fucillo wrote:
> 
>> On Mar 24, 2022, at 6:41 PM, Mike Kravetz <mike.kravetz@...cle.com> wrote:
>>
>> I also seem to remember thinking about the possibility of
>> avoiding the synchronization if pmd sharing was not possible.  That may be
>> a relatively easy way to speed things up.  Not sure if pmd sharing comes
>> into play in your customer environments, my guess would be yes (shared
>> mappings ranges more than 1GB in size and aligned to 1GB).
> 
> Hi Mike, 
> 
> This is one very large shared memory segment allocated at database startup.  It's common for it to be hundreds of GB.  We allocate it with shmget() passing SHM_HUGETLB (when huge pages have been reserved for us).  Not sure if that answers...

Yes, so there would be shared pmds for that large shared mapping.  I assume
this is x86 or arm64 which are the only architectures which support shared
pmds.

So, the easy change of "don't take semaphore if pmd sharing is not possible"
would not apply.

>> Also, do you have any specifics about the regressions your customers are
>> seeing?  Specifically what paths are holding i_mmap_rwsem in write mode
>> for long periods of time.  I would expect something related to unmap.
>> Truncation can have long hold times especially if there are may shared
>> mapping.  Always worth checking specifics, but more likely this is a general
>> issue.
> 
> We've seen the write lock originate from calling shmat(), shmdt() and process exit.  We've also seen it from a fork() off of one of the processes that are attached to the shared memory segment.  Some evidence suggests that fork is a more costly case.  However, while there are some important places where we'd use fork(), it's more unusual because most process creation will vfork() and execv() a new database process (which then attaches with shmat()).

Thanks.

I will continue to look at this.  A quick check of the fork code shows the
semaphore held in read mode for the duration of the page table copy.
-- 
Mike Kravetz