linux-kernel - Re: [RFC PATCH 00/16] 1GB THP support on x86

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <20200903171847.GJ1152540@nvidia.com>
Date:   Thu, 3 Sep 2020 14:18:47 -0300
From:   Jason Gunthorpe <jgg@...dia.com>
To:     Matthew Wilcox <willy@...radead.org>
CC:     Roman Gushchin <guro@...com>, Michal Hocko <mhocko@...e.com>,
        Zi Yan <ziy@...dia.com>, <linux-mm@...ck.org>,
        Rik van Riel <riel@...riel.com>,
        "Kirill A . Shutemov" <kirill.shutemov@...ux.intel.com>,
        Shakeel Butt <shakeelb@...gle.com>,
        Yang Shi <yang.shi@...ux.alibaba.com>,
        David Nellans <dnellans@...dia.com>,
        <linux-kernel@...r.kernel.org>
Subject: Re: [RFC PATCH 00/16] 1GB THP support on x86_64

On Thu, Sep 03, 2020 at 06:01:57PM +0100, Matthew Wilcox wrote:
> On Thu, Sep 03, 2020 at 01:50:51PM -0300, Jason Gunthorpe wrote:
> > At least from a RDMA NIC perspective I've heard from a lot of users
> > that higher order pages at the DMA level is giving big speed ups too.
> > 
> > It is basically the same dynamic as CPU TLB, except missing a 'TLB'
> > cache in a PCI-E device is dramatically more expensive to refill. With
> > 200G and soon 400G networking these misses are a growing problem.
> > 
> > With HPC nodes now pushing 1TB of actual physical RAM and single
> > applications basically using all of it, there is definately some
> > meaningful return - if pages can be reliably available.
> > 
> > At least for HPC where the node returns to an idle state after each
> > job and most of the 1TB memory becomes freed up again, it seems more
> > believable to me that a large cache of 1G pages could be available?
> 
> You may be interested in trying out my current THP patchset:
> 
> http://git.infradead.org/users/willy/pagecache.git
> 
> It doesn't allocate pages larger than PMD size, but it does allocate pages
> *up to* PMD size for the page cache which means that larger pages are
> easier to create as larger pages aren't fragmented all over the system.

Yeah, I saw that, it looks like a great direction.

> If someone wants to opportunistically allocate pages larger than PMD
> size, I've put some preliminary support in for that, but I've never
> tested any of it.  That's not my goal at the moment.
> 
> I'm not clear whether these HPC users primarily use page cache or
> anonymous memory (with O_DIRECT).  Probably a mixture.

There are defiantly HPC systems now that are filesystem-less - they
import data for computation from the network using things like blob
storage or some other kind of non-POSIX userspace based data storage
scheme.

Jason