linux-kernel - Re: [PATCH rdma-next 4/4] RDMA/umem: Move to allocate SG table from pages

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <20200908141015.GF9166@nvidia.com>
Date:   Tue, 8 Sep 2020 11:10:15 -0300
From:   Jason Gunthorpe <jgg@...dia.com>
To:     Christoph Hellwig <hch@....de>
CC:     Leon Romanovsky <leon@...nel.org>,
        Doug Ledford <dledford@...hat.com>,
        Maor Gottlieb <maorg@...dia.com>,
        <linux-kernel@...r.kernel.org>, <linux-rdma@...r.kernel.org>
Subject: Re: [PATCH rdma-next 4/4] RDMA/umem: Move to allocate SG table from
 pages

On Mon, Sep 07, 2020 at 09:29:26AM +0200, Christoph Hellwig wrote:
> On Thu, Sep 03, 2020 at 03:18:53PM +0300, Leon Romanovsky wrote:
> > From: Maor Gottlieb <maorg@...dia.com>
> > 
> > Remove the implementation of ib_umem_add_sg_table and instead
> > call to sg_alloc_table_append which already has the logic to
> > merge contiguous pages.
> > 
> > Besides that it removes duplicated functionality, it reduces the
> > memory consumption of the SG table significantly. Prior to this
> > patch, the SG table was allocated in advance regardless consideration
> > of contiguous pages.
> > 
> > In huge pages system of 2MB page size, without this change, the SG table
> > would contain x512 SG entries.
> > E.g. for 100GB memory registration:
> > 
> > 	 Number of entries	Size
> > Before 	      26214400          600.0MB
> > After            51200		  1.2MB
> > 
> > Signed-off-by: Maor Gottlieb <maorg@...dia.com>
> > Signed-off-by: Leon Romanovsky <leonro@...dia.com>
> 
> Looks sensible for now, but the real fix is of course to avoid
> the scatterlist here entirely, and provide a bvec based
> pin_user_pages_fast.  I'll need to finally get that done..

I'm working on cleaning all the DMA RDMA drivers using ib_umem to the
point where doing something like this would become fairly simple.

pin_user_pages_fast_bvec/whatever would be a huge improvement here,
calling in a loop like this just to get a partial page list to copy to
a SGL is horrificly slow due to all the extra overheads. Going
directly to the bvec/sgl/etc inside all the locks will be a lot faster

Jason