linux-kernel - Re: [PATCH v2] IB/mlx5: Reduce max order of memory allocated for xlt update

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-Id: <0DFF7518-8818-445B-94AC-8EB2096446BE@oracle.com>
Date:   Tue, 23 Mar 2021 21:27:38 -0700
From:   Aruna Ramakrishna <aruna.ramakrishna@...cle.com>
To:     Jason Gunthorpe <jgg@...pe.ca>
Cc:     Praveen Kumar Kannoju <praveen.kannoju@...cle.com>,
        leon@...nel.org, dledford@...hat.com, linux-rdma@...r.kernel.org,
        linux-kernel@...r.kernel.org,
        Rajesh Sivaramasubramaniom 
        <rajesh.sivaramasubramaniom@...cle.com>,
        Rama Nichanamatlu <rama.nichanamatlu@...cle.com>,
        Jeffery Yoder <jeffery.yoder@...cle.com>
Subject: Re: [PATCH v2] IB/mlx5: Reduce max order of memory allocated for xlt
 update


> On Mar 23, 2021, at 4:13 PM, Jason Gunthorpe <jgg@...pe.ca> wrote:
> 
> On Tue, Mar 23, 2021 at 12:41:51PM -0700, Aruna Ramakrishna wrote:
>>   There is a far greater possibility of an order-8 allocation failing,
>>   esp. with the addition of __GFP_NORETRY , and the code would have to
>>   fall back to a lower order allocation more often than not (esp. on a
>>   long running system). Unless the performance gains from using order-8
>>   pages is significant (and it does not seem that way to me), we can just
>>   skip this step and directly go to the lower order allocation.
> 
> Do not send HTML mails.

I apologize; I’ve fixed the setting now.

> 
> Do you have benchmarks that show the performance of the high order
> pages is not relavent? I'm a bit surprised to hear that
> 

I guess my point was more to the effect that an order-8 alloc will fail more often than not, in this flow. For instance, when we were debugging the latency spikes here, this was the typical buddyinfo output on that system:

Node 0, zone      DMA      0      1      1      2      3      0      1      0      1      1      3 
Node 0, zone    DMA32      7      7      7      6     10      2      6      7      6      2    306 
Node 0, zone   Normal   3390  51354  17574   6556   1586     26      2      1      0      0      0 
Node 1, zone   Normal  11519  23315  23306   9738     73      2      0      1      0      0      0 

I think this level of fragmentation is pretty normal on long running systems. Here, in the reg_mr flow, the first try (order-8) alloc will probably fail 9 times out of 10 (esp. after the addition of GFP_NORETRY flag), and then as fallback, the code tries to allocate a lower order, and if that too fails, it allocates a page. I think it makes sense to just avoid trying an order-8 alloc here.

Thanks,
Aruna


> This code really needs some attention to use a proper
> scatter/gather. I understand the chip can do it, just some of the
> software layers need to be stripped away so it can form the right SGL
> in the HW.
> 
> Jason