linux-kernel - Re: [PATCH rdma-next v1 0/2] Enable relaxed ordering for ULPs

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <20210526193021.GA3644646@nvidia.com>
Date:   Wed, 26 May 2021 16:30:21 -0300
From:   Jason Gunthorpe <jgg@...dia.com>
To:     Leon Romanovsky <leon@...nel.org>
Cc:     Doug Ledford <dledford@...hat.com>,
        Leon Romanovsky <leonro@...dia.com>,
        Avihai Horon <avihaih@...dia.com>,
        linux-kernel@...r.kernel.org, linux-rdma@...r.kernel.org,
        Christoph Hellwig <hch@....de>,
        Bart Van Assche <bvanassche@....org>,
        Tom Talpey <tom@...pey.com>,
        Santosh Shilimkar <santosh.shilimkar@...cle.com>,
        Chuck Lever III <chuck.lever@...cle.com>,
        Keith Busch <kbusch@...nel.org>,
        David Laight <David.Laight@...LAB.COM>,
        Honggang LI <honli@...hat.com>,
        Max Gurtovoy <mgurtovoy@...dia.com>
Subject: Re: [PATCH rdma-next v1 0/2] Enable relaxed ordering for ULPs

On Thu, May 20, 2021 at 01:13:34PM +0300, Leon Romanovsky wrote:
> From: Leon Romanovsky <leonro@...dia.com>
> 
> Changelog:
> v1:
>  * Enabled by default RO in IB/core instead of changing all users
> v0: https://lore.kernel.org/lkml/20210405052404.213889-1-leon@kernel.org
> 
> >From Avihai,
> 
> Relaxed Ordering is a PCIe mechanism that relaxes the strict ordering
> imposed on PCI transactions, and thus, can improve performance for
> applications that can handle this lack of strict ordering.
> 
> Currently, relaxed ordering can be set only by user space applications
> for user MRs. Not all user space applications support relaxed ordering
> and for this reason it was added as an optional capability that is
> disabled by default. This behavior is not changed as part of this series,
> and relaxed ordering remains disabled by default for user space.
> 
> On the other hand, kernel users should universally support relaxed
> ordering, as they are designed to read data only after observing the CQE
> and use the DMA API correctly. There are a few platforms with broken
> relaxed ordering implementation, but for them relaxed ordering is expected
> to be turned off globally in the PCI level. In addition, note that this is
> not the first use of relaxed ordering. Relaxed ordering has been enabled
> by default in mlx5 ethernet driver, and user space apps use it as well for
> quite a while.
> 
> Hence, this series enabled relaxed ordering by default for kernel users so
> they can benefit as well from the performance improvements.
> 
> The following test results show the performance improvement achieved
> with relaxed ordering. The test was performed by running FIO traffic
> between a NVIDIA DGX A100 (ConnectX-6 NICs and AMD CPUs) and a NVMe
> storage fabric, using NFSoRDMA:
> 
> Without Relaxed Ordering:
> READ: bw=16.5GiB/s (17.7GB/s), 16.5GiB/s-16.5GiB/s (17.7GB/s-17.7GB/s),
> io=1987GiB (2133GB), run=120422-120422msec
> 
> With relaxed ordering:
> READ: bw=72.9GiB/s (78.2GB/s), 72.9GiB/s-72.9GiB/s (78.2GB/s-78.2GB/s),
> io=2367GiB (2542GB), run=32492-32492msec
> 
> The series has been tested over NVMe, iSER, SRP and NFS with ConnectX-6
> NIC. The tests included FIO verify and stress tests, and various
> resiliency tests (shutting down NIC port in the middle of traffic,
> rebooting the target in the middle of traffic etc.).

There was such a big discussion on the last version I wondered why
this was so quiet. I guess because the cc list isn't very big..

Adding the people from the original thread, here is the patches:

https://lore.kernel.org/linux-rdma/cover.1621505111.git.leonro@nvidia.com/

I think this is the general approach that was asked for, to special case
uverbs and turn it on in kernel universally
 
Jason