netdev - Re: [net-next 10/10] net/mlx5e: Add support for PCI relaxed ordering

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <20200623143118.51373eb7@kicinski-fedora-pc1c0hjn.dhcp.thefacebook.com>
Date:   Tue, 23 Jun 2020 14:31:18 -0700
From:   Jakub Kicinski <kuba@...nel.org>
To:     Saeed Mahameed <saeedm@...lanox.com>
Cc:     "David S. Miller" <davem@...emloft.net>, netdev@...r.kernel.org,
        Aya Levin <ayal@...lanox.com>,
        Tariq Toukan <tariqt@...lanox.com>,
        Michal Kubecek <mkubecek@...e.cz>
Subject: Re: [net-next 10/10] net/mlx5e: Add support for PCI relaxed
 ordering

On Tue, 23 Jun 2020 12:52:29 -0700 Saeed Mahameed wrote:
> From: Aya Levin <ayal@...lanox.com>
> 
> The concept of Relaxed Ordering in the PCI Express environment allows
> switches in the path between the Requester and Completer to reorder some
> transactions just received before others that were previously enqueued.
> 
> In ETH driver, there is no question of write integrity since each memory
> segment is written only once per cycle. In addition, the driver doesn't
> access the memory shared with the hardware until the corresponding CQE
> arrives indicating all PCI transactions are done.

Assuming the device sets the RO bits appropriately, right? Otherwise
CQE write could theoretically surpass the data write, no?

> With relaxed ordering set, traffic on the remote-numa is at the same
> level as when on the local numa.

Same level of? Achievable bandwidth?

> Running TCP single stream over ConnectX-4 LX, ARM CPU on remote-numa
> has 300% improvement in the bandwidth.
> With relaxed ordering turned off: BW:10 [GB/s]
> With relaxed ordering turned on:  BW:40 [GB/s]
> 
> The driver turns relaxed ordering off by default. It exposes 2 boolean
> private-flags in ethtool: pci_ro_read and pci_ro_write for user
> control.
> 
> $ ethtool --show-priv-flags eth2
> Private flags for eth2:
> ...
> pci_ro_read        : off
> pci_ro_write       : off
> 
> $ ethtool --set-priv-flags eth2 pci_ro_write on
> $ ethtool --set-priv-flags eth2 pci_ro_read on

I think Michal will rightly complain that this does not belong in
private flags any more. As (/if?) ARM deployments take a foothold 
in DC this will become a common setting for most NICs.