netdev - Re: [PATCH net-next 2/2] net/mlx4_core: Disable BF when write combining is not available

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Date:	Thu, 2 Oct 2014 17:37:58 +0300
From:	Or Gerlitz <ogerlitz@...lanox.com>
To:	Alexei Starovoitov <alexei.starovoitov@...il.com>
CC:	"David S. Miller" <davem@...emloft.net>,
	"netdev@...r.kernel.org" <netdev@...r.kernel.org>,
	Amir Vadai <amirv@...lanox.com>,
	Jack Morgenstein <jackm@....mellanox.co.il>,
	Moshe Lazer <moshel@...lanox.com>,
	Tal Alon <talal@...lanox.com>,
	Yevgeny Petrilin <yevgenyp@...lanox.com>,
	Amir Ancel <amira@...lanox.com>
Subject: Re: [PATCH net-next 2/2] net/mlx4_core: Disable BF when write combining
 is not available

On 10/1/2014 7:52 PM, Alexei Starovoitov wrote:
> On Wed, Oct 1, 2014 at 7:54 AM, Or Gerlitz <ogerlitz@...lanox.com> wrote:
>> From: Moshe Lazer <moshel@...lanox.com>
>>
>> In mlx4 for better latency, we write send descriptors to a write-combining
>> (WC) mapped buffer instead of ringing a doorbell and having the HW fetch
>> the descriptor from system memory.
>>
>> However, if write-combining is not supported on the host, then we
>> obtain better latency by using the doorbell-ring/HW fetch mechanism.
>>
>> The mechanism that uses WC is called Blue-Flame (BF). BF is beneficial
>> only when the system supports write combining. When the BF buffer is
>> mapped as a write-combine buffer, the HCA receives data in multi-word
>> bursts. However, if the BF buffer is mapped only as non-cached, the
>> HCA receives data in individual dword chunks, which harms performance.
>>
>> Therefore, disable blueflame when write combining is not available.
> curious, what numbers you're seeing:
> - [1] bf=on with wc
> - [2] bf=on without wc
> - [3] bf=off and doorbell
> they will help to justify this change.

Sure, see below:

The 1st set of results was obtained from running latency test
with the HCA being passthrough-ed into VM running over KVM
host -- so WC isn't available.

The problematic range is 32-128B, for example with 128 bytes
message, using BF has latency of 1.47us and no usage of BF
only 1us. When WC isn't really available every write of 64B
would actually translate into 8 writes of 8 bytes which obviously
hurts the latency.

# /usr/bin/taskset -c 0 ib_write_lat -d mlx4_0 -i 1  -F -a -n 1000000

[2] BF on without WC
  #bytes #iterations    t_min[usec]    t_max[usec] t_typical[usec]
  2       1000000          0.74           186.16       0.79
  4       1000000          0.70           103.62       0.78
  8       1000000          0.74           77.02        0.78
  16      1000000          0.65           640.75       0.86
  32      1000000          0.90           134.63       0.96
  64      1000000          1.05           808.52       1.11
  128     1000000          1.05           405.58       1.47

[3] BF off and using doorbell
  #bytes #iterations    t_min[usec]    t_max[usec] t_typical[usec]
  2       1000000          0.85           107.29       0.89
  4       1000000          0.84           705.90       0.89
  8       1000000          0.85           457.72       0.89
  16      1000000          0.85           1041.43      0.90
  32      1000000          0.88           773.67       0.92
  64      1000000          0.90           82.70        0.93
  128     1000000          0.96           78.20        1.00


The 2nd set of results was obtained from running latency test
over bare-metal host where WC is available. Clearly we gain
better latency when BF is used vs. the doorbell base (around 300ns
of improvement, where there are systems which this climbs to 500ns).

# /usr/bin/taskset -c 0 ib_write_lat -d mlx4_0 -i 1  -F -a -n 1000000

[1] BF on, WC available
#bytes #iterations    t_min[usec]    t_max[usec] t_typical[usec]
2       1000000          0.74           131.62       0.79
4       1000000          0.74           134.51       0.79
8       1000000          0.74           154.30       0.79
16      1000000          0.74           1437.57      0.79
32      1000000          0.79           138.23       0.83
64      1000000          0.82           135.86       0.85
128     1000000          0.94           131.11       0.98

[3] BF off and using doorbell
#bytes #iterations    t_min[usec]    t_max[usec] t_typical[usec]
2       1000000          1.05           137.55       1.10
4       1000000          1.04           422.50       1.10
8       1000000          1.05           141.26       1.10
16      1000000          1.06           1261.99      1.11
32      1000000          1.09           141.47       1.14
64      1000000          1.11           435.44       1.16
128     1000000          1.22           212.19       1.27

--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html