netdev - Re: tx-nocache-copy performance

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <1389041855.12212.210.camel@edumazet-glaptop2.roam.corp.google.com>
Date:	Mon, 06 Jan 2014 12:57:35 -0800
From:	Eric Dumazet <eric.dumazet@...il.com>
To:	Benjamin Poirier <bpoirier@...e.de>
Cc:	Tom Herbert <therbert@...gle.com>, netdev@...r.kernel.org
Subject: Re: tx-nocache-copy performance

On Mon, 2014-01-06 at 15:27 -0500, Benjamin Poirier wrote:
> Hi Tom,
> 
> In commit "c6e1a0d net: Allow no-cache copy from user on transmit
> (v3.0-rc1)" you introduced the tx-nocache-copy performance optimization
> and set it to on by default. I've tried to reproduce your testcase, as
> well as a few more, but I did not find any performance improvement from
> turning on tx-nocache-copy. Do you think tx-nocache-copy is still a
> worthwhile optimization and it should remain on by default? In which
> situations does it help?
> 
> I've ran latency tests similar to the ones you described in the commit
> log. I've also tested how the option affects single stream throughput
> tests. According to the results I obtained, it seems that
> tx-nocache-copy has either no impact (in the latency test) or a negative
> impact (in the throughput test).
> 
> My test results follow. I tested using 3.12.6 on one Intel Xeon W3565
> and one i7 920 connected by ixgbe adapters. The results are from the
> Xeon, but they're similar on the i7. All numbers report the mean±stddev
> over 10 runs of 10s.
> 
> 1) latency tests similar to what you described
> There is no statistically significant difference between tx-nocache-copy
> on/off.
> nic irqs spread out (one queue per cpu)
> 
> 200x netperf -r 1400,1
> tx-nocache-copy off
>         692000±1000 tps
>         50/90/95/99% latency (us): 275±2/643.8±0.4/799±1/2474.4±0.3
> tx-nocache-copy on
>         693000±1000 tps
>         50/90/95/99% latency (us): 274±1/644.1±0.7/800±2/2474.5±0.7
> 
> 200x netperf -r 14000,14000
> tx-nocache-copy off
>         86450±80 tps
>         50/90/95/99% latency (us): 334.37±0.02/838±1/2100±20/3990±40
> tx-nocache-copy on
>         86110±60 tps
>         50/90/95/99% latency (us): 334.28±0.01/837±2/2110±20/3990±20
> 
> 2) single stream throughput tests
> tx-nocache-copy leads to higher service demand
> 
>                         throughput  cpu0        cpu1        demand
>                         (Gb/s)      (Gcycle)    (Gcycle)    (cycle/B)
> 
> nic irqs and netperf on cpu0 (1x netperf -T0,0 -t omni -- -d send)
> 
> tx-nocache-copy off     9402±5      9.4±0.2                 0.80±0.01
> tx-nocache-copy on      9403±3      9.85±0.04               0.838±0.004
> 
> nic irqs on cpu0, netperf on cpu1 (1x netperf -T1,1 -t omni -- -d send)
> 
> tx-nocache-copy off     9401±5      5.83±0.03   5.0±0.1     0.923±0.007
> tx-nocache-copy on      9404±2      5.74±0.03   5.523±0.009 0.958±0.002
> 
> -Benjamin

We raised this point last year, for same reasons.

http://www.spinics.net/lists/netdev/msg226501.html

I agree we should revert the default.

I'll post new perf data using latest net-next kernel and a bnx2x NIC.



--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html