| lists.openwall.net | lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC | |
|
Open Source and information security mailing list archives
| ||
|
Date: Mon, 13 Aug 2012 19:23:09 +0300 From: Claudiu Manoil <claudiu.manoil@...escale.com> To: Tomas Hruby <thruby@...il.com>, Eric Dumazet <eric.dumazet@...il.com>, Paul Gortmaker <paul.gortmaker@...driver.com> CC: <netdev@...r.kernel.org>, "David S. Miller" <davem@...emloft.net> Subject: Re: [RFC net-next 0/4] gianfar: Use separate NAPI for Tx confirmation processing On 08/09/2012 06:07 PM, Claudiu Manoil wrote: > On 8/9/2012 2:06 AM, Tomas Hruby wrote: >> On Wed, Aug 8, 2012 at 9:44 AM, Eric Dumazet <eric.dumazet@...il.com> >> wrote: >>> On Wed, 2012-08-08 at 12:24 -0400, Paul Gortmaker wrote: >>>> [[RFC net-next 0/4] gianfar: Use separate NAPI for Tx confirmation >>>> processing] On 08/08/2012 (Wed 15:26) Claudiu Manoil wrote: >>>> >>>>> Hi all, >>>>> This set of patches basically splits the existing napi poll >>>>> routine into >>>>> two separate napi functions, one for Rx processing (triggered by >>>>> frame >>>>> receive interrupts only) and one for the Tx confirmation path >>>>> processing >>>>> (triggerred by Tx confirmation interrupts only). The polling >>>>> algorithm >>>>> behind remains much the same. >>>>> >>>>> Important throughput improvements have been noted on low power >>>>> boards with >>>>> this set of changes. >>>>> For instance, for the following netperf test: >>>>> netperf -l 20 -cC -H 192.168.10.1 -t TCP_STREAM -- -m 1500 >>>>> yields a throughput gain from oscilating ~500-~700 Mbps to steady >>>>> ~940 Mbps, >>>>> (if the Rx/Tx paths are processed on different cores), w/ no >>>>> increase in CPU%, >>>>> on a p1020rdb - 2 core machine featuring etsec2.0 (Multi-Queue >>>>> Multi-Group >>>>> driver mode). >>>> >>>> It would be interesting to know more about what was causing that large >>>> an oscillation -- presumably you will have it reappear once one core >>>> becomes 100% utilized. Also, any thoughts on how the change will >>>> change >>>> performance on an older low power single core gianfar system (e.g. >>>> 83xx)? >>> >>> I also was wondering if this low performance could be caused by BQL >>> >>> Since TCP stack is driven by incoming ACKS, a NAPI run could have to >>> handle 10 TCP acks in a row, and resulting xmits could hit BQL and >>> transit on qdisc (Because NAPI handler wont handle TX completions in >>> the >>> middle of RX handler) >> >> Does disabling BQL help? Is the BQL limit stable? To what value is it >> set? I would be very much interested in more data if the issue is BQL >> related. >> >> . >> > > I agree that more tests should be run to investigate why gianfar under- > performs on the low power p1020rdb platform, and BQL seems to be > a good starting point (thanks for the hint). What I can say now is that > the issue is not apparent on p2020rdb, for instance, which is a more > powerful platform: the CPUs - 1200 MHz instead of 800 MHz; twice the > size of L2 cache (512 KB), greater bus (CCB) frequency ... On this > board (p2020rdb) the netperf test reaches 940Mbps both w/ and w/o these > patches. > > For a single core system I'm not expecting any performance degradation, > simply because I don't see why the proposed napi poll implementation > would be slower than the existing one. I'll do some measurements on a > p1010rdb too (single core, CPU:800 MHz) and get back to you with the > results. > Hi all, Please find below the netperf measurements performed on a p1010rdb machine (single core, low power). Three kernel images were used: 1) Linux version 3.5.0-20970-gaae06bf -- net-next commit aae06bf 2) Linux version 3.5.0-20974-g2920464 -- commit aae06bf + Tx NAPI patches 3) Linux version 3.5.0-20970-gaae06bf-dirty -- commit aae06bf + CONFIG_BQL set to 'n' The results show that, on *Image 1)*, by adjusting tcp_limit_output_bytes no substantial improvements are seen, as the throughput stays in the 580-60x Mbps range . By changing the coalescing settings from default* (rx coalescing off, tx-usecs: 10, tx-frames: 16) to: "ethtool -C eth1 rx-frames 22 tx-frames 22 rx-usecs 32 tx-usecs 32" we get a throughput of ~710 Mbps. For *Image 2)*, using the default tcp_limit_output_bytes value (131072) - I've noticed that "tweaking" tcp_limit_output_bytes does not improve the throughput -, we get the following performance numbers: * default coalescing settings: ~650 Mbps * rx-frames tx-frames 22 rx-usecs 32 tx-usecs 32: ~860-880 Mbps For *Image 3)*, by disabling BQL (CONFIG_BQL = n), there's *no* relevant performance improvement compared to Image 1). (note: For all the measurements, rx and tx BD ring sizes have been set to 64, for best performance.) So, I really tend to believe that the performance degradation comes primarily from the driver, and the napi poll processing turns out to be an important source for that. The proposed patches show substantial improvement, especially for SMP systems where Tx and Rx processing may be done in parallel. What do you think? Is it ok to proceed by re-spinning the patches? Do you recommend additional measurements? Regards, Claudiu //=Image 1)================ root@...10rdb:~# cat /proc/version Linux version 3.5.0-20970-gaae06bf [...] root@...10rdb:~# zcat /proc/config.gz | grep BQL CONFIG_BQL=y root@...10rdb:~# cat /proc/sys/net/ipv4/tcp_limit_output_bytes 131072 root@...10rdb:~# netperf -l 20 -cC -H 192.168.10.1 -t TCP_STREAM -- -m 1500 TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 192.168.10.1 (192.168.10.1) port 0 AF_INET Recv Send Send Utilization Service Demand Socket Socket Message Elapsed Send Recv Send Recv Size Size Size Time Throughput local remote local remote bytes bytes bytes secs. 10^6bits/s % S % S us/KB us/KB 87380 16384 1500 20.00 580.76 99.95 11.76 14.099 1.659 root@...10rdb:~# netperf -l 20 -cC -H 192.168.10.1 -t TCP_STREAM -- -m 1500 TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 192.168.10.1 (192.168.10.1) port 0 AF_INET Recv Send Send Utilization Service Demand Socket Socket Message Elapsed Send Recv Send Recv Size Size Size Time Throughput local remote local remote bytes bytes bytes secs. 10^6bits/s % S % S us/KB us/KB 87380 16384 1500 20.00 598.21 99.95 10.91 13.687 1.493 root@...10rdb:~# netperf -l 20 -cC -H 192.168.10.1 -t TCP_STREAM -- -m 1500 TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 192.168.10.1 (192.168.10.1) port 0 AF_INET Recv Send Send Utilization Service Demand Socket Socket Message Elapsed Send Recv Send Recv Size Size Size Time Throughput local remote local remote bytes bytes bytes secs. 10^6bits/s % S % S us/KB us/KB 87380 16384 1500 20.00 583.04 99.95 11.25 14.043 1.581 root@...10rdb:~# cat /proc/sys/net/ipv4/tcp_limit_output_bytes 65536 root@...10rdb:~# netperf -l 20 -cC -H 192.168.10.1 -t TCP_STREAM -- -m 1500 TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 192.168.10.1 (192.168.10.1) port 0 AF_INET Recv Send Send Utilization Service Demand Socket Socket Message Elapsed Send Recv Send Recv Size Size Size Time Throughput local remote local remote bytes bytes bytes secs. 10^6bits/s % S % S us/KB us/KB 87380 16384 1500 20.00 604.29 99.95 11.15 13.550 1.512 root@...10rdb:~# netperf -l 20 -cC -H 192.168.10.1 -t TCP_STREAM -- -m 1500 TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 192.168.10.1 (192.168.10.1) port 0 AF_INET Recv Send Send Utilization Service Demand Socket Socket Message Elapsed Send Recv Send Recv Size Size Size Time Throughput local remote local remote bytes bytes bytes secs. 10^6bits/s % S % S us/KB us/KB 87380 16384 1500 20.00 603.52 99.50 12.57 13.506 1.706 root@...10rdb:~# netperf -l 20 -cC -H 192.168.10.1 -t TCP_STREAM -- -m 1500 TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 192.168.10.1 (192.168.10.1) port 0 AF_INET Recv Send Send Utilization Service Demand Socket Socket Message Elapsed Send Recv Send Recv Size Size Size Time Throughput local remote local remote bytes bytes bytes secs. 10^6bits/s % S % S us/KB us/KB 87380 16384 1500 20.00 596.18 99.95 12.81 13.734 1.760 root@...10rdb:~# cat /proc/sys/net/ipv4/tcp_limit_output_bytes 32768 root@...10rdb:~# netperf -l 20 -cC -H 192.168.10.1 -t TCP_STREAM -- -m 1500 TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 192.168.10.1 (192.168.10.1) port 0 AF_INET Recv Send Send Utilization Service Demand Socket Socket Message Elapsed Send Recv Send Recv Size Size Size Time Throughput local remote local remote bytes bytes bytes secs. 10^6bits/s % S % S us/KB us/KB 87380 16384 1500 20.00 582.32 99.95 12.96 14.061 1.824 root@...10rdb:~# netperf -l 20 -cC -H 192.168.10.1 -t TCP_STREAM -- -m 1500 TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 192.168.10.1 (192.168.10.1) port 0 AF_INET Recv Send Send Utilization Service Demand Socket Socket Message Elapsed Send Recv Send Recv Size Size Size Time Throughput local remote local remote bytes bytes bytes secs. 10^6bits/s % S % S us/KB us/KB 87380 16384 1500 20.00 583.79 99.95 11.19 14.026 1.571 root@...10rdb:~# netperf -l 20 -cC -H 192.168.10.1 -t TCP_STREAM -- -m 1500 TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 192.168.10.1 (192.168.10.1) port 0 AF_INET Recv Send Send Utilization Service Demand Socket Socket Message Elapsed Send Recv Send Recv Size Size Size Time Throughput local remote local remote bytes bytes bytes secs. 10^6bits/s % S % S us/KB us/KB 87380 16384 1500 20.00 584.16 99.95 11.36 14.016 1.592 root@...10rdb:~# ethtool -C eth1 rx-frames 22 tx-frames 22 rx-usecs 32 tx-usecs 32 root@...10rdb:~# netperf -l 20 -cC -H 192.168.10.1 -t TCP_STREAM -- -m 1500 TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 192.168.10.1 (192.168.10.1) port 0 AF_INET Recv Send Send Utilization Service Demand Socket Socket Message Elapsed Send Recv Send Recv Size Size Size Time Throughput local remote local remote bytes bytes bytes secs. 10^6bits/s % S % S us/KB us/KB 87380 16384 1500 20.00 708.77 99.85 13.32 11.541 1.540 root@...10rdb:~# netperf -l 20 -cC -H 192.168.10.1 -t TCP_STREAM -- -m 1500 TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 192.168.10.1 (192.168.10.1) port 0 AF_INET Recv Send Send Utilization Service Demand Socket Socket Message Elapsed Send Recv Send Recv Size Size Size Time Throughput local remote local remote bytes bytes bytes secs. 10^6bits/s % S % S us/KB us/KB 87380 16384 1500 20.00 710.50 99.95 12.46 11.524 1.437 root@...10rdb:~# netperf -l 20 -cC -H 192.168.10.1 -t TCP_STREAM -- -m 1500 TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 192.168.10.1 (192.168.10.1) port 0 AF_INET Recv Send Send Utilization Service Demand Socket Socket Message Elapsed Send Recv Send Recv Size Size Size Time Throughput local remote local remote bytes bytes bytes secs. 10^6bits/s % S % S us/KB us/KB 87380 16384 1500 20.00 709.95 99.95 14.15 11.533 1.633 //=Image 2)================ root@...10rdb:~# cat /proc/version Linux version 3.5.0-20974-g2920464 [...] root@...10rdb:~# netperf -l 20 -cC -H 192.168.10.1 -t TCP_STREAM -- -m 1500 TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 192.168.10.1 (192.168.10.1) port 0 AF_INET Recv Send Send Utilization Service Demand Socket Socket Message Elapsed Send Recv Send Recv Size Size Size Time Throughput local remote local remote bytes bytes bytes secs. 10^6bits/s % S % S us/KB us/KB 87380 16384 1500 20.00 652.60 99.95 13.05 12.547 1.638 root@...10rdb:~# netperf -l 20 -cC -H 192.168.10.1 -t TCP_STREAM -- -m 1500 TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 192.168.10.1 (192.168.10.1) port 0 AF_INET Recv Send Send Utilization Service Demand Socket Socket Message Elapsed Send Recv Send Recv Size Size Size Time Throughput local remote local remote bytes bytes bytes secs. 10^6bits/s % S % S us/KB us/KB 87380 16384 1500 20.00 657.47 99.95 11.81 12.454 1.471 root@...10rdb:~# netperf -l 20 -cC -H 192.168.10.1 -t TCP_STREAM -- -m 1500 TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 192.168.10.1 (192.168.10.1) port 0 AF_INET Recv Send Send Utilization Service Demand Socket Socket Message Elapsed Send Recv Send Recv Size Size Size Time Throughput local remote local remote bytes bytes bytes secs. 10^6bits/s % S % S us/KB us/KB 87380 16384 1500 20.00 655.77 99.95 11.80 12.486 1.474 root@...10rdb:~# ethtool -C eth1 rx-frames 22 rx-usecs 32 tx-frames 22 tx-usecs 32 root@...10rdb:~# cat /proc/sys/net/ipv4/tcp_limit_output_bytes 131072 root@...10rdb:~# netperf -l 20 -cC -H 192.168.10.1 -t TCP_STREAM -- -m 1500 TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 192.168.10.1 (192.168.10.1) port 0 AF_INET Recv Send Send Utilization Service Demand Socket Socket Message Elapsed Send Recv Send Recv Size Size Size Time Throughput local remote local remote bytes bytes bytes secs. 10^6bits/s % S % S us/KB us/KB 87380 16384 1500 20.01 882.42 99.20 18.06 9.209 1.676 root@...10rdb:~# netperf -l 20 -cC -H 192.168.10.1 -t TCP_STREAM -- -m 1500 TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 192.168.10.1 (192.168.10.1) port 0 AF_INET Recv Send Send Utilization Service Demand Socket Socket Message Elapsed Send Recv Send Recv Size Size Size Time Throughput local remote local remote bytes bytes bytes secs. 10^6bits/s % S % S us/KB us/KB 87380 16384 1500 20.00 867.02 99.75 16.21 9.425 1.531 root@...10rdb:~# netperf -l 20 -cC -H 192.168.10.1 -t TCP_STREAM -- -m 1500 TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 192.168.10.1 (192.168.10.1) port 0 AF_INET Recv Send Send Utilization Service Demand Socket Socket Message Elapsed Send Recv Send Recv Size Size Size Time Throughput local remote local remote bytes bytes bytes secs. 10^6bits/s % S % S us/KB us/KB 87380 16384 1500 20.01 874.29 99.85 15.25 9.356 1.429 //=Image 3)================ Linux version 3.5.0-20970-gaae06bf-dirty [...] //CONFIG_BQL = n root@...10rdb:~# cat /proc/version Linux version 3.5.0-20970-gaae06bf-dirty (b08782@...04-ws574.ea.freescale.net) (gcc version 4.6.2 (GCC) ) #3 Mon Aug 13 13:58:25 EEST 2012 root@...10rdb:~# zcat /proc/config.gz | grep BQL # CONFIG_BQL is not set root@...10rdb:~# netperf -l 20 -cC -H 192.168.10.1 -t TCP_STREAM -- -m 1500 TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 192.168.10.1 (192.168.10.1) port 0 AF_INET Recv Send Send Utilization Service Demand Socket Socket Message Elapsed Send Recv Send Recv Size Size Size Time Throughput local remote local remote bytes bytes bytes secs. 10^6bits/s % S % S us/KB us/KB 87380 16384 1500 20.00 595.08 99.95 12.51 13.759 1.722 root@...10rdb:~# netperf -l 20 -cC -H 192.168.10.1 -t TCP_STREAM -- -m 1500 TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 192.168.10.1 (192.168.10.1) port 0 AF_INET Recv Send Send Utilization Service Demand Socket Socket Message Elapsed Send Recv Send Recv Size Size Size Time Throughput local remote local remote bytes bytes bytes secs. 10^6bits/s % S % S us/KB us/KB 87380 16384 1500 20.00 593.95 99.95 10.96 13.785 1.511 root@...10rdb:~# netperf -l 20 -cC -H 192.168.10.1 -t TCP_STREAM -- -m 1500 TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 192.168.10.1 (192.168.10.1) port 0 AF_INET Recv Send Send Utilization Service Demand Socket Socket Message Elapsed Send Recv Send Recv Size Size Size Time Throughput local remote local remote bytes bytes bytes secs. 10^6bits/s % S % S us/KB us/KB 87380 16384 1500 20.00 595.30 99.90 11.11 13.747 1.528 root@...10rdb:~# ethtool -C eth1 rx-frames 22 rx-usecs 32 tx-frames 22 tx-usecs 32 root@...10rdb:~# netperf -l 20 -cC -H 192.168.10.1 -t TCP_STREAM -- -m 1500 TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 192.168.10.1 (192.168.10.1) port 0 AF_INET Recv Send Send Utilization Service Demand Socket Socket Message Elapsed Send Recv Send Recv Size Size Size Time Throughput local remote local remote bytes bytes bytes secs. 10^6bits/s % S % S us/KB us/KB 87380 16384 1500 20.00 710.46 99.95 12.46 11.525 1.437 root@...10rdb:~# netperf -l 20 -cC -H 192.168.10.1 -t TCP_STREAM -- -m 1500 TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 192.168.10.1 (192.168.10.1) port 0 AF_INET Recv Send Send Utilization Service Demand Socket Socket Message Elapsed Send Recv Send Recv Size Size Size Time Throughput local remote local remote bytes bytes bytes secs. 10^6bits/s % S % S us/KB us/KB 87380 16384 1500 20.00 714.27 99.95 14.05 11.463 1.611 root@...10rdb:~# netperf -l 20 -cC -H 192.168.10.1 -t TCP_STREAM -- -m 1500 TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 192.168.10.1 (192.168.10.1) port 0 AF_INET Recv Send Send Utilization Service Demand Socket Socket Message Elapsed Send Recv Send Recv Size Size Size Time Throughput local remote local remote bytes bytes bytes secs. 10^6bits/s % S % S us/KB us/KB 87380 16384 1500 20.00 717.69 99.95 12.56 11.409 1.433 . -- To unsubscribe from this list: send the line "unsubscribe netdev" in the body of a message to majordomo@...r.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Powered by blists - more mailing lists