| lists.openwall.net | lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC | |
|
Open Source and information security mailing list archives
| ||
|
Date: Sun, 11 Nov 2018 11:26:08 +0100
From: Paweł Staszewski <pstaszewski@...are.pl>
To: Jesper Dangaard Brouer <brouer@...hat.com>
Cc: Saeed Mahameed <saeedm@...lanox.com>,
"netdev@...r.kernel.org" <netdev@...r.kernel.org>
Subject: Re: Kernel 4.19 network performance - forwarding/routing normal users
traffic
W dniu 11.11.2018 o 09:03, Jesper Dangaard Brouer pisze:
> On Sat, 10 Nov 2018 23:19:50 +0100
> Paweł Staszewski <pstaszewski@...are.pl> wrote:
>
>> W dniu 10.11.2018 o 23:06, Jesper Dangaard Brouer pisze:
>>> On Sat, 10 Nov 2018 20:56:02 +0100
>>> Paweł Staszewski <pstaszewski@...are.pl> wrote:
>>>
>>>> W dniu 10.11.2018 o 20:49, Paweł Staszewski pisze:
>>>>> W dniu 10.11.2018 o 20:34, Jesper Dangaard Brouer pisze:
>>>>>> On Fri, 9 Nov 2018 23:20:38 +0100 Paweł Staszewski
>>>>>> <pstaszewski@...are.pl> wrote:
>>>>>>
>>>>>>> W dniu 08.11.2018 o 20:12, Paweł Staszewski pisze:
> [...]
>>> Do notice, the per CPU squeeze is not too large.
>> Yes - but im searching invisible thing now :) something invisible is
>> slowing down packet processing :)
>> So trying to find any counter that have something to do with packet
>> processing.
> NOTICE, I have given you the counters you need (below)
Yes noticed this :)
>>> [...]
>>>>>>> Remember those tests are now on two separate connectx5 connected to
>>>>>>> two separate pcie x16 gen 3.0
>>>>>> That is strange... I still suspect some HW NIC issue, can you provide
>>>>>> ethtool stats info via tool:
>>>>>>
>>>>>> https://github.com/netoptimizer/network-testing/blob/master/bin/ethtool_stats.pl
>>>>>>
>>>>>> $ ethtool_stats.pl --dev enp175s0 --dev enp216s0
>>>>>>
>>>>>> The tool remove zero-stats counters and report per sec stats. It makes
>>>>>> it easier to spot that is relevant for the given workload.
>>>>> yes mlnx have just too many counters that are always 0 for my case :)
>>>>> Will try this also
>>>>>
>>>> But still alot of non 0 counters
>>>> Show adapter(s) (enp175s0 enp216s0) statistics (ONLY that changed!)
>>>> Ethtool(enp175s0) stat: 8891 ( 8,891) <= ch0_arm /sec
>>> [...]
>>>
>>> I have copied the stats over in another document so I can better looks
>>> at it... and I've found some interesting stats.
>>>
>>> E.g. we can see that the NIC hardware is dropping packets.
>>>
>>> RX-drops on enp175s0:
>>>
>>> (enp175s0) stat: 4850734036 ( 4,850,734,036) <= rx_bytes /sec
>>> (enp175s0) stat: 5069043007 ( 5,069,043,007) <= rx_bytes_phy /sec
>>> -218308971 ( -218,308,971) Dropped bytes /sec
>>>
>>> (enp175s0) stat: 139602 ( 139,602) <= rx_discards_phy /sec
>>>
>>> (enp175s0) stat: 3717148 ( 3,717,148) <= rx_packets /sec
>>> (enp175s0) stat: 3862420 ( 3,862,420) <= rx_packets_phy /sec
>>> -145272 ( -145,272) Dropped packets /sec
>>>
>>>
>>> RX-drops on enp216s0 is less:
>>>
>>> (enp216s0) stat: 2592286809 ( 2,592,286,809) <= rx_bytes /sec
>>> (enp216s0) stat: 2633575771 ( 2,633,575,771) <= rx_bytes_phy /sec
>>> -41288962 ( -41,288,962) Dropped bytes /sec
>>>
>>> (enp216s0) stat: 464 (464) <= rx_discards_phy /sec
>>>
>>> (enp216s0) stat: 4971677 ( 4,971,677) <= rx_packets /sec
>>> (enp216s0) stat: 4975563 ( 4,975,563) <= rx_packets_phy /sec
>>> -3886 ( -3,886) Dropped packets /sec
>>>
>
> I would recommend, that you use ethtool stats and monitor rx_discards_phy.
> The PHY are the counters from the hardware, and it shows that packets
> are getting dropped at HW level. This can be because software is not
> fast enough to empty RX-queue, but in this case where CPUs are mostly
> idle I don't think that is the case.
>
That is why i was searching some counter for software - where is
something wrong.
Cause in earlier reports from ethtool there was also phy drops reported
- just when cpu's was saturated that was normal for me that phy can drop
packets if no more cpu cycles available to pickup them from hw
But in case where i have 50% idle cpu's - there should be no problem -
that is why i start to modify ethtool params for tx/rx ring and coalescence
Currently waiting for more traffic with new ethtool settings:
ethtool -g enp175s0
Ring parameters for enp175s0:
Pre-set maximums:
RX: 8192
RX Mini: 0
RX Jumbo: 0
TX: 8192
Current hardware settings:
RX: 4096
RX Mini: 0
RX Jumbo: 0
TX: 128
ethtool -c enp175s0
Coalesce parameters for enp175s0:
Adaptive RX: off TX: on
stats-block-usecs: 0
sample-interval: 0
pkt-rate-low: 0
pkt-rate-high: 0
dmac: 32517
rx-usecs: 64
rx-frames: 128
rx-usecs-irq: 0
rx-frames-irq: 0
tx-usecs: 8
tx-frames: 128
tx-usecs-irq: 0
tx-frames-irq: 0
rx-usecs-low: 0
rx-frame-low: 0
tx-usecs-low: 0
tx-frame-low: 0
rx-usecs-high: 0
rx-frame-high: 0
tx-usecs-high: 0
tx-frame-high: 0
Both ports same settings.
Current traffic:
bwm-ng v0.6.1 (probing every 1.000s), press 'h' for help
input: /proc/net/dev type: rate
| iface Rx Tx Total
==============================================================================
enp175s0: 37.85 Gb/s 7.77 Gb/s
45.62 Gb/s
enp216s0: 7.80 Gb/s 37.90 Gb/s
45.70 Gb/s
------------------------------------------------------------------------------
total: 45.61 Gb/s 45.63 Gb/s
91.24 Gb/s
and mpstat for cpu's
Average: CPU %usr %nice %sys %iowait %irq %soft %steal
%guest %gnice %idle
Average: all 0.33 0.00 1.48 0.01 0.00 12.11 0.00
0.00 0.00 86.06
Average: 0 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 100.00
Average: 1 0.00 0.00 0.90 0.00 0.00 0.00 0.00
0.00 0.00 99.10
Average: 2 0.10 0.00 0.20 0.80 0.00 0.00 0.00
0.00 0.00 98.90
Average: 3 0.10 0.00 0.30 0.00 0.00 0.00 0.00
0.00 0.00 99.60
Average: 4 14.10 0.00 1.00 0.00 0.00 0.00 0.00
0.00 0.00 84.90
Average: 5 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 100.00
Average: 6 0.00 0.00 1.50 0.00 0.00 0.00 0.00
0.00 0.00 98.50
Average: 7 0.20 0.00 2.00 0.00 0.00 0.00 0.00
0.00 0.00 97.80
Average: 8 0.10 0.00 0.40 0.00 0.00 0.00 0.00
0.00 0.00 99.50
Average: 9 0.00 0.00 0.60 0.00 0.00 0.00 0.00
0.00 0.00 99.40
Average: 10 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 100.00
Average: 11 0.00 0.00 5.60 0.00 0.00 0.00 0.00
0.00 0.00 94.40
Average: 12 0.00 0.00 4.10 0.00 0.00 0.00 0.00
0.00 0.00 95.90
Average: 13 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 100.00
Average: 14 0.00 0.00 1.90 0.00 0.00 27.30 0.00
0.00 0.00 70.80
Average: 15 0.00 0.00 2.10 0.00 0.00 26.00 0.00
0.00 0.00 71.90
Average: 16 0.00 0.00 2.10 0.00 0.00 25.40 0.00
0.00 0.00 72.50
Average: 17 0.20 0.00 1.80 0.00 0.00 23.10 0.00
0.00 0.00 74.90
Average: 18 0.00 0.00 2.00 0.00 0.00 25.50 0.00
0.00 0.00 72.50
Average: 19 0.00 0.00 1.90 0.00 0.00 20.20 0.00
0.00 0.00 77.90
Average: 20 0.10 0.00 1.00 0.00 0.00 26.90 0.00
0.00 0.00 72.00
Average: 21 0.10 0.00 2.80 0.00 0.00 24.70 0.00
0.00 0.00 72.40
Average: 22 0.80 0.00 3.30 0.00 0.00 24.30 0.00
0.00 0.00 71.60
Average: 23 0.10 0.00 1.80 0.00 0.00 26.60 0.00
0.00 0.00 71.50
Average: 24 0.10 0.00 1.20 0.00 0.00 23.60 0.00
0.00 0.00 75.10
Average: 25 0.00 0.00 1.80 0.00 0.00 26.60 0.00
0.00 0.00 71.60
Average: 26 0.00 0.00 1.50 0.00 0.00 26.70 0.00
0.00 0.00 71.80
Average: 27 0.10 0.00 0.70 0.00 0.00 26.70 0.00
0.00 0.00 72.50
Average: 28 0.70 0.00 0.30 0.00 0.00 0.00 0.00
0.00 0.00 99.00
Average: 29 0.20 0.00 1.50 0.00 0.00 0.00 0.00
0.00 0.00 98.30
Average: 30 0.10 0.00 0.60 0.00 0.00 0.00 0.00
0.00 0.00 99.30
Average: 31 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 100.00
Average: 32 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 100.00
Average: 33 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 100.00
Average: 34 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 100.00
Average: 35 0.10 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 99.90
Average: 36 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 100.00
Average: 37 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 100.00
Average: 38 0.00 0.00 2.80 0.00 0.00 0.00 0.00
0.00 0.00 97.20
Average: 39 0.00 0.00 7.40 0.00 0.00 0.00 0.00
0.00 0.00 92.60
Average: 40 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 100.00
Average: 41 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 100.00
Average: 42 0.00 0.00 2.10 0.00 0.00 28.40 0.00
0.00 0.00 69.50
Average: 43 0.00 0.00 1.60 0.00 0.00 25.00 0.00
0.00 0.00 73.40
Average: 44 0.10 0.00 1.60 0.00 0.00 23.90 0.00
0.00 0.00 74.40
Average: 45 0.00 0.00 1.60 0.00 0.00 21.00 0.00
0.00 0.00 77.40
Average: 46 0.00 0.00 2.20 0.00 0.00 28.00 0.00
0.00 0.00 69.80
Average: 47 0.00 0.00 2.80 0.00 0.00 20.30 0.00
0.00 0.00 76.90
Average: 48 0.00 0.00 2.50 0.00 0.00 21.60 0.00
0.00 0.00 75.90
Average: 49 0.00 0.00 0.80 0.00 0.00 22.50 0.00
0.00 0.00 76.70
Average: 50 0.40 0.00 3.00 0.00 0.00 23.50 0.00
0.00 0.00 73.10
Average: 51 0.60 0.00 2.50 0.00 0.00 25.00 0.00
0.00 0.00 71.90
Average: 52 0.10 0.00 1.30 0.00 0.00 20.70 0.00
0.00 0.00 77.90
Average: 53 0.00 0.00 2.20 0.00 0.00 22.80 0.00
0.00 0.00 75.00
Average: 54 0.00 0.00 1.40 0.00 0.00 20.80 0.00
0.00 0.00 77.80
Average: 55 0.00 0.00 2.10 0.00 0.00 21.30 0.00
0.00 0.00 76.60
Powered by blists - more mailing lists