netdev - Re: Kernel 4.19 network performance - forwarding/routing normal users traffic

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives
Hash Suite: Windows password security audit tool. GUI, reports in PDF.
[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]
Message-ID: <41525a83-9e8a-52e1-d536-d383399c795c@itcare.pl>
Date:   Thu, 1 Nov 2018 22:24:28 +0100
From:   Paweł Staszewski <pstaszewski@...are.pl>
To:     Saeed Mahameed <saeedm@...lanox.com>,
        "netdev@...r.kernel.org" <netdev@...r.kernel.org>
Subject: Re: Kernel 4.19 network performance - forwarding/routing normal users
 traffic



W dniu 01.11.2018 o 22:18, Paweł Staszewski pisze:
>
>
> W dniu 01.11.2018 o 21:37, Saeed Mahameed pisze:
>> On Thu, 2018-11-01 at 12:09 +0100, Paweł Staszewski wrote:
>>> W dniu 01.11.2018 o 10:50, Saeed Mahameed pisze:
>>>> On Wed, 2018-10-31 at 22:57 +0100, Paweł Staszewski wrote:
>>>>> Hi
>>>>>
>>>>> So maybee someone will be interested how linux kernel handles
>>>>> normal
>>>>> traffic (not pktgen :) )
>>>>>
>>>>>
>>>>> Server HW configuration:
>>>>>
>>>>> CPU : Intel(R) Xeon(R) Gold 6132 CPU @ 2.60GHz
>>>>>
>>>>> NIC's: 2x 100G Mellanox ConnectX-4 (connected to x16 pcie 8GT)
>>>>>
>>>>>
>>>>> Server software:
>>>>>
>>>>> FRR - as routing daemon
>>>>>
>>>>> enp175s0f0 (100G) - 16 vlans from upstreams (28 RSS binded to
>>>>> local
>>>>> numa
>>>>> node)
>>>>>
>>>>> enp175s0f1 (100G) - 343 vlans to clients (28 RSS binded to local
>>>>> numa
>>>>> node)
>>>>>
>>>>>
>>>>> Maximum traffic that server can handle:
>>>>>
>>>>> Bandwidth
>>>>>
>>>>>     bwm-ng v0.6.1 (probing every 1.000s), press 'h' for help
>>>>>      input: /proc/net/dev type: rate
>>>>>      \         iface                   Rx Tx                Total
>>>>> =================================================================
>>>>> ====
>>>>> =========
>>>>>           enp175s0f1:          28.51 Gb/s           37.24
>>>>> Gb/s
>>>>> 65.74 Gb/s
>>>>>           enp175s0f0:          38.07 Gb/s           28.44
>>>>> Gb/s
>>>>> 66.51 Gb/s
>>>>> ---------------------------------------------------------------
>>>>> ----
>>>>> -----------
>>>>>                total:          66.58 Gb/s           65.67
>>>>> Gb/s
>>>>> 132.25 Gb/s
>>>>>
>>>>>
>>>>> Packets per second:
>>>>>
>>>>>     bwm-ng v0.6.1 (probing every 1.000s), press 'h' for help
>>>>>      input: /proc/net/dev type: rate
>>>>>      -         iface                   Rx Tx                Total
>>>>> =================================================================
>>>>> ====
>>>>> =========
>>>>>           enp175s0f1:      5248589.00 P/s       3486617.75 P/s
>>>>> 8735207.00 P/s
>>>>>           enp175s0f0:      3557944.25 P/s       5232516.00 P/s
>>>>> 8790460.00 P/s
>>>>> ---------------------------------------------------------------
>>>>> ----
>>>>> -----------
>>>>>                total:      8806533.00 P/s       8719134.00 P/s
>>>>> 17525668.00 P/s
>>>>>
>>>>>
>>>>> After reaching that limits nics on the upstream side (more RX
>>>>> traffic)
>>>>> start to drop packets
>>>>>
>>>>>
>>>>> I just dont understand that server can't handle more bandwidth
>>>>> (~40Gbit/s is limit where all cpu's are 100% util) - where pps on
>>>>> RX
>>>>> side are increasing.
>>>>>
>>>> Where do you see 40 Gb/s ? you showed that both ports on the same
>>>> NIC (
>>>> same pcie link) are doing  66.58 Gb/s (RX) + 65.67 Gb/s (TX) =
>>>> 132.25
>>>> Gb/s which aligns with your pcie link limit, what am i missing ?
>>> hmm yes that was my concern also - cause cant find anywhere
>>> informations
>>> about that bandwidth is uni or bidirectional - so if 126Gbit for x16
>>> 8GT
>>> is unidir - then bidir will be 126/2 ~68Gbit - which will fit total
>>> bw
>>> on both ports
>> i think it is bidir
>>
>>> This can explain maybee also why cpuload is rising rapidly from
>>> 120Gbit/s in total to 132Gbit (counters of bwmng are from /proc/net -
>>> so
>>> there can be some error in reading them when offloading (gro/gso/tso)
>>> on
>>> nic's is enabled that is why
>>>
>>>>> Was thinking that maybee reached some pcie x16 limit - but x16
>>>>> 8GT
>>>>> is
>>>>> 126Gbit - and also when testing with pktgen i can reach more bw
>>>>> and
>>>>> pps
>>>>> (like 4x more comparing to normal internet traffic)
>>>>>
>>>> Are you forwarding when using pktgen as well or you just testing
>>>> the RX
>>>> side pps ?
>>> Yes pktgen was tested on single port RX
>>> Can check also forwarding to eliminate pciex limits
>>>
>> So this explains why you have more RX pps, since tx is idle and pcie
>> will be free to do only rx.
>>
>> [...]
>>
>>
>>>>> ethtool -S enp175s0f1
>>>>> NIC statistics:
>>>>>         rx_packets: 173730800927
>>>>>         rx_bytes: 99827422751332
>>>>>         tx_packets: 142532009512
>>>>>         tx_bytes: 184633045911222
>>>>>         tx_tso_packets: 25989113891
>>>>>         tx_tso_bytes: 132933363384458
>>>>>         tx_tso_inner_packets: 0
>>>>>         tx_tso_inner_bytes: 0
>>>>>         tx_added_vlan_packets: 74630239613
>>>>>         tx_nop: 2029817748
>>>>>         rx_lro_packets: 0
>>>>>         rx_lro_bytes: 0
>>>>>         rx_ecn_mark: 0
>>>>>         rx_removed_vlan_packets: 173730800927
>>>>>         rx_csum_unnecessary: 0
>>>>>         rx_csum_none: 434357
>>>>>         rx_csum_complete: 173730366570
>>>>>         rx_csum_unnecessary_inner: 0
>>>>>         rx_xdp_drop: 0
>>>>>         rx_xdp_redirect: 0
>>>>>         rx_xdp_tx_xmit: 0
>>>>>         rx_xdp_tx_full: 0
>>>>>         rx_xdp_tx_err: 0
>>>>>         rx_xdp_tx_cqe: 0
>>>>>         tx_csum_none: 38260960853
>>>>>         tx_csum_partial: 36369278774
>>>>>         tx_csum_partial_inner: 0
>>>>>         tx_queue_stopped: 1
>>>>>         tx_queue_dropped: 0
>>>>>         tx_xmit_more: 748638099
>>>>>         tx_recover: 0
>>>>>         tx_cqes: 73881645031
>>>>>         tx_queue_wake: 1
>>>>>         tx_udp_seg_rem: 0
>>>>>         tx_cqe_err: 0
>>>>>         tx_xdp_xmit: 0
>>>>>         tx_xdp_full: 0
>>>>>         tx_xdp_err: 0
>>>>>         tx_xdp_cqes: 0
>>>>>         rx_wqe_err: 0
>>>>>         rx_mpwqe_filler_cqes: 0
>>>>>         rx_mpwqe_filler_strides: 0
>>>>>         rx_buff_alloc_err: 0
>>>>>         rx_cqe_compress_blks: 0
>>>>>         rx_cqe_compress_pkts: 0
>>>> If this is a pcie bottleneck it might be useful to  enable CQE
>>>> compression (to reduce PCIe completion descriptors transactions)
>>>> you should see the above rx_cqe_compress_pkts increasing when
>>>> enabled.
>>>>
>>>> $ ethtool  --set-priv-flags enp175s0f1 rx_cqe_compress on
>>>> $ ethtool --show-priv-flags enp175s0f1
>>>> Private flags for p6p1:
>>>> rx_cqe_moder       : on
>>>> cqe_moder          : off
>>>> rx_cqe_compress    : on
>>>> ...
>>>>
>>>> try this on both interfaces.
>>> Done
>>> ethtool --show-priv-flags enp175s0f1
>>> Private flags for enp175s0f1:
>>> rx_cqe_moder       : on
>>> tx_cqe_moder       : off
>>> rx_cqe_compress    : on
>>> rx_striding_rq     : off
>>> rx_no_csum_complete: off
>>>
>>> ethtool --show-priv-flags enp175s0f0
>>> Private flags for enp175s0f0:
>>> rx_cqe_moder       : on
>>> tx_cqe_moder       : off
>>> rx_cqe_compress    : on
>>> rx_striding_rq     : off
>>> rx_no_csum_complete: off
>>>
>> did it help reduce the load on the pcie  ? do you see more pps ?
>> what is the ratio between rx_cqe_compress_pkts and over all rx packets
>> ?
> So - a little more pps
> Before change top - graph / after bottom -> image with graph stats 
> from proc/net/dev
Attached link to graph
https://uploadfiles.io/5vgbh


> cqe_compress enabled at 11:55
>
> Sorry - but for real life traffic it is hard to do any counter 
> differences - cause traffic just rising alone from minute to minute :)
> But for that time the change is visible on graph - cause was almost 
> same for past 20minutes before change.
>
>
> full ethtool below:
> NIC statistics:
>      rx_packets: 516522465438
>      rx_bytes: 680052911258729
>      tx_packets: 677697545586
>      tx_bytes: 413647643141709
>      tx_tso_packets: 42530913279
>      tx_tso_bytes: 235655668554142
>      tx_tso_inner_packets: 0
>      tx_tso_inner_bytes: 0
>      tx_added_vlan_packets: 551156530885
>      tx_nop: 8536823558
>      rx_lro_packets: 0
>      rx_lro_bytes: 0
>      rx_ecn_mark: 0
>      rx_removed_vlan_packets: 516522465438
>      rx_csum_unnecessary: 0
>      rx_csum_none: 50382868
>      rx_csum_complete: 516472082570
>      rx_csum_unnecessary_inner: 0
>      rx_xdp_drop: 0
>      rx_xdp_redirect: 0
>      rx_xdp_tx_xmit: 0
>      rx_xdp_tx_full: 0
>      rx_xdp_tx_err: 0
>      rx_xdp_tx_cqe: 0
>      tx_csum_none: 494075047017
>      tx_csum_partial: 57081483898
>      tx_csum_partial_inner: 0
>      tx_queue_stopped: 518624
>      tx_queue_dropped: 0
>      tx_xmit_more: 1717880628
>      tx_recover: 0
>      tx_cqes: 549438869029
>      tx_queue_wake: 518627
>      tx_udp_seg_rem: 0
>      tx_cqe_err: 0
>      tx_xdp_xmit: 0
>      tx_xdp_full: 0
>      tx_xdp_err: 0
>      tx_xdp_cqes: 0
>      rx_wqe_err: 0
>      rx_mpwqe_filler_cqes: 0
>      rx_mpwqe_filler_strides: 0
>      rx_buff_alloc_err: 0
>      rx_cqe_compress_blks: 11483228712
>      rx_cqe_compress_pkts: 25794213324
>      rx_page_reuse: 0
>      rx_cache_reuse: 63610249810
>      rx_cache_full: 194650916511
>      rx_cache_empty: 1118208
>      rx_cache_busy: 194650982430
>      rx_cache_waive: 0
>      rx_congst_umr: 0
>      rx_arfs_err: 0
>      ch_events: 119556002196
>      ch_poll: 121107424977
>      ch_arm: 115856746008
>      ch_aff_change: 31
>      ch_eq_rearm: 0
>      rx_out_of_buffer: 6880325
>      rx_if_down_packets: 2062529
>      rx_vport_unicast_packets: 517433716795
>      rx_vport_unicast_bytes: 683464347301443
>      tx_vport_unicast_packets: 677697453738
>      tx_vport_unicast_bytes: 415788589663315
>      rx_vport_multicast_packets: 208258309
>      rx_vport_multicast_bytes: 14224046052
>      tx_vport_multicast_packets: 21689
>      tx_vport_multicast_bytes: 2158334
>      rx_vport_broadcast_packets: 75838646
>      rx_vport_broadcast_bytes: 4697944695
>      tx_vport_broadcast_packets: 68730
>      tx_vport_broadcast_bytes: 4123800
>      rx_vport_rdma_unicast_packets: 0
>      rx_vport_rdma_unicast_bytes: 0
>      tx_vport_rdma_unicast_packets: 0
>      tx_vport_rdma_unicast_bytes: 0
>      rx_vport_rdma_multicast_packets: 0
>      rx_vport_rdma_multicast_bytes: 0
>      tx_vport_rdma_multicast_packets: 0
>      tx_vport_rdma_multicast_bytes: 0
>      tx_packets_phy: 677697543252
>      rx_packets_phy: 521319491878
>      rx_crc_errors_phy: 0
>      tx_bytes_phy: 418499385791411
>      rx_bytes_phy: 690322537017274
>      tx_multicast_phy: 21689
>      tx_broadcast_phy: 68730
>      rx_multicast_phy: 208258305
>      rx_broadcast_phy: 75838646
>      rx_in_range_len_errors_phy: 4
>      rx_out_of_range_len_phy: 0
>      rx_oversize_pkts_phy: 0
>      rx_symbol_err_phy: 0
>      tx_mac_control_phy: 0
>      rx_mac_control_phy: 0
>      rx_unsupported_op_phy: 0
>      rx_pause_ctrl_phy: 0
>      tx_pause_ctrl_phy: 0
>      rx_discards_phy: 3601449265
>      tx_discards_phy: 0
>      tx_errors_phy: 0
>      rx_undersize_pkts_phy: 0
>      rx_fragments_phy: 0
>      rx_jabbers_phy: 0
>      rx_64_bytes_phy: 1416456771
>      rx_65_to_127_bytes_phy: 40750434737
>      rx_128_to_255_bytes_phy: 11518110310
>      rx_256_to_511_bytes_phy: 7055850637
>      rx_512_to_1023_bytes_phy: 7811550424
>      rx_1024_to_1518_bytes_phy: 265547564845
>      rx_1519_to_2047_bytes_phy: 187219522899
>      rx_2048_to_4095_bytes_phy: 0
>      rx_4096_to_8191_bytes_phy: 0
>      rx_8192_to_10239_bytes_phy: 0
>      link_down_events_phy: 0
>      rx_pcs_symbol_err_phy: 0
>      rx_corrected_bits_phy: 0
>      rx_pci_signal_integrity: 0
>      tx_pci_signal_integrity: 48
>      rx_prio0_bytes: 688807632117485
>      rx_prio0_packets: 516310309931
>      tx_prio0_bytes: 418499382756025
>      tx_prio0_packets: 677697534982
>      rx_prio1_bytes: 1497701612877
>      rx_prio1_packets: 1206768094
>      tx_prio1_bytes: 0
>      tx_prio1_packets: 0
>      rx_prio2_bytes: 112271227
>      rx_prio2_packets: 337295
>      tx_prio2_bytes: 0
>      tx_prio2_packets: 0
>      rx_prio3_bytes: 1165455555
>      rx_prio3_packets: 1544310
>      tx_prio3_bytes: 0
>      tx_prio3_packets: 0
>      rx_prio4_bytes: 161857240
>      rx_prio4_packets: 341392
>      tx_prio4_bytes: 0
>      tx_prio4_packets: 0
>      rx_prio5_bytes: 455031612
>      rx_prio5_packets: 2861469
>      tx_prio5_bytes: 0
>      tx_prio5_packets: 0
>      rx_prio6_bytes: 1873928697
>      rx_prio6_packets: 5146981
>      tx_prio6_bytes: 0
>      tx_prio6_packets: 0
>      rx_prio7_bytes: 13423452430
>      rx_prio7_packets: 190724796
>      tx_prio7_bytes: 0
>      tx_prio7_packets: 0
>      module_unplug: 0
>      module_bus_stuck: 0
>      module_high_temp: 0
>      module_bad_shorted: 0
>      ch0_events: 4252266777
>      ch0_poll: 4330804273
>      ch0_arm: 4120233182
>      ch0_aff_change: 2
>      ch0_eq_rearm: 0
>      ch1_events: 3938415938
>      ch1_poll: 4012621322
>      ch1_arm: 3810131188
>      ch1_aff_change: 2
>      ch1_eq_rearm: 0
>      ch2_events: 3897428860
>      ch2_poll: 3973886848
>      ch2_arm: 3773019397
>      ch2_aff_change: 1
>      ch2_eq_rearm: 0
>      ch3_events: 4108000541
>      ch3_poll: 4180139872
>      ch3_arm: 3982093366
>      ch3_aff_change: 1
>      ch3_eq_rearm: 0
>      ch4_events: 4652570079
>      ch4_poll: 4720541090
>      ch4_arm: 4524475054
>      ch4_aff_change: 2
>      ch4_eq_rearm: 0
>      ch5_events: 3899177385
>      ch5_poll: 3974274186
>      ch5_arm: 3772299186
>      ch5_aff_change: 2
>      ch5_eq_rearm: 0
>      ch6_events: 3915161350
>      ch6_poll: 3992338199
>      ch6_arm: 3794710989
>      ch6_aff_change: 0
>      ch6_eq_rearm: 0
>      ch7_events: 4008175631
>      ch7_poll: 4081321248
>      ch7_arm: 3882263723
>      ch7_aff_change: 0
>      ch7_eq_rearm: 0
>      ch8_events: 4207422352
>      ch8_poll: 4276465449
>      ch8_arm: 4077650366
>      ch8_aff_change: 0
>      ch8_eq_rearm: 0
>      ch9_events: 4036491879
>      ch9_poll: 4108975987
>      ch9_arm: 3914493694
>      ch9_aff_change: 0
>      ch9_eq_rearm: 0
>      ch10_events: 4066261595
>      ch10_poll: 4134419606
>      ch10_arm: 3936637711
>      ch10_aff_change: 1
>      ch10_eq_rearm: 0
>      ch11_events: 4440494043
>      ch11_poll: 4507578730
>      ch11_arm: 4318629438
>      ch11_aff_change: 0
>      ch11_eq_rearm: 0
>      ch12_events: 4066958252
>      ch12_poll: 4130191506
>      ch12_arm: 3934337782
>      ch12_aff_change: 0
>      ch12_eq_rearm: 0
>      ch13_events: 4051309159
>      ch13_poll: 4118864120
>      ch13_arm: 3921011919
>      ch13_aff_change: 0
>      ch13_eq_rearm: 0
>      ch14_events: 4321664800
>      ch14_poll: 4382433680
>      ch14_arm: 4186130552
>      ch14_aff_change: 0
>      ch14_eq_rearm: 0
>      ch15_events: 4701102075
>      ch15_poll: 4760373932
>      ch15_arm: 4570151468
>      ch15_aff_change: 0
>      ch15_eq_rearm: 0
>      ch16_events: 4311052687
>      ch16_poll: 4345937129
>      ch16_arm: 4170883819
>      ch16_aff_change: 0
>      ch16_eq_rearm: 0
>      ch17_events: 4647570931
>      ch17_poll: 4680218533
>      ch17_arm: 4509426288
>      ch17_aff_change: 0
>      ch17_eq_rearm: 0
>      ch18_events: 4598195702
>      ch18_poll: 4631314898
>      ch18_arm: 4457267084
>      ch18_aff_change: 0
>      ch18_eq_rearm: 0
>      ch19_events: 4808094560
>      ch19_poll: 4841368340
>      ch19_arm: 4670604358
>      ch19_aff_change: 0
>      ch19_eq_rearm: 0
>      ch20_events: 4240910605
>      ch20_poll: 4276531502
>      ch20_arm: 4101767278
>      ch20_aff_change: 1
>      ch20_eq_rearm: 0
>      ch21_events: 4389371472
>      ch21_poll: 4426870311
>      ch21_arm: 4249339045
>      ch21_aff_change: 2
>      ch21_eq_rearm: 0
>      ch22_events: 4282958754
>      ch22_poll: 4319228073
>      ch22_arm: 4145102991
>      ch22_aff_change: 2
>      ch22_eq_rearm: 0
>      ch23_events: 4440196528
>      ch23_poll: 4474090188
>      ch23_arm: 4300837147
>      ch23_aff_change: 2
>      ch23_eq_rearm: 0
>      ch24_events: 4326875785
>      ch24_poll: 4364971263
>      ch24_arm: 4186404526
>      ch24_aff_change: 2
>      ch24_eq_rearm: 0
>      ch25_events: 4286528453
>      ch25_poll: 4324089445
>      ch25_arm: 4147222616
>      ch25_aff_change: 3
>      ch25_eq_rearm: 0
>      ch26_events: 4098043104
>      ch26_poll: 4138133745
>      ch26_arm: 3967438971
>      ch26_aff_change: 4
>      ch26_eq_rearm: 0
>      ch27_events: 4563302840
>      ch27_poll: 4599441446
>      ch27_arm: 4432182806
>      ch27_aff_change: 4
>      ch27_eq_rearm: 0
>      ch28_events: 4
>      ch28_poll: 4
>      ch28_arm: 4
>      ch28_aff_change: 0
>      ch28_eq_rearm: 0
>      ch29_events: 6
>      ch29_poll: 6
>      ch29_arm: 6
>      ch29_aff_change: 0
>      ch29_eq_rearm: 0
>      ch30_events: 4
>      ch30_poll: 4
>      ch30_arm: 4
>      ch30_aff_change: 0
>      ch30_eq_rearm: 0
>      ch31_events: 4
>      ch31_poll: 4
>      ch31_arm: 4
>      ch31_aff_change: 0
>      ch31_eq_rearm: 0
>      ch32_events: 4
>      ch32_poll: 4
>      ch32_arm: 4
>      ch32_aff_change: 0
>      ch32_eq_rearm: 0
>      ch33_events: 4
>      ch33_poll: 4
>      ch33_arm: 4
>      ch33_aff_change: 0
>      ch33_eq_rearm: 0
>      ch34_events: 4
>      ch34_poll: 4
>      ch34_arm: 4
>      ch34_aff_change: 0
>      ch34_eq_rearm: 0
>      ch35_events: 4
>      ch35_poll: 4
>      ch35_arm: 4
>      ch35_aff_change: 0
>      ch35_eq_rearm: 0
>      ch36_events: 4
>      ch36_poll: 4
>      ch36_arm: 4
>      ch36_aff_change: 0
>      ch36_eq_rearm: 0
>      ch37_events: 4
>      ch37_poll: 4
>      ch37_arm: 4
>      ch37_aff_change: 0
>      ch37_eq_rearm: 0
>      ch38_events: 4
>      ch38_poll: 4
>      ch38_arm: 4
>      ch38_aff_change: 0
>      ch38_eq_rearm: 0
>      ch39_events: 4
>      ch39_poll: 4
>      ch39_arm: 4
>      ch39_aff_change: 0
>      ch39_eq_rearm: 0
>      ch40_events: 4
>      ch40_poll: 4
>      ch40_arm: 4
>      ch40_aff_change: 0
>      ch40_eq_rearm: 0
>      ch41_events: 4
>      ch41_poll: 4
>      ch41_arm: 4
>      ch41_aff_change: 0
>      ch41_eq_rearm: 0
>      ch42_events: 4
>      ch42_poll: 4
>      ch42_arm: 4
>      ch42_aff_change: 0
>      ch42_eq_rearm: 0
>      ch43_events: 4
>      ch43_poll: 4
>      ch43_arm: 4
>      ch43_aff_change: 0
>      ch43_eq_rearm: 0
>      ch44_events: 4
>      ch44_poll: 4
>      ch44_arm: 4
>      ch44_aff_change: 0
>      ch44_eq_rearm: 0
>      ch45_events: 4
>      ch45_poll: 4
>      ch45_arm: 4
>      ch45_aff_change: 0
>      ch45_eq_rearm: 0
>      ch46_events: 4
>      ch46_poll: 4
>      ch46_arm: 4
>      ch46_aff_change: 0
>      ch46_eq_rearm: 0
>      ch47_events: 4
>      ch47_poll: 4
>      ch47_arm: 4
>      ch47_aff_change: 0
>      ch47_eq_rearm: 0
>      ch48_events: 4
>      ch48_poll: 4
>      ch48_arm: 4
>      ch48_aff_change: 0
>      ch48_eq_rearm: 0
>      ch49_events: 4
>      ch49_poll: 4
>      ch49_arm: 4
>      ch49_aff_change: 0
>      ch49_eq_rearm: 0
>      ch50_events: 4
>      ch50_poll: 4
>      ch50_arm: 4
>      ch50_aff_change: 0
>      ch50_eq_rearm: 0
>      ch51_events: 4
>      ch51_poll: 4
>      ch51_arm: 4
>      ch51_aff_change: 0
>      ch51_eq_rearm: 0
>      ch52_events: 4
>      ch52_poll: 4
>      ch52_arm: 4
>      ch52_aff_change: 0
>      ch52_eq_rearm: 0
>      ch53_events: 4
>      ch53_poll: 4
>      ch53_arm: 4
>      ch53_aff_change: 0
>      ch53_eq_rearm: 0
>      ch54_events: 4
>      ch54_poll: 4
>      ch54_arm: 4
>      ch54_aff_change: 0
>      ch54_eq_rearm: 0
>      ch55_events: 4
>      ch55_poll: 4
>      ch55_arm: 4
>      ch55_aff_change: 0
>      ch55_eq_rearm: 0
>      rx0_packets: 21390033774
>      rx0_bytes: 27326856299122
>      rx0_csum_complete: 21339650906
>      rx0_csum_unnecessary: 0
>      rx0_csum_unnecessary_inner: 0
>      rx0_csum_none: 50382868
>      rx0_xdp_drop: 0
>      rx0_xdp_redirect: 0
>      rx0_lro_packets: 0
>      rx0_lro_bytes: 0
>      rx0_ecn_mark: 0
>      rx0_removed_vlan_packets: 21390033774
>      rx0_wqe_err: 0
>      rx0_mpwqe_filler_cqes: 0
>      rx0_mpwqe_filler_strides: 0
>      rx0_buff_alloc_err: 0
>      rx0_cqe_compress_blks: 481077641
>      rx0_cqe_compress_pkts: 1085647489
>      rx0_page_reuse: 0
>      rx0_cache_reuse: 19050049
>      rx0_cache_full: 10675964285
>      rx0_cache_empty: 37376
>      rx0_cache_busy: 10675966819
>      rx0_cache_waive: 0
>      rx0_congst_umr: 0
>      rx0_arfs_err: 0
>      rx0_xdp_tx_xmit: 0
>      rx0_xdp_tx_full: 0
>      rx0_xdp_tx_err: 0
>      rx0_xdp_tx_cqes: 0
>      rx1_packets: 19868919527
>      rx1_bytes: 26149716991561
>      rx1_csum_complete: 19868919527
>      rx1_csum_unnecessary: 0
>      rx1_csum_unnecessary_inner: 0
>      rx1_csum_none: 0
>      rx1_xdp_drop: 0
>      rx1_xdp_redirect: 0
>      rx1_lro_packets: 0
>      rx1_lro_bytes: 0
>      rx1_ecn_mark: 0
>      rx1_removed_vlan_packets: 19868919527
>      rx1_wqe_err: 0
>      rx1_mpwqe_filler_cqes: 0
>      rx1_mpwqe_filler_strides: 0
>      rx1_buff_alloc_err: 0
>      rx1_cqe_compress_blks: 420210560
>      rx1_cqe_compress_pkts: 941233388
>      rx1_page_reuse: 0
>      rx1_cache_reuse: 46200002
>      rx1_cache_full: 9888257242
>      rx1_cache_empty: 37376
>      rx1_cache_busy: 9888259746
>      rx1_cache_waive: 0
>      rx1_congst_umr: 0
>      rx1_arfs_err: 0
>      rx1_xdp_tx_xmit: 0
>      rx1_xdp_tx_full: 0
>      rx1_xdp_tx_err: 0
>      rx1_xdp_tx_cqes: 0
>      rx2_packets: 19575013662
>      rx2_bytes: 25759818417945
>      rx2_csum_complete: 19575013662
>      rx2_csum_unnecessary: 0
>      rx2_csum_unnecessary_inner: 0
>      rx2_csum_none: 0
>      rx2_xdp_drop: 0
>      rx2_xdp_redirect: 0
>      rx2_lro_packets: 0
>      rx2_lro_bytes: 0
>      rx2_ecn_mark: 0
>      rx2_removed_vlan_packets: 19575013662
>      rx2_wqe_err: 0
>      rx2_mpwqe_filler_cqes: 0
>      rx2_mpwqe_filler_strides: 0
>      rx2_buff_alloc_err: 0
>      rx2_cqe_compress_blks: 412345511
>      rx2_cqe_compress_pkts: 923376167
>      rx2_page_reuse: 0
>      rx2_cache_reuse: 38837731
>      rx2_cache_full: 9748666548
>      rx2_cache_empty: 37376
>      rx2_cache_busy: 9748669093
>      rx2_cache_waive: 0
>      rx2_congst_umr: 0
>      rx2_arfs_err: 0
>      rx2_xdp_tx_xmit: 0
>      rx2_xdp_tx_full: 0
>      rx2_xdp_tx_err: 0
>      rx2_xdp_tx_cqes: 0
>      rx3_packets: 19795911749
>      rx3_bytes: 25969475566905
>      rx3_csum_complete: 19795911749
>      rx3_csum_unnecessary: 0
>      rx3_csum_unnecessary_inner: 0
>      rx3_csum_none: 0
>      rx3_xdp_drop: 0
>      rx3_xdp_redirect: 0
>      rx3_lro_packets: 0
>      rx3_lro_bytes: 0
>      rx3_ecn_mark: 0
>      rx3_removed_vlan_packets: 19795911749
>      rx3_wqe_err: 0
>      rx3_mpwqe_filler_cqes: 0
>      rx3_mpwqe_filler_strides: 0
>      rx3_buff_alloc_err: 0
>      rx3_cqe_compress_blks: 416658765
>      rx3_cqe_compress_pkts: 934986266
>      rx3_page_reuse: 0
>      rx3_cache_reuse: 34542124
>      rx3_cache_full: 9863411232
>      rx3_cache_empty: 37376
>      rx3_cache_busy: 9863413732
>      rx3_cache_waive: 0
>      rx3_congst_umr: 0
>      rx3_arfs_err: 0
>      rx3_xdp_tx_xmit: 0
>      rx3_xdp_tx_full: 0
>      rx3_xdp_tx_err: 0
>      rx3_xdp_tx_cqes: 0
>      rx4_packets: 20445652378
>      rx4_bytes: 26949065110265
>      rx4_csum_complete: 20445652378
>      rx4_csum_unnecessary: 0
>      rx4_csum_unnecessary_inner: 0
>      rx4_csum_none: 0
>      rx4_xdp_drop: 0
>      rx4_xdp_redirect: 0
>      rx4_lro_packets: 0
>      rx4_lro_bytes: 0
>      rx4_ecn_mark: 0
>      rx4_removed_vlan_packets: 20445652378
>      rx4_wqe_err: 0
>      rx4_mpwqe_filler_cqes: 0
>      rx4_mpwqe_filler_strides: 0
>      rx4_buff_alloc_err: 0
>      rx4_cqe_compress_blks: 506085858
>      rx4_cqe_compress_pkts: 1147860328
>      rx4_page_reuse: 0
>      rx4_cache_reuse: 10122542864
>      rx4_cache_full: 100281206
>      rx4_cache_empty: 37376
>      rx4_cache_busy: 100283304
>      rx4_cache_waive: 0
>      rx4_congst_umr: 0
>      rx4_arfs_err: 0
>      rx4_xdp_tx_xmit: 0
>      rx4_xdp_tx_full: 0
>      rx4_xdp_tx_err: 0
>      rx4_xdp_tx_cqes: 0
>      rx5_packets: 19622362246
>      rx5_bytes: 25843450982982
>      rx5_csum_complete: 19622362246
>      rx5_csum_unnecessary: 0
>      rx5_csum_unnecessary_inner: 0
>      rx5_csum_none: 0
>      rx5_xdp_drop: 0
>      rx5_xdp_redirect: 0
>      rx5_lro_packets: 0
>      rx5_lro_bytes: 0
>      rx5_ecn_mark: 0
>      rx5_removed_vlan_packets: 19622362246
>      rx5_wqe_err: 0
>      rx5_mpwqe_filler_cqes: 0
>      rx5_mpwqe_filler_strides: 0
>      rx5_buff_alloc_err: 0
>      rx5_cqe_compress_blks: 422840924
>      rx5_cqe_compress_pkts: 948005878
>      rx5_page_reuse: 0
>      rx5_cache_reuse: 31285453
>      rx5_cache_full: 9779893117
>      rx5_cache_empty: 37376
>      rx5_cache_busy: 9779895647
>      rx5_cache_waive: 0
>      rx5_congst_umr: 0
>      rx5_arfs_err: 0
>      rx5_xdp_tx_xmit: 0
>      rx5_xdp_tx_full: 0
>      rx5_xdp_tx_err: 0
>      rx5_xdp_tx_cqes: 0
>      rx6_packets: 19788231278
>      rx6_bytes: 25985783006486
>      rx6_csum_complete: 19788231278
>      rx6_csum_unnecessary: 0
>      rx6_csum_unnecessary_inner: 0
>      rx6_csum_none: 0
>      rx6_xdp_drop: 0
>      rx6_xdp_redirect: 0
>      rx6_lro_packets: 0
>      rx6_lro_bytes: 0
>      rx6_ecn_mark: 0
>      rx6_removed_vlan_packets: 19788231278
>      rx6_wqe_err: 0
>      rx6_mpwqe_filler_cqes: 0
>      rx6_mpwqe_filler_strides: 0
>      rx6_buff_alloc_err: 0
>      rx6_cqe_compress_blks: 418799056
>      rx6_cqe_compress_pkts: 938282685
>      rx6_page_reuse: 0
>      rx6_cache_reuse: 18114793
>      rx6_cache_full: 9875998295
>      rx6_cache_empty: 37376
>      rx6_cache_busy: 9876000831
>      rx6_cache_waive: 0
>      rx6_congst_umr: 0
>      rx6_arfs_err: 0
>      rx6_xdp_tx_xmit: 0
>      rx6_xdp_tx_full: 0
>      rx6_xdp_tx_err: 0
>      rx6_xdp_tx_cqes: 0
>      rx7_packets: 19795759168
>      rx7_bytes: 26085056586860
>      rx7_csum_complete: 19795759168
>      rx7_csum_unnecessary: 0
>      rx7_csum_unnecessary_inner: 0
>      rx7_csum_none: 0
>      rx7_xdp_drop: 0
>      rx7_xdp_redirect: 0
>      rx7_lro_packets: 0
>      rx7_lro_bytes: 0
>      rx7_ecn_mark: 0
>      rx7_removed_vlan_packets: 19795759168
>      rx7_wqe_err: 0
>      rx7_mpwqe_filler_cqes: 0
>      rx7_mpwqe_filler_strides: 0
>      rx7_buff_alloc_err: 0
>      rx7_cqe_compress_blks: 413959224
>      rx7_cqe_compress_pkts: 927675936
>      rx7_page_reuse: 0
>      rx7_cache_reuse: 23902990
>      rx7_cache_full: 9873974042
>      rx7_cache_empty: 37376
>      rx7_cache_busy: 9873976574
>      rx7_cache_waive: 0
>      rx7_congst_umr: 0
>      rx7_arfs_err: 0
>      rx7_xdp_tx_xmit: 0
>      rx7_xdp_tx_full: 0
>      rx7_xdp_tx_err: 0
>      rx7_xdp_tx_cqes: 0
>      rx8_packets: 19963477439
>      rx8_bytes: 26384640501789
>      rx8_csum_complete: 19963477439
>      rx8_csum_unnecessary: 0
>      rx8_csum_unnecessary_inner: 0
>      rx8_csum_none: 0
>      rx8_xdp_drop: 0
>      rx8_xdp_redirect: 0
>      rx8_lro_packets: 0
>      rx8_lro_bytes: 0
>      rx8_ecn_mark: 0
>      rx8_removed_vlan_packets: 19963477439
>      rx8_wqe_err: 0
>      rx8_mpwqe_filler_cqes: 0
>      rx8_mpwqe_filler_strides: 0
>      rx8_buff_alloc_err: 0
>      rx8_cqe_compress_blks: 420422857
>      rx8_cqe_compress_pkts: 942720292
>      rx8_page_reuse: 0
>      rx8_cache_reuse: 88181713
>      rx8_cache_full: 9893554525
>      rx8_cache_empty: 37376
>      rx8_cache_busy: 9893556983
>      rx8_cache_waive: 0
>      rx8_congst_umr: 0
>      rx8_arfs_err: 0
>      rx8_xdp_tx_xmit: 0
>      rx8_xdp_tx_full: 0
>      rx8_xdp_tx_err: 0
>      rx8_xdp_tx_cqes: 0
>      rx9_packets: 19726642138
>      rx9_bytes: 26063924286499
>      rx9_csum_complete: 19726642138
>      rx9_csum_unnecessary: 0
>      rx9_csum_unnecessary_inner: 0
>      rx9_csum_none: 0
>      rx9_xdp_drop: 0
>      rx9_xdp_redirect: 0
>      rx9_lro_packets: 0
>      rx9_lro_bytes: 0
>      rx9_ecn_mark: 0
>      rx9_removed_vlan_packets: 19726642138
>      rx9_wqe_err: 0
>      rx9_mpwqe_filler_cqes: 0
>      rx9_mpwqe_filler_strides: 0
>      rx9_buff_alloc_err: 0
>      rx9_cqe_compress_blks: 424227411
>      rx9_cqe_compress_pkts: 951534873
>      rx9_page_reuse: 0
>      rx9_cache_reuse: 482901440
>      rx9_cache_full: 9380417487
>      rx9_cache_empty: 37376
>      rx9_cache_busy: 9380419608
>      rx9_cache_waive: 0
>      rx9_congst_umr: 0
>      rx9_arfs_err: 0
>      rx9_xdp_tx_xmit: 0
>      rx9_xdp_tx_full: 0
>      rx9_xdp_tx_err: 0
>      rx9_xdp_tx_cqes: 0
>      rx10_packets: 19901229170
>      rx10_bytes: 26300854495044
>      rx10_csum_complete: 19901229170
>      rx10_csum_unnecessary: 0
>      rx10_csum_unnecessary_inner: 0
>      rx10_csum_none: 0
>      rx10_xdp_drop: 0
>      rx10_xdp_redirect: 0
>      rx10_lro_packets: 0
>      rx10_lro_bytes: 0
>      rx10_ecn_mark: 0
>      rx10_removed_vlan_packets: 19901229170
>      rx10_wqe_err: 0
>      rx10_mpwqe_filler_cqes: 0
>      rx10_mpwqe_filler_strides: 0
>      rx10_buff_alloc_err: 0
>      rx10_cqe_compress_blks: 419082938
>      rx10_cqe_compress_pkts: 940791347
>      rx10_page_reuse: 0
>      rx10_cache_reuse: 14896055
>      rx10_cache_full: 9935715977
>      rx10_cache_empty: 37376
>      rx10_cache_busy: 9935718513
>      rx10_cache_waive: 0
>      rx10_congst_umr: 0
>      rx10_arfs_err: 0
>      rx10_xdp_tx_xmit: 0
>      rx10_xdp_tx_full: 0
>      rx10_xdp_tx_err: 0
>      rx10_xdp_tx_cqes: 0
>      rx11_packets: 20352190494
>      rx11_bytes: 26851034425372
>      rx11_csum_complete: 20352190494
>      rx11_csum_unnecessary: 0
>      rx11_csum_unnecessary_inner: 0
>      rx11_csum_none: 0
>      rx11_xdp_drop: 0
>      rx11_xdp_redirect: 0
>      rx11_lro_packets: 0
>      rx11_lro_bytes: 0
>      rx11_ecn_mark: 0
>      rx11_removed_vlan_packets: 20352190494
>      rx11_wqe_err: 0
>      rx11_mpwqe_filler_cqes: 0
>      rx11_mpwqe_filler_strides: 0
>      rx11_buff_alloc_err: 0
>      rx11_cqe_compress_blks: 501992147
>      rx11_cqe_compress_pkts: 1140398610
>      rx11_page_reuse: 0
>      rx11_cache_reuse: 10071721531
>      rx11_cache_full: 104371621
>      rx11_cache_empty: 37376
>      rx11_cache_busy: 104373697
>      rx11_cache_waive: 0
>      rx11_congst_umr: 0
>      rx11_arfs_err: 0
>      rx11_xdp_tx_xmit: 0
>      rx11_xdp_tx_full: 0
>      rx11_xdp_tx_err: 0
>      rx11_xdp_tx_cqes: 0
>      rx12_packets: 19934747149
>      rx12_bytes: 26296478787829
>      rx12_csum_complete: 19934747149
>      rx12_csum_unnecessary: 0
>      rx12_csum_unnecessary_inner: 0
>      rx12_csum_none: 0
>      rx12_xdp_drop: 0
>      rx12_xdp_redirect: 0
>      rx12_lro_packets: 0
>      rx12_lro_bytes: 0
>      rx12_ecn_mark: 0
>      rx12_removed_vlan_packets: 19934747149
>      rx12_wqe_err: 0
>      rx12_mpwqe_filler_cqes: 0
>      rx12_mpwqe_filler_strides: 0
>      rx12_buff_alloc_err: 0
>      rx12_cqe_compress_blks: 443350570
>      rx12_cqe_compress_pkts: 995997220
>      rx12_page_reuse: 0
>      rx12_cache_reuse: 9864934174
>      rx12_cache_full: 102437428
>      rx12_cache_empty: 37376
>      rx12_cache_busy: 102439382
>      rx12_cache_waive: 0
>      rx12_congst_umr: 0
>      rx12_arfs_err: 0
>      rx12_xdp_tx_xmit: 0
>      rx12_xdp_tx_full: 0
>      rx12_xdp_tx_err: 0
>      rx12_xdp_tx_cqes: 0
>      rx13_packets: 19866908096
>      rx13_bytes: 26160931936186
>      rx13_csum_complete: 19866908096
>      rx13_csum_unnecessary: 0
>      rx13_csum_unnecessary_inner: 0
>      rx13_csum_none: 0
>      rx13_xdp_drop: 0
>      rx13_xdp_redirect: 0
>      rx13_lro_packets: 0
>      rx13_lro_bytes: 0
>      rx13_ecn_mark: 0
>      rx13_removed_vlan_packets: 19866908096
>      rx13_wqe_err: 0
>      rx13_mpwqe_filler_cqes: 0
>      rx13_mpwqe_filler_strides: 0
>      rx13_buff_alloc_err: 0
>      rx13_cqe_compress_blks: 413640141
>      rx13_cqe_compress_pkts: 926175066
>      rx13_page_reuse: 0
>      rx13_cache_reuse: 36358610
>      rx13_cache_full: 9897092921
>      rx13_cache_empty: 37376
>      rx13_cache_busy: 9897095422
>      rx13_cache_waive: 0
>      rx13_congst_umr: 0
>      rx13_arfs_err: 0
>      rx13_xdp_tx_xmit: 0
>      rx13_xdp_tx_full: 0
>      rx13_xdp_tx_err: 0
>      rx13_xdp_tx_cqes: 0
>      rx14_packets: 20229035746
>      rx14_bytes: 26655092809172
>      rx14_csum_complete: 20229035746
>      rx14_csum_unnecessary: 0
>      rx14_csum_unnecessary_inner: 0
>      rx14_csum_none: 0
>      rx14_xdp_drop: 0
>      rx14_xdp_redirect: 0
>      rx14_lro_packets: 0
>      rx14_lro_bytes: 0
>      rx14_ecn_mark: 0
>      rx14_removed_vlan_packets: 20229035746
>      rx14_wqe_err: 0
>      rx14_mpwqe_filler_cqes: 0
>      rx14_mpwqe_filler_strides: 0
>      rx14_buff_alloc_err: 0
>      rx14_cqe_compress_blks: 460990337
>      rx14_cqe_compress_pkts: 1041287948
>      rx14_page_reuse: 0
>      rx14_cache_reuse: 25649275
>      rx14_cache_full: 10088866045
>      rx14_cache_empty: 37376
>      rx14_cache_busy: 10088868574
>      rx14_cache_waive: 0
>      rx14_congst_umr: 0
>      rx14_arfs_err: 0
>      rx14_xdp_tx_xmit: 0
>      rx14_xdp_tx_full: 0
>      rx14_xdp_tx_err: 0
>      rx14_xdp_tx_cqes: 0
>      rx15_packets: 20528177154
>      rx15_bytes: 27029263893264
>      rx15_csum_complete: 20528177154
>      rx15_csum_unnecessary: 0
>      rx15_csum_unnecessary_inner: 0
>      rx15_csum_none: 0
>      rx15_xdp_drop: 0
>      rx15_xdp_redirect: 0
>      rx15_lro_packets: 0
>      rx15_lro_bytes: 0
>      rx15_ecn_mark: 0
>      rx15_removed_vlan_packets: 20528177154
>      rx15_wqe_err: 0
>      rx15_mpwqe_filler_cqes: 0
>      rx15_mpwqe_filler_strides: 0
>      rx15_buff_alloc_err: 0
>      rx15_cqe_compress_blks: 476776176
>      rx15_cqe_compress_pkts: 1076153263
>      rx15_page_reuse: 0
>      rx15_cache_reuse: 48426735
>      rx15_cache_full: 10215659289
>      rx15_cache_empty: 37376
>      rx15_cache_busy: 10215661817
>      rx15_cache_waive: 0
>      rx15_congst_umr: 0
>      rx15_arfs_err: 0
>      rx15_xdp_tx_xmit: 0
>      rx15_xdp_tx_full: 0
>      rx15_xdp_tx_err: 0
>      rx15_xdp_tx_cqes: 0
>      rx16_packets: 16104078098
>      rx16_bytes: 21256361789679
>      rx16_csum_complete: 16104078098
>      rx16_csum_unnecessary: 0
>      rx16_csum_unnecessary_inner: 0
>      rx16_csum_none: 0
>      rx16_xdp_drop: 0
>      rx16_xdp_redirect: 0
>      rx16_lro_packets: 0
>      rx16_lro_bytes: 0
>      rx16_ecn_mark: 0
>      rx16_removed_vlan_packets: 16104078098
>      rx16_wqe_err: 0
>      rx16_mpwqe_filler_cqes: 0
>      rx16_mpwqe_filler_strides: 0
>      rx16_buff_alloc_err: 0
>      rx16_cqe_compress_blks: 352082054
>      rx16_cqe_compress_pkts: 787161670
>      rx16_page_reuse: 0
>      rx16_cache_reuse: 25912567
>      rx16_cache_full: 8026124051
>      rx16_cache_empty: 37376
>      rx16_cache_busy: 8026126465
>      rx16_cache_waive: 0
>      rx16_congst_umr: 0
>      rx16_arfs_err: 0
>      rx16_xdp_tx_xmit: 0
>      rx16_xdp_tx_full: 0
>      rx16_xdp_tx_err: 0
>      rx16_xdp_tx_cqes: 0
>      rx17_packets: 16314055017
>      rx17_bytes: 21589139030173
>      rx17_csum_complete: 16314055017
>      rx17_csum_unnecessary: 0
>      rx17_csum_unnecessary_inner: 0
>      rx17_csum_none: 0
>      rx17_xdp_drop: 0
>      rx17_xdp_redirect: 0
>      rx17_lro_packets: 0
>      rx17_lro_bytes: 0
>      rx17_ecn_mark: 0
>      rx17_removed_vlan_packets: 16314055017
>      rx17_wqe_err: 0
>      rx17_mpwqe_filler_cqes: 0
>      rx17_mpwqe_filler_strides: 0
>      rx17_buff_alloc_err: 0
>      rx17_cqe_compress_blks: 387834541
>      rx17_cqe_compress_pkts: 871851081
>      rx17_page_reuse: 0
>      rx17_cache_reuse: 24021313
>      rx17_cache_full: 8133003829
>      rx17_cache_empty: 37376
>      rx17_cache_busy: 8133006175
>      rx17_cache_waive: 0
>      rx17_congst_umr: 0
>      rx17_arfs_err: 0
>      rx17_xdp_tx_xmit: 0
>      rx17_xdp_tx_full: 0
>      rx17_xdp_tx_err: 0
>      rx17_xdp_tx_cqes: 0
>      rx18_packets: 16439016814
>      rx18_bytes: 21648651917475
>      rx18_csum_complete: 16439016814
>      rx18_csum_unnecessary: 0
>      rx18_csum_unnecessary_inner: 0
>      rx18_csum_none: 0
>      rx18_xdp_drop: 0
>      rx18_xdp_redirect: 0
>      rx18_lro_packets: 0
>      rx18_lro_bytes: 0
>      rx18_ecn_mark: 0
>      rx18_removed_vlan_packets: 16439016814
>      rx18_wqe_err: 0
>      rx18_mpwqe_filler_cqes: 0
>      rx18_mpwqe_filler_strides: 0
>      rx18_buff_alloc_err: 0
>      rx18_cqe_compress_blks: 375066666
>      rx18_cqe_compress_pkts: 843563974
>      rx18_page_reuse: 0
>      rx18_cache_reuse: 8151064266
>      rx18_cache_full: 68442025
>      rx18_cache_empty: 37376
>      rx18_cache_busy: 68444122
>      rx18_cache_waive: 0
>      rx18_congst_umr: 0
>      rx18_arfs_err: 0
>      rx18_xdp_tx_xmit: 0
>      rx18_xdp_tx_full: 0
>      rx18_xdp_tx_err: 0
>      rx18_xdp_tx_cqes: 0
>      rx19_packets: 16641223506
>      rx19_bytes: 21964749940935
>      rx19_csum_complete: 16641223506
>      rx19_csum_unnecessary: 0
>      rx19_csum_unnecessary_inner: 0
>      rx19_csum_none: 0
>      rx19_xdp_drop: 0
>      rx19_xdp_redirect: 0
>      rx19_lro_packets: 0
>      rx19_lro_bytes: 0
>      rx19_ecn_mark: 0
>      rx19_removed_vlan_packets: 16641223506
>      rx19_wqe_err: 0
>      rx19_mpwqe_filler_cqes: 0
>      rx19_mpwqe_filler_strides: 0
>      rx19_buff_alloc_err: 0
>      rx19_cqe_compress_blks: 387825932
>      rx19_cqe_compress_pkts: 872266355
>      rx19_page_reuse: 0
>      rx19_cache_reuse: 116433620
>      rx19_cache_full: 8204175954
>      rx19_cache_empty: 37376
>      rx19_cache_busy: 8204178120
>      rx19_cache_waive: 0
>      rx19_congst_umr: 0
>      rx19_arfs_err: 0
>      rx19_xdp_tx_xmit: 0
>      rx19_xdp_tx_full: 0
>      rx19_xdp_tx_err: 0
>      rx19_xdp_tx_cqes: 0
>      rx20_packets: 16206927741
>      rx20_bytes: 21387447038430
>      rx20_csum_complete: 16206927741
>      rx20_csum_unnecessary: 0
>      rx20_csum_unnecessary_inner: 0
>      rx20_csum_none: 0
>      rx20_xdp_drop: 0
>      rx20_xdp_redirect: 0
>      rx20_lro_packets: 0
>      rx20_lro_bytes: 0
>      rx20_ecn_mark: 0
>      rx20_removed_vlan_packets: 16206927741
>      rx20_wqe_err: 0
>      rx20_mpwqe_filler_cqes: 0
>      rx20_mpwqe_filler_strides: 0
>      rx20_buff_alloc_err: 0
>      rx20_cqe_compress_blks: 370144620
>      rx20_cqe_compress_pkts: 829122671
>      rx20_page_reuse: 0
>      rx20_cache_reuse: 8053733744
>      rx20_cache_full: 49728026
>      rx20_cache_empty: 37376
>      rx20_cache_busy: 49730116
>      rx20_cache_waive: 0
>      rx20_congst_umr: 0
>      rx20_arfs_err: 0
>      rx20_xdp_tx_xmit: 0
>      rx20_xdp_tx_full: 0
>      rx20_xdp_tx_err: 0
>      rx20_xdp_tx_cqes: 0
>      rx21_packets: 16562361314
>      rx21_bytes: 21856653284356
>      rx21_csum_complete: 16562361314
>      rx21_csum_unnecessary: 0
>      rx21_csum_unnecessary_inner: 0
>      rx21_csum_none: 0
>      rx21_xdp_drop: 0
>      rx21_xdp_redirect: 0
>      rx21_lro_packets: 0
>      rx21_lro_bytes: 0
>      rx21_ecn_mark: 0
>      rx21_removed_vlan_packets: 16562361314
>      rx21_wqe_err: 0
>      rx21_mpwqe_filler_cqes: 0
>      rx21_mpwqe_filler_strides: 0
>      rx21_buff_alloc_err: 0
>      rx21_cqe_compress_blks: 350790425
>      rx21_cqe_compress_pkts: 783850729
>      rx21_page_reuse: 0
>      rx21_cache_reuse: 28077493
>      rx21_cache_full: 8253100706
>      rx21_cache_empty: 37376
>      rx21_cache_busy: 8253103147
>      rx21_cache_waive: 0
>      rx21_congst_umr: 0
>      rx21_arfs_err: 0
>      rx21_xdp_tx_xmit: 0
>      rx21_xdp_tx_full: 0
>      rx21_xdp_tx_err: 0
>      rx21_xdp_tx_cqes: 0
>      rx22_packets: 16350307571
>      rx22_bytes: 21408575325592
>      rx22_csum_complete: 16350307571
>      rx22_csum_unnecessary: 0
>      rx22_csum_unnecessary_inner: 0
>      rx22_csum_none: 0
>      rx22_xdp_drop: 0
>      rx22_xdp_redirect: 0
>      rx22_lro_packets: 0
>      rx22_lro_bytes: 0
>      rx22_ecn_mark: 0
>      rx22_removed_vlan_packets: 16350307571
>      rx22_wqe_err: 0
>      rx22_mpwqe_filler_cqes: 0
>      rx22_mpwqe_filler_strides: 0
>      rx22_buff_alloc_err: 0
>      rx22_cqe_compress_blks: 353531065
>      rx22_cqe_compress_pkts: 790814415
>      rx22_page_reuse: 0
>      rx22_cache_reuse: 16934343
>      rx22_cache_full: 8158216889
>      rx22_cache_empty: 37376
>      rx22_cache_busy: 8158219417
>      rx22_cache_waive: 0
>      rx22_congst_umr: 0
>      rx22_arfs_err: 0
>      rx22_xdp_tx_xmit: 0
>      rx22_xdp_tx_full: 0
>      rx22_xdp_tx_err: 0
>      rx22_xdp_tx_cqes: 0
>      rx23_packets: 16019811764
>      rx23_bytes: 21137182570985
>      rx23_csum_complete: 16019811764
>      rx23_csum_unnecessary: 0
>      rx23_csum_unnecessary_inner: 0
>      rx23_csum_none: 0
>      rx23_xdp_drop: 0
>      rx23_xdp_redirect: 0
>      rx23_lro_packets: 0
>      rx23_lro_bytes: 0
>      rx23_ecn_mark: 0
>      rx23_removed_vlan_packets: 16019811764
>      rx23_wqe_err: 0
>      rx23_mpwqe_filler_cqes: 0
>      rx23_mpwqe_filler_strides: 0
>      rx23_buff_alloc_err: 0
>      rx23_cqe_compress_blks: 349733033
>      rx23_cqe_compress_pkts: 781248862
>      rx23_page_reuse: 0
>      rx23_cache_reuse: 33422343
>      rx23_cache_full: 7976481152
>      rx23_cache_empty: 37376
>      rx23_cache_busy: 7976483525
>      rx23_cache_waive: 0
>      rx23_congst_umr: 0
>      rx23_arfs_err: 0
>      rx23_xdp_tx_xmit: 0
>      rx23_xdp_tx_full: 0
>      rx23_xdp_tx_err: 0
>      rx23_xdp_tx_cqes: 0
>      rx24_packets: 16212040646
>      rx24_bytes: 21393399325700
>      rx24_csum_complete: 16212040646
>      rx24_csum_unnecessary: 0
>      rx24_csum_unnecessary_inner: 0
>      rx24_csum_none: 0
>      rx24_xdp_drop: 0
>      rx24_xdp_redirect: 0
>      rx24_lro_packets: 0
>      rx24_lro_bytes: 0
>      rx24_ecn_mark: 0
>      rx24_removed_vlan_packets: 16212040646
>      rx24_wqe_err: 0
>      rx24_mpwqe_filler_cqes: 0
>      rx24_mpwqe_filler_strides: 0
>      rx24_buff_alloc_err: 0
>      rx24_cqe_compress_blks: 379833752
>      rx24_cqe_compress_pkts: 852020179
>      rx24_page_reuse: 0
>      rx24_cache_reuse: 8033552512
>      rx24_cache_full: 72465843
>      rx24_cache_empty: 37376
>      rx24_cache_busy: 72467789
>      rx24_cache_waive: 0
>      rx24_congst_umr: 0
>      rx24_arfs_err: 0
>      rx24_xdp_tx_xmit: 0
>      rx24_xdp_tx_full: 0
>      rx24_xdp_tx_err: 0
>      rx24_xdp_tx_cqes: 0
>      rx25_packets: 16412186257
>      rx25_bytes: 21651198388407
>      rx25_csum_complete: 16412186257
>      rx25_csum_unnecessary: 0
>      rx25_csum_unnecessary_inner: 0
>      rx25_csum_none: 0
>      rx25_xdp_drop: 0
>      rx25_xdp_redirect: 0
>      rx25_lro_packets: 0
>      rx25_lro_bytes: 0
>      rx25_ecn_mark: 0
>      rx25_removed_vlan_packets: 16412186257
>      rx25_wqe_err: 0
>      rx25_mpwqe_filler_cqes: 0
>      rx25_mpwqe_filler_strides: 0
>      rx25_buff_alloc_err: 0
>      rx25_cqe_compress_blks: 383979685
>      rx25_cqe_compress_pkts: 861985772
>      rx25_page_reuse: 0
>      rx25_cache_reuse: 8129807841
>      rx25_cache_full: 76283342
>      rx25_cache_empty: 37376
>      rx25_cache_busy: 76285271
>      rx25_cache_waive: 0
>      rx25_congst_umr: 0
>      rx25_arfs_err: 0
>      rx25_xdp_tx_xmit: 0
>      rx25_xdp_tx_full: 0
>      rx25_xdp_tx_err: 0
>      rx25_xdp_tx_cqes: 0
>      rx26_packets: 16304310003
>      rx26_bytes: 21571217538721
>      rx26_csum_complete: 16304310003
>      rx26_csum_unnecessary: 0
>      rx26_csum_unnecessary_inner: 0
>      rx26_csum_none: 0
>      rx26_xdp_drop: 0
>      rx26_xdp_redirect: 0
>      rx26_lro_packets: 0
>      rx26_lro_bytes: 0
>      rx26_ecn_mark: 0
>      rx26_removed_vlan_packets: 16304310003
>      rx26_wqe_err: 0
>      rx26_mpwqe_filler_cqes: 0
>      rx26_mpwqe_filler_strides: 0
>      rx26_buff_alloc_err: 0
>      rx26_cqe_compress_blks: 353314041
>      rx26_cqe_compress_pkts: 788838424
>      rx26_page_reuse: 0
>      rx26_cache_reuse: 19673790
>      rx26_cache_full: 8132478659
>      rx26_cache_empty: 37376
>      rx26_cache_busy: 8132481198
>      rx26_cache_waive: 0
>      rx26_congst_umr: 0
>      rx26_arfs_err: 0
>      rx26_xdp_tx_xmit: 0
>      rx26_xdp_tx_full: 0
>      rx26_xdp_tx_err: 0
>      rx26_xdp_tx_cqes: 0
>      rx27_packets: 16171856079
>      rx27_bytes: 21376891736540
>      rx27_csum_complete: 16171856079
>      rx27_csum_unnecessary: 0
>      rx27_csum_unnecessary_inner: 0
>      rx27_csum_none: 0
>      rx27_xdp_drop: 0
>      rx27_xdp_redirect: 0
>      rx27_lro_packets: 0
>      rx27_lro_bytes: 0
>      rx27_ecn_mark: 0
>      rx27_removed_vlan_packets: 16171856079
>      rx27_wqe_err: 0
>      rx27_mpwqe_filler_cqes: 0
>      rx27_mpwqe_filler_strides: 0
>      rx27_buff_alloc_err: 0
>      rx27_cqe_compress_blks: 386632845
>      rx27_cqe_compress_pkts: 869362576
>      rx27_page_reuse: 0
>      rx27_cache_reuse: 10070560
>      rx27_cache_full: 8075854928
>      rx27_cache_empty: 37376
>      rx27_cache_busy: 8075857468
>      rx27_cache_waive: 0
>      rx27_congst_umr: 0
>      rx27_arfs_err: 0
>      rx27_xdp_tx_xmit: 0
>      rx27_xdp_tx_full: 0
>      rx27_xdp_tx_err: 0
>      rx27_xdp_tx_cqes: 0
>      rx28_packets: 0
>      rx28_bytes: 0
>      rx28_csum_complete: 0
>      rx28_csum_unnecessary: 0
>      rx28_csum_unnecessary_inner: 0
>      rx28_csum_none: 0
>      rx28_xdp_drop: 0
>      rx28_xdp_redirect: 0
>      rx28_lro_packets: 0
>      rx28_lro_bytes: 0
>      rx28_ecn_mark: 0
>      rx28_removed_vlan_packets: 0
>      rx28_wqe_err: 0
>      rx28_mpwqe_filler_cqes: 0
>      rx28_mpwqe_filler_strides: 0
>      rx28_buff_alloc_err: 0
>      rx28_cqe_compress_blks: 0
>      rx28_cqe_compress_pkts: 0
>      rx28_page_reuse: 0
>      rx28_cache_reuse: 0
>      rx28_cache_full: 0
>      rx28_cache_empty: 2560
>      rx28_cache_busy: 0
>      rx28_cache_waive: 0
>      rx28_congst_umr: 0
>      rx28_arfs_err: 0
>      rx28_xdp_tx_xmit: 0
>      rx28_xdp_tx_full: 0
>      rx28_xdp_tx_err: 0
>      rx28_xdp_tx_cqes: 0
>      rx29_packets: 0
>      rx29_bytes: 0
>      rx29_csum_complete: 0
>      rx29_csum_unnecessary: 0
>      rx29_csum_unnecessary_inner: 0
>      rx29_csum_none: 0
>      rx29_xdp_drop: 0
>      rx29_xdp_redirect: 0
>      rx29_lro_packets: 0
>      rx29_lro_bytes: 0
>      rx29_ecn_mark: 0
>      rx29_removed_vlan_packets: 0
>      rx29_wqe_err: 0
>      rx29_mpwqe_filler_cqes: 0
>      rx29_mpwqe_filler_strides: 0
>      rx29_buff_alloc_err: 0
>      rx29_cqe_compress_blks: 0
>      rx29_cqe_compress_pkts: 0
>      rx29_page_reuse: 0
>      rx29_cache_reuse: 0
>      rx29_cache_full: 0
>      rx29_cache_empty: 2560
>      rx29_cache_busy: 0
>      rx29_cache_waive: 0
>      rx29_congst_umr: 0
>      rx29_arfs_err: 0
>      rx29_xdp_tx_xmit: 0
>      rx29_xdp_tx_full: 0
>      rx29_xdp_tx_err: 0
>      rx29_xdp_tx_cqes: 0
>      rx30_packets: 0
>      rx30_bytes: 0
>      rx30_csum_complete: 0
>      rx30_csum_unnecessary: 0
>      rx30_csum_unnecessary_inner: 0
>      rx30_csum_none: 0
>      rx30_xdp_drop: 0
>      rx30_xdp_redirect: 0
>      rx30_lro_packets: 0
>      rx30_lro_bytes: 0
>      rx30_ecn_mark: 0
>      rx30_removed_vlan_packets: 0
>      rx30_wqe_err: 0
>      rx30_mpwqe_filler_cqes: 0
>      rx30_mpwqe_filler_strides: 0
>      rx30_buff_alloc_err: 0
>      rx30_cqe_compress_blks: 0
>      rx30_cqe_compress_pkts: 0
>      rx30_page_reuse: 0
>      rx30_cache_reuse: 0
>      rx30_cache_full: 0
>      rx30_cache_empty: 2560
>      rx30_cache_busy: 0
>      rx30_cache_waive: 0
>      rx30_congst_umr: 0
>      rx30_arfs_err: 0
>      rx30_xdp_tx_xmit: 0
>      rx30_xdp_tx_full: 0
>      rx30_xdp_tx_err: 0
>      rx30_xdp_tx_cqes: 0
>      rx31_packets: 0
>      rx31_bytes: 0
>      rx31_csum_complete: 0
>      rx31_csum_unnecessary: 0
>      rx31_csum_unnecessary_inner: 0
>      rx31_csum_none: 0
>      rx31_xdp_drop: 0
>      rx31_xdp_redirect: 0
>      rx31_lro_packets: 0
>      rx31_lro_bytes: 0
>      rx31_ecn_mark: 0
>      rx31_removed_vlan_packets: 0
>      rx31_wqe_err: 0
>      rx31_mpwqe_filler_cqes: 0
>      rx31_mpwqe_filler_strides: 0
>      rx31_buff_alloc_err: 0
>      rx31_cqe_compress_blks: 0
>      rx31_cqe_compress_pkts: 0
>      rx31_page_reuse: 0
>      rx31_cache_reuse: 0
>      rx31_cache_full: 0
>      rx31_cache_empty: 2560
>      rx31_cache_busy: 0
>      rx31_cache_waive: 0
>      rx31_congst_umr: 0
>      rx31_arfs_err: 0
>      rx31_xdp_tx_xmit: 0
>      rx31_xdp_tx_full: 0
>      rx31_xdp_tx_err: 0
>      rx31_xdp_tx_cqes: 0
>      rx32_packets: 0
>      rx32_bytes: 0
>      rx32_csum_complete: 0
>      rx32_csum_unnecessary: 0
>      rx32_csum_unnecessary_inner: 0
>      rx32_csum_none: 0
>      rx32_xdp_drop: 0
>      rx32_xdp_redirect: 0
>      rx32_lro_packets: 0
>      rx32_lro_bytes: 0
>      rx32_ecn_mark: 0
>      rx32_removed_vlan_packets: 0
>      rx32_wqe_err: 0
>      rx32_mpwqe_filler_cqes: 0
>      rx32_mpwqe_filler_strides: 0
>      rx32_buff_alloc_err: 0
>      rx32_cqe_compress_blks: 0
>      rx32_cqe_compress_pkts: 0
>      rx32_page_reuse: 0
>      rx32_cache_reuse: 0
>      rx32_cache_full: 0
>      rx32_cache_empty: 2560
>      rx32_cache_busy: 0
>      rx32_cache_waive: 0
>      rx32_congst_umr: 0
>      rx32_arfs_err: 0
>      rx32_xdp_tx_xmit: 0
>      rx32_xdp_tx_full: 0
>      rx32_xdp_tx_err: 0
>      rx32_xdp_tx_cqes: 0
>      rx33_packets: 0
>      rx33_bytes: 0
>      rx33_csum_complete: 0
>      rx33_csum_unnecessary: 0
>      rx33_csum_unnecessary_inner: 0
>      rx33_csum_none: 0
>      rx33_xdp_drop: 0
>      rx33_xdp_redirect: 0
>      rx33_lro_packets: 0
>      rx33_lro_bytes: 0
>      rx33_ecn_mark: 0
>      rx33_removed_vlan_packets: 0
>      rx33_wqe_err: 0
>      rx33_mpwqe_filler_cqes: 0
>      rx33_mpwqe_filler_strides: 0
>      rx33_buff_alloc_err: 0
>      rx33_cqe_compress_blks: 0
>      rx33_cqe_compress_pkts: 0
>      rx33_page_reuse: 0
>      rx33_cache_reuse: 0
>      rx33_cache_full: 0
>      rx33_cache_empty: 2560
>      rx33_cache_busy: 0
>      rx33_cache_waive: 0
>      rx33_congst_umr: 0
>      rx33_arfs_err: 0
>      rx33_xdp_tx_xmit: 0
>      rx33_xdp_tx_full: 0
>      rx33_xdp_tx_err: 0
>      rx33_xdp_tx_cqes: 0
>      rx34_packets: 0
>      rx34_bytes: 0
>      rx34_csum_complete: 0
>      rx34_csum_unnecessary: 0
>      rx34_csum_unnecessary_inner: 0
>      rx34_csum_none: 0
>      rx34_xdp_drop: 0
>      rx34_xdp_redirect: 0
>      rx34_lro_packets: 0
>      rx34_lro_bytes: 0
>      rx34_ecn_mark: 0
>      rx34_removed_vlan_packets: 0
>      rx34_wqe_err: 0
>      rx34_mpwqe_filler_cqes: 0
>      rx34_mpwqe_filler_strides: 0
>      rx34_buff_alloc_err: 0
>      rx34_cqe_compress_blks: 0
>      rx34_cqe_compress_pkts: 0
>      rx34_page_reuse: 0
>      rx34_cache_reuse: 0
>      rx34_cache_full: 0
>      rx34_cache_empty: 2560
>      rx34_cache_busy: 0
>      rx34_cache_waive: 0
>      rx34_congst_umr: 0
>      rx34_arfs_err: 0
>      rx34_xdp_tx_xmit: 0
>      rx34_xdp_tx_full: 0
>      rx34_xdp_tx_err: 0
>      rx34_xdp_tx_cqes: 0
>      rx35_packets: 0
>      rx35_bytes: 0
>      rx35_csum_complete: 0
>      rx35_csum_unnecessary: 0
>      rx35_csum_unnecessary_inner: 0
>      rx35_csum_none: 0
>      rx35_xdp_drop: 0
>      rx35_xdp_redirect: 0
>      rx35_lro_packets: 0
>      rx35_lro_bytes: 0
>      rx35_ecn_mark: 0
>      rx35_removed_vlan_packets: 0
>      rx35_wqe_err: 0
>      rx35_mpwqe_filler_cqes: 0
>      rx35_mpwqe_filler_strides: 0
>      rx35_buff_alloc_err: 0
>      rx35_cqe_compress_blks: 0
>      rx35_cqe_compress_pkts: 0
>      rx35_page_reuse: 0
>      rx35_cache_reuse: 0
>      rx35_cache_full: 0
>      rx35_cache_empty: 2560
>      rx35_cache_busy: 0
>      rx35_cache_waive: 0
>      rx35_congst_umr: 0
>      rx35_arfs_err: 0
>      rx35_xdp_tx_xmit: 0
>      rx35_xdp_tx_full: 0
>      rx35_xdp_tx_err: 0
>      rx35_xdp_tx_cqes: 0
>      rx36_packets: 0
>      rx36_bytes: 0
>      rx36_csum_complete: 0
>      rx36_csum_unnecessary: 0
>      rx36_csum_unnecessary_inner: 0
>      rx36_csum_none: 0
>      rx36_xdp_drop: 0
>      rx36_xdp_redirect: 0
>      rx36_lro_packets: 0
>      rx36_lro_bytes: 0
>      rx36_ecn_mark: 0
>      rx36_removed_vlan_packets: 0
>      rx36_wqe_err: 0
>      rx36_mpwqe_filler_cqes: 0
>      rx36_mpwqe_filler_strides: 0
>      rx36_buff_alloc_err: 0
>      rx36_cqe_compress_blks: 0
>      rx36_cqe_compress_pkts: 0
>      rx36_page_reuse: 0
>      rx36_cache_reuse: 0
>      rx36_cache_full: 0
>      rx36_cache_empty: 2560
>      rx36_cache_busy: 0
>      rx36_cache_waive: 0
>      rx36_congst_umr: 0
>      rx36_arfs_err: 0
>      rx36_xdp_tx_xmit: 0
>      rx36_xdp_tx_full: 0
>      rx36_xdp_tx_err: 0
>      rx36_xdp_tx_cqes: 0
>      rx37_packets: 0
>      rx37_bytes: 0
>      rx37_csum_complete: 0
>      rx37_csum_unnecessary: 0
>      rx37_csum_unnecessary_inner: 0
>      rx37_csum_none: 0
>      rx37_xdp_drop: 0
>      rx37_xdp_redirect: 0
>      rx37_lro_packets: 0
>      rx37_lro_bytes: 0
>      rx37_ecn_mark: 0
>      rx37_removed_vlan_packets: 0
>      rx37_wqe_err: 0
>      rx37_mpwqe_filler_cqes: 0
>      rx37_mpwqe_filler_strides: 0
>      rx37_buff_alloc_err: 0
>      rx37_cqe_compress_blks: 0
>      rx37_cqe_compress_pkts: 0
>      rx37_page_reuse: 0
>      rx37_cache_reuse: 0
>      rx37_cache_full: 0
>      rx37_cache_empty: 2560
>      rx37_cache_busy: 0
>      rx37_cache_waive: 0
>      rx37_congst_umr: 0
>      rx37_arfs_err: 0
>      rx37_xdp_tx_xmit: 0
>      rx37_xdp_tx_full: 0
>      rx37_xdp_tx_err: 0
>      rx37_xdp_tx_cqes: 0
>      rx38_packets: 0
>      rx38_bytes: 0
>      rx38_csum_complete: 0
>      rx38_csum_unnecessary: 0
>      rx38_csum_unnecessary_inner: 0
>      rx38_csum_none: 0
>      rx38_xdp_drop: 0
>      rx38_xdp_redirect: 0
>      rx38_lro_packets: 0
>      rx38_lro_bytes: 0
>      rx38_ecn_mark: 0
>      rx38_removed_vlan_packets: 0
>      rx38_wqe_err: 0
>      rx38_mpwqe_filler_cqes: 0
>      rx38_mpwqe_filler_strides: 0
>      rx38_buff_alloc_err: 0
>      rx38_cqe_compress_blks: 0
>      rx38_cqe_compress_pkts: 0
>      rx38_page_reuse: 0
>      rx38_cache_reuse: 0
>      rx38_cache_full: 0
>      rx38_cache_empty: 2560
>      rx38_cache_busy: 0
>      rx38_cache_waive: 0
>      rx38_congst_umr: 0
>      rx38_arfs_err: 0
>      rx38_xdp_tx_xmit: 0
>      rx38_xdp_tx_full: 0
>      rx38_xdp_tx_err: 0
>      rx38_xdp_tx_cqes: 0
>      rx39_packets: 0
>      rx39_bytes: 0
>      rx39_csum_complete: 0
>      rx39_csum_unnecessary: 0
>      rx39_csum_unnecessary_inner: 0
>      rx39_csum_none: 0
>      rx39_xdp_drop: 0
>      rx39_xdp_redirect: 0
>      rx39_lro_packets: 0
>      rx39_lro_bytes: 0
>      rx39_ecn_mark: 0
>      rx39_removed_vlan_packets: 0
>      rx39_wqe_err: 0
>      rx39_mpwqe_filler_cqes: 0
>      rx39_mpwqe_filler_strides: 0
>      rx39_buff_alloc_err: 0
>      rx39_cqe_compress_blks: 0
>      rx39_cqe_compress_pkts: 0
>      rx39_page_reuse: 0
>      rx39_cache_reuse: 0
>      rx39_cache_full: 0
>      rx39_cache_empty: 2560
>      rx39_cache_busy: 0
>      rx39_cache_waive: 0
>      rx39_congst_umr: 0
>      rx39_arfs_err: 0
>      rx39_xdp_tx_xmit: 0
>      rx39_xdp_tx_full: 0
>      rx39_xdp_tx_err: 0
>      rx39_xdp_tx_cqes: 0
>      rx40_packets: 0
>      rx40_bytes: 0
>      rx40_csum_complete: 0
>      rx40_csum_unnecessary: 0
>      rx40_csum_unnecessary_inner: 0
>      rx40_csum_none: 0
>      rx40_xdp_drop: 0
>      rx40_xdp_redirect: 0
>      rx40_lro_packets: 0
>      rx40_lro_bytes: 0
>      rx40_ecn_mark: 0
>      rx40_removed_vlan_packets: 0
>      rx40_wqe_err: 0
>      rx40_mpwqe_filler_cqes: 0
>      rx40_mpwqe_filler_strides: 0
>      rx40_buff_alloc_err: 0
>      rx40_cqe_compress_blks: 0
>      rx40_cqe_compress_pkts: 0
>      rx40_page_reuse: 0
>      rx40_cache_reuse: 0
>      rx40_cache_full: 0
>      rx40_cache_empty: 2560
>      rx40_cache_busy: 0
>      rx40_cache_waive: 0
>      rx40_congst_umr: 0
>      rx40_arfs_err: 0
>      rx40_xdp_tx_xmit: 0
>      rx40_xdp_tx_full: 0
>      rx40_xdp_tx_err: 0
>      rx40_xdp_tx_cqes: 0
>      rx41_packets: 0
>      rx41_bytes: 0
>      rx41_csum_complete: 0
>      rx41_csum_unnecessary: 0
>      rx41_csum_unnecessary_inner: 0
>      rx41_csum_none: 0
>      rx41_xdp_drop: 0
>      rx41_xdp_redirect: 0
>      rx41_lro_packets: 0
>      rx41_lro_bytes: 0
>      rx41_ecn_mark: 0
>      rx41_removed_vlan_packets: 0
>      rx41_wqe_err: 0
>      rx41_mpwqe_filler_cqes: 0
>      rx41_mpwqe_filler_strides: 0
>      rx41_buff_alloc_err: 0
>      rx41_cqe_compress_blks: 0
>      rx41_cqe_compress_pkts: 0
>      rx41_page_reuse: 0
>      rx41_cache_reuse: 0
>      rx41_cache_full: 0
>      rx41_cache_empty: 2560
>      rx41_cache_busy: 0
>      rx41_cache_waive: 0
>      rx41_congst_umr: 0
>      rx41_arfs_err: 0
>      rx41_xdp_tx_xmit: 0
>      rx41_xdp_tx_full: 0
>      rx41_xdp_tx_err: 0
>      rx41_xdp_tx_cqes: 0
>      rx42_packets: 0
>      rx42_bytes: 0
>      rx42_csum_complete: 0
>      rx42_csum_unnecessary: 0
>      rx42_csum_unnecessary_inner: 0
>      rx42_csum_none: 0
>      rx42_xdp_drop: 0
>      rx42_xdp_redirect: 0
>      rx42_lro_packets: 0
>      rx42_lro_bytes: 0
>      rx42_ecn_mark: 0
>      rx42_removed_vlan_packets: 0
>      rx42_wqe_err: 0
>      rx42_mpwqe_filler_cqes: 0
>      rx42_mpwqe_filler_strides: 0
>      rx42_buff_alloc_err: 0
>      rx42_cqe_compress_blks: 0
>      rx42_cqe_compress_pkts: 0
>      rx42_page_reuse: 0
>      rx42_cache_reuse: 0
>      rx42_cache_full: 0
>      rx42_cache_empty: 2560
>      rx42_cache_busy: 0
>      rx42_cache_waive: 0
>      rx42_congst_umr: 0
>      rx42_arfs_err: 0
>      rx42_xdp_tx_xmit: 0
>      rx42_xdp_tx_full: 0
>      rx42_xdp_tx_err: 0
>      rx42_xdp_tx_cqes: 0
>      rx43_packets: 0
>      rx43_bytes: 0
>      rx43_csum_complete: 0
>      rx43_csum_unnecessary: 0
>      rx43_csum_unnecessary_inner: 0
>      rx43_csum_none: 0
>      rx43_xdp_drop: 0
>      rx43_xdp_redirect: 0
>      rx43_lro_packets: 0
>      rx43_lro_bytes: 0
>      rx43_ecn_mark: 0
>      rx43_removed_vlan_packets: 0
>      rx43_wqe_err: 0
>      rx43_mpwqe_filler_cqes: 0
>      rx43_mpwqe_filler_strides: 0
>      rx43_buff_alloc_err: 0
>      rx43_cqe_compress_blks: 0
>      rx43_cqe_compress_pkts: 0
>      rx43_page_reuse: 0
>      rx43_cache_reuse: 0
>      rx43_cache_full: 0
>      rx43_cache_empty: 2560
>      rx43_cache_busy: 0
>      rx43_cache_waive: 0
>      rx43_congst_umr: 0
>      rx43_arfs_err: 0
>      rx43_xdp_tx_xmit: 0
>      rx43_xdp_tx_full: 0
>      rx43_xdp_tx_err: 0
>      rx43_xdp_tx_cqes: 0
>      rx44_packets: 0
>      rx44_bytes: 0
>      rx44_csum_complete: 0
>      rx44_csum_unnecessary: 0
>      rx44_csum_unnecessary_inner: 0
>      rx44_csum_none: 0
>      rx44_xdp_drop: 0
>      rx44_xdp_redirect: 0
>      rx44_lro_packets: 0
>      rx44_lro_bytes: 0
>      rx44_ecn_mark: 0
>      rx44_removed_vlan_packets: 0
>      rx44_wqe_err: 0
>      rx44_mpwqe_filler_cqes: 0
>      rx44_mpwqe_filler_strides: 0
>      rx44_buff_alloc_err: 0
>      rx44_cqe_compress_blks: 0
>      rx44_cqe_compress_pkts: 0
>      rx44_page_reuse: 0
>      rx44_cache_reuse: 0
>      rx44_cache_full: 0
>      rx44_cache_empty: 2560
>      rx44_cache_busy: 0
>      rx44_cache_waive: 0
>      rx44_congst_umr: 0
>      rx44_arfs_err: 0
>      rx44_xdp_tx_xmit: 0
>      rx44_xdp_tx_full: 0
>      rx44_xdp_tx_err: 0
>      rx44_xdp_tx_cqes: 0
>      rx45_packets: 0
>      rx45_bytes: 0
>      rx45_csum_complete: 0
>      rx45_csum_unnecessary: 0
>      rx45_csum_unnecessary_inner: 0
>      rx45_csum_none: 0
>      rx45_xdp_drop: 0
>      rx45_xdp_redirect: 0
>      rx45_lro_packets: 0
>      rx45_lro_bytes: 0
>      rx45_ecn_mark: 0
>      rx45_removed_vlan_packets: 0
>      rx45_wqe_err: 0
>      rx45_mpwqe_filler_cqes: 0
>      rx45_mpwqe_filler_strides: 0
>      rx45_buff_alloc_err: 0
>      rx45_cqe_compress_blks: 0
>      rx45_cqe_compress_pkts: 0
>      rx45_page_reuse: 0
>      rx45_cache_reuse: 0
>      rx45_cache_full: 0
>      rx45_cache_empty: 2560
>      rx45_cache_busy: 0
>      rx45_cache_waive: 0
>      rx45_congst_umr: 0
>      rx45_arfs_err: 0
>      rx45_xdp_tx_xmit: 0
>      rx45_xdp_tx_full: 0
>      rx45_xdp_tx_err: 0
>      rx45_xdp_tx_cqes: 0
>      rx46_packets: 0
>      rx46_bytes: 0
>      rx46_csum_complete: 0
>      rx46_csum_unnecessary: 0
>      rx46_csum_unnecessary_inner: 0
>      rx46_csum_none: 0
>      rx46_xdp_drop: 0
>      rx46_xdp_redirect: 0
>      rx46_lro_packets: 0
>      rx46_lro_bytes: 0
>      rx46_ecn_mark: 0
>      rx46_removed_vlan_packets: 0
>      rx46_wqe_err: 0
>      rx46_mpwqe_filler_cqes: 0
>      rx46_mpwqe_filler_strides: 0
>      rx46_buff_alloc_err: 0
>      rx46_cqe_compress_blks: 0
>      rx46_cqe_compress_pkts: 0
>      rx46_page_reuse: 0
>      rx46_cache_reuse: 0
>      rx46_cache_full: 0
>      rx46_cache_empty: 2560
>      rx46_cache_busy: 0
>      rx46_cache_waive: 0
>      rx46_congst_umr: 0
>      rx46_arfs_err: 0
>      rx46_xdp_tx_xmit: 0
>      rx46_xdp_tx_full: 0
>      rx46_xdp_tx_err: 0
>      rx46_xdp_tx_cqes: 0
>      rx47_packets: 0
>      rx47_bytes: 0
>      rx47_csum_complete: 0
>      rx47_csum_unnecessary: 0
>      rx47_csum_unnecessary_inner: 0
>      rx47_csum_none: 0
>      rx47_xdp_drop: 0
>      rx47_xdp_redirect: 0
>      rx47_lro_packets: 0
>      rx47_lro_bytes: 0
>      rx47_ecn_mark: 0
>      rx47_removed_vlan_packets: 0
>      rx47_wqe_err: 0
>      rx47_mpwqe_filler_cqes: 0
>      rx47_mpwqe_filler_strides: 0
>      rx47_buff_alloc_err: 0
>      rx47_cqe_compress_blks: 0
>      rx47_cqe_compress_pkts: 0
>      rx47_page_reuse: 0
>      rx47_cache_reuse: 0
>      rx47_cache_full: 0
>      rx47_cache_empty: 2560
>      rx47_cache_busy: 0
>      rx47_cache_waive: 0
>      rx47_congst_umr: 0
>      rx47_arfs_err: 0
>      rx47_xdp_tx_xmit: 0
>      rx47_xdp_tx_full: 0
>      rx47_xdp_tx_err: 0
>      rx47_xdp_tx_cqes: 0
>      rx48_packets: 0
>      rx48_bytes: 0
>      rx48_csum_complete: 0
>      rx48_csum_unnecessary: 0
>      rx48_csum_unnecessary_inner: 0
>      rx48_csum_none: 0
>      rx48_xdp_drop: 0
>      rx48_xdp_redirect: 0
>      rx48_lro_packets: 0
>      rx48_lro_bytes: 0
>      rx48_ecn_mark: 0
>      rx48_removed_vlan_packets: 0
>      rx48_wqe_err: 0
>      rx48_mpwqe_filler_cqes: 0
>      rx48_mpwqe_filler_strides: 0
>      rx48_buff_alloc_err: 0
>      rx48_cqe_compress_blks: 0
>      rx48_cqe_compress_pkts: 0
>      rx48_page_reuse: 0
>      rx48_cache_reuse: 0
>      rx48_cache_full: 0
>      rx48_cache_empty: 2560
>      rx48_cache_busy: 0
>      rx48_cache_waive: 0
>      rx48_congst_umr: 0
>      rx48_arfs_err: 0
>      rx48_xdp_tx_xmit: 0
>      rx48_xdp_tx_full: 0
>      rx48_xdp_tx_err: 0
>      rx48_xdp_tx_cqes: 0
>      rx49_packets: 0
>      rx49_bytes: 0
>      rx49_csum_complete: 0
>      rx49_csum_unnecessary: 0
>      rx49_csum_unnecessary_inner: 0
>      rx49_csum_none: 0
>      rx49_xdp_drop: 0
>      rx49_xdp_redirect: 0
>      rx49_lro_packets: 0
>      rx49_lro_bytes: 0
>      rx49_ecn_mark: 0
>      rx49_removed_vlan_packets: 0
>      rx49_wqe_err: 0
>      rx49_mpwqe_filler_cqes: 0
>      rx49_mpwqe_filler_strides: 0
>      rx49_buff_alloc_err: 0
>      rx49_cqe_compress_blks: 0
>      rx49_cqe_compress_pkts: 0
>      rx49_page_reuse: 0
>      rx49_cache_reuse: 0
>      rx49_cache_full: 0
>      rx49_cache_empty: 2560
>      rx49_cache_busy: 0
>      rx49_cache_waive: 0
>      rx49_congst_umr: 0
>      rx49_arfs_err: 0
>      rx49_xdp_tx_xmit: 0
>      rx49_xdp_tx_full: 0
>      rx49_xdp_tx_err: 0
>      rx49_xdp_tx_cqes: 0
>      rx50_packets: 0
>      rx50_bytes: 0
>      rx50_csum_complete: 0
>      rx50_csum_unnecessary: 0
>      rx50_csum_unnecessary_inner: 0
>      rx50_csum_none: 0
>      rx50_xdp_drop: 0
>      rx50_xdp_redirect: 0
>      rx50_lro_packets: 0
>      rx50_lro_bytes: 0
>      rx50_ecn_mark: 0
>      rx50_removed_vlan_packets: 0
>      rx50_wqe_err: 0
>      rx50_mpwqe_filler_cqes: 0
>      rx50_mpwqe_filler_strides: 0
>      rx50_buff_alloc_err: 0
>      rx50_cqe_compress_blks: 0
>      rx50_cqe_compress_pkts: 0
>      rx50_page_reuse: 0
>      rx50_cache_reuse: 0
>      rx50_cache_full: 0
>      rx50_cache_empty: 2560
>      rx50_cache_busy: 0
>      rx50_cache_waive: 0
>      rx50_congst_umr: 0
>      rx50_arfs_err: 0
>      rx50_xdp_tx_xmit: 0
>      rx50_xdp_tx_full: 0
>      rx50_xdp_tx_err: 0
>      rx50_xdp_tx_cqes: 0
>      rx51_packets: 0
>      rx51_bytes: 0
>      rx51_csum_complete: 0
>      rx51_csum_unnecessary: 0
>      rx51_csum_unnecessary_inner: 0
>      rx51_csum_none: 0
>      rx51_xdp_drop: 0
>      rx51_xdp_redirect: 0
>      rx51_lro_packets: 0
>      rx51_lro_bytes: 0
>      rx51_ecn_mark: 0
>      rx51_removed_vlan_packets: 0
>      rx51_wqe_err: 0
>      rx51_mpwqe_filler_cqes: 0
>      rx51_mpwqe_filler_strides: 0
>      rx51_buff_alloc_err: 0
>      rx51_cqe_compress_blks: 0
>      rx51_cqe_compress_pkts: 0
>      rx51_page_reuse: 0
>      rx51_cache_reuse: 0
>      rx51_cache_full: 0
>      rx51_cache_empty: 2560
>      rx51_cache_busy: 0
>      rx51_cache_waive: 0
>      rx51_congst_umr: 0
>      rx51_arfs_err: 0
>      rx51_xdp_tx_xmit: 0
>      rx51_xdp_tx_full: 0
>      rx51_xdp_tx_err: 0
>      rx51_xdp_tx_cqes: 0
>      rx52_packets: 0
>      rx52_bytes: 0
>      rx52_csum_complete: 0
>      rx52_csum_unnecessary: 0
>      rx52_csum_unnecessary_inner: 0
>      rx52_csum_none: 0
>      rx52_xdp_drop: 0
>      rx52_xdp_redirect: 0
>      rx52_lro_packets: 0
>      rx52_lro_bytes: 0
>      rx52_ecn_mark: 0
>      rx52_removed_vlan_packets: 0
>      rx52_wqe_err: 0
>      rx52_mpwqe_filler_cqes: 0
>      rx52_mpwqe_filler_strides: 0
>      rx52_buff_alloc_err: 0
>      rx52_cqe_compress_blks: 0
>      rx52_cqe_compress_pkts: 0
>      rx52_page_reuse: 0
>      rx52_cache_reuse: 0
>      rx52_cache_full: 0
>      rx52_cache_empty: 2560
>      rx52_cache_busy: 0
>      rx52_cache_waive: 0
>      rx52_congst_umr: 0
>      rx52_arfs_err: 0
>      rx52_xdp_tx_xmit: 0
>      rx52_xdp_tx_full: 0
>      rx52_xdp_tx_err: 0
>      rx52_xdp_tx_cqes: 0
>      rx53_packets: 0
>      rx53_bytes: 0
>      rx53_csum_complete: 0
>      rx53_csum_unnecessary: 0
>      rx53_csum_unnecessary_inner: 0
>      rx53_csum_none: 0
>      rx53_xdp_drop: 0
>      rx53_xdp_redirect: 0
>      rx53_lro_packets: 0
>      rx53_lro_bytes: 0
>      rx53_ecn_mark: 0
>      rx53_removed_vlan_packets: 0
>      rx53_wqe_err: 0
>      rx53_mpwqe_filler_cqes: 0
>      rx53_mpwqe_filler_strides: 0
>      rx53_buff_alloc_err: 0
>      rx53_cqe_compress_blks: 0
>      rx53_cqe_compress_pkts: 0
>      rx53_page_reuse: 0
>      rx53_cache_reuse: 0
>      rx53_cache_full: 0
>      rx53_cache_empty: 2560
>      rx53_cache_busy: 0
>      rx53_cache_waive: 0
>      rx53_congst_umr: 0
>      rx53_arfs_err: 0
>      rx53_xdp_tx_xmit: 0
>      rx53_xdp_tx_full: 0
>      rx53_xdp_tx_err: 0
>      rx53_xdp_tx_cqes: 0
>      rx54_packets: 0
>      rx54_bytes: 0
>      rx54_csum_complete: 0
>      rx54_csum_unnecessary: 0
>      rx54_csum_unnecessary_inner: 0
>      rx54_csum_none: 0
>      rx54_xdp_drop: 0
>      rx54_xdp_redirect: 0
>      rx54_lro_packets: 0
>      rx54_lro_bytes: 0
>      rx54_ecn_mark: 0
>      rx54_removed_vlan_packets: 0
>      rx54_wqe_err: 0
>      rx54_mpwqe_filler_cqes: 0
>      rx54_mpwqe_filler_strides: 0
>      rx54_buff_alloc_err: 0
>      rx54_cqe_compress_blks: 0
>      rx54_cqe_compress_pkts: 0
>      rx54_page_reuse: 0
>      rx54_cache_reuse: 0
>      rx54_cache_full: 0
>      rx54_cache_empty: 2560
>      rx54_cache_busy: 0
>      rx54_cache_waive: 0
>      rx54_congst_umr: 0
>      rx54_arfs_err: 0
>      rx54_xdp_tx_xmit: 0
>      rx54_xdp_tx_full: 0
>      rx54_xdp_tx_err: 0
>      rx54_xdp_tx_cqes: 0
>      rx55_packets: 0
>      rx55_bytes: 0
>      rx55_csum_complete: 0
>      rx55_csum_unnecessary: 0
>      rx55_csum_unnecessary_inner: 0
>      rx55_csum_none: 0
>      rx55_xdp_drop: 0
>      rx55_xdp_redirect: 0
>      rx55_lro_packets: 0
>      rx55_lro_bytes: 0
>      rx55_ecn_mark: 0
>      rx55_removed_vlan_packets: 0
>      rx55_wqe_err: 0
>      rx55_mpwqe_filler_cqes: 0
>      rx55_mpwqe_filler_strides: 0
>      rx55_buff_alloc_err: 0
>      rx55_cqe_compress_blks: 0
>      rx55_cqe_compress_pkts: 0
>      rx55_page_reuse: 0
>      rx55_cache_reuse: 0
>      rx55_cache_full: 0
>      rx55_cache_empty: 2560
>      rx55_cache_busy: 0
>      rx55_cache_waive: 0
>      rx55_congst_umr: 0
>      rx55_arfs_err: 0
>      rx55_xdp_tx_xmit: 0
>      rx55_xdp_tx_full: 0
>      rx55_xdp_tx_err: 0
>      rx55_xdp_tx_cqes: 0
>      tx0_packets: 24512439668
>      tx0_bytes: 15287569052791
>      tx0_tso_packets: 1536157106
>      tx0_tso_bytes: 8571753637944
>      tx0_tso_inner_packets: 0
>      tx0_tso_inner_bytes: 0
>      tx0_csum_partial: 2132156117
>      tx0_csum_partial_inner: 0
>      tx0_added_vlan_packets: 19906601448
>      tx0_nop: 308098536
>      tx0_csum_none: 17774445331
>      tx0_stopped: 19625
>      tx0_dropped: 0
>      tx0_xmit_more: 67864870
>      tx0_recover: 0
>      tx0_cqes: 19838744246
>      tx0_wake: 19624
>      tx0_cqe_err: 0
>      tx1_packets: 22598557053
>      tx1_bytes: 13568850145010
>      tx1_tso_packets: 1369529475
>      tx1_tso_bytes: 7661777265382
>      tx1_tso_inner_packets: 0
>      tx1_tso_inner_bytes: 0
>      tx1_csum_partial: 1884639496
>      tx1_csum_partial_inner: 0
>      tx1_added_vlan_packets: 18468333696
>      tx1_nop: 281301783
>      tx1_csum_none: 16583694200
>      tx1_stopped: 19457
>      tx1_dropped: 0
>      tx1_xmit_more: 55170875
>      tx1_recover: 0
>      tx1_cqes: 18413169824
>      tx1_wake: 19455
>      tx1_cqe_err: 0
>      tx2_packets: 22821611433
>      tx2_bytes: 13752535163683
>      tx2_tso_packets: 1396978825
>      tx2_tso_bytes: 7774704508463
>      tx2_tso_inner_packets: 0
>      tx2_tso_inner_bytes: 0
>      tx2_csum_partial: 1897834558
>      tx2_csum_partial_inner: 0
>      tx2_added_vlan_packets: 18641958085
>      tx2_nop: 286934891
>      tx2_csum_none: 16744123527
>      tx2_stopped: 13214
>      tx2_dropped: 0
>      tx2_xmit_more: 61749446
>      tx2_recover: 0
>      tx2_cqes: 18580215654
>      tx2_wake: 13214
>      tx2_cqe_err: 0
>      tx3_packets: 22580809948
>      tx3_bytes: 13730542936609
>      tx3_tso_packets: 1370434579
>      tx3_tso_bytes: 7605636711455
>      tx3_tso_inner_packets: 0
>      tx3_tso_inner_bytes: 0
>      tx3_csum_partial: 1865573748
>      tx3_csum_partial_inner: 0
>      tx3_added_vlan_packets: 18491873644
>      tx3_nop: 281195875
>      tx3_csum_none: 16626299896
>      tx3_stopped: 12542
>      tx3_dropped: 0
>      tx3_xmit_more: 57681647
>      tx3_recover: 0
>      tx3_cqes: 18434198757
>      tx3_wake: 12540
>      tx3_cqe_err: 0
>      tx4_packets: 27801801208
>      tx4_bytes: 17058453171137
>      tx4_tso_packets: 1740500105
>      tx4_tso_bytes: 9474905622036
>      tx4_tso_inner_packets: 0
>      tx4_tso_inner_bytes: 0
>      tx4_csum_partial: 2279225376
>      tx4_csum_partial_inner: 0
>      tx4_added_vlan_packets: 22744081633
>      tx4_nop: 349753979
>      tx4_csum_none: 20464856257
>      tx4_stopped: 14816
>      tx4_dropped: 0
>      tx4_xmit_more: 65469322
>      tx4_recover: 0
>      tx4_cqes: 22678618972
>      tx4_wake: 14816
>      tx4_cqe_err: 0
>      tx5_packets: 25099783024
>      tx5_bytes: 14917740698381
>      tx5_tso_packets: 1512988013
>      tx5_tso_bytes: 8571208921023
>      tx5_tso_inner_packets: 0
>      tx5_tso_inner_bytes: 0
>      tx5_csum_partial: 2078498561
>      tx5_csum_partial_inner: 0
>      tx5_added_vlan_packets: 20465533760
>      tx5_nop: 312614719
>      tx5_csum_none: 18387035199
>      tx5_stopped: 4605
>      tx5_dropped: 0
>      tx5_xmit_more: 64188936
>      tx5_recover: 0
>      tx5_cqes: 20401350718
>      tx5_wake: 4604
>      tx5_cqe_err: 0
>      tx6_packets: 25025504896
>      tx6_bytes: 14908021946070
>      tx6_tso_packets: 1515718977
>      tx6_tso_bytes: 8511442522461
>      tx6_tso_inner_packets: 0
>      tx6_tso_inner_bytes: 0
>      tx6_csum_partial: 2056378610
>      tx6_csum_partial_inner: 0
>      tx6_added_vlan_packets: 20434066400
>      tx6_nop: 310594020
>      tx6_csum_none: 18377687790
>      tx6_stopped: 15234
>      tx6_dropped: 0
>      tx6_xmit_more: 61130422
>      tx6_recover: 0
>      tx6_cqes: 20372943611
>      tx6_wake: 15234
>      tx6_cqe_err: 0
>      tx7_packets: 25457096169
>      tx7_bytes: 15456289446172
>      tx7_tso_packets: 1553342799
>      tx7_tso_bytes: 8764550988105
>      tx7_tso_inner_packets: 0
>      tx7_tso_inner_bytes: 0
>      tx7_csum_partial: 2105765233
>      tx7_csum_partial_inner: 0
>      tx7_added_vlan_packets: 20720382377
>      tx7_nop: 319044853
>      tx7_csum_none: 18614617145
>      tx7_stopped: 18745
>      tx7_dropped: 0
>      tx7_xmit_more: 57050107
>      tx7_recover: 0
>      tx7_cqes: 20663340775
>      tx7_wake: 18746
>      tx7_cqe_err: 0
>      tx8_packets: 25389771649
>      tx8_bytes: 15225503883962
>      tx8_tso_packets: 1563367648
>      tx8_tso_bytes: 8710384514258
>      tx8_tso_inner_packets: 0
>      tx8_tso_inner_bytes: 0
>      tx8_csum_partial: 2106586634
>      tx8_csum_partial_inner: 0
>      tx8_added_vlan_packets: 20704676274
>      tx8_nop: 318149261
>      tx8_csum_none: 18598089640
>      tx8_stopped: 4733
>      tx8_dropped: 0
>      tx8_xmit_more: 61014317
>      tx8_recover: 0
>      tx8_cqes: 20643667301
>      tx8_wake: 4735
>      tx8_cqe_err: 0
>      tx9_packets: 25521500166
>      tx9_bytes: 15302293145755
>      tx9_tso_packets: 1546316697
>      tx9_tso_bytes: 8770688145926
>      tx9_tso_inner_packets: 0
>      tx9_tso_inner_bytes: 0
>      tx9_csum_partial: 2097652880
>      tx9_csum_partial_inner: 0
>      tx9_added_vlan_packets: 20778408432
>      tx9_nop: 318538543
>      tx9_csum_none: 18680755556
>      tx9_stopped: 16118
>      tx9_dropped: 0
>      tx9_xmit_more: 68509728
>      tx9_recover: 0
>      tx9_cqes: 20709906498
>      tx9_wake: 16118
>      tx9_cqe_err: 0
>      tx10_packets: 25451605829
>      tx10_bytes: 15386896170792
>      tx10_tso_packets: 1576473520
>      tx10_tso_bytes: 8880888676383
>      tx10_tso_inner_packets: 0
>      tx10_tso_inner_bytes: 0
>      tx10_csum_partial: 2129796141
>      tx10_csum_partial_inner: 0
>      tx10_added_vlan_packets: 20659622590
>      tx10_nop: 319117433
>      tx10_csum_none: 18529826450
>      tx10_stopped: 20187
>      tx10_dropped: 0
>      tx10_xmit_more: 58892184
>      tx10_recover: 0
>      tx10_cqes: 20600737739
>      tx10_wake: 20188
>      tx10_cqe_err: 0
>      tx11_packets: 27008919793
>      tx11_bytes: 16587719213058
>      tx11_tso_packets: 1734884654
>      tx11_tso_bytes: 9475681471870
>      tx11_tso_inner_packets: 0
>      tx11_tso_inner_bytes: 0
>      tx11_csum_partial: 2296162292
>      tx11_csum_partial_inner: 0
>      tx11_added_vlan_packets: 21943096263
>      tx11_nop: 344188182
>      tx11_csum_none: 19646933971
>      tx11_stopped: 9703
>      tx11_dropped: 0
>      tx11_xmit_more: 66530718
>      tx11_recover: 0
>      tx11_cqes: 21876571667
>      tx11_wake: 9704
>      tx11_cqe_err: 0
>      tx12_packets: 25969493269
>      tx12_bytes: 15980767963416
>      tx12_tso_packets: 1671396456
>      tx12_tso_bytes: 9268973672821
>      tx12_tso_inner_packets: 0
>      tx12_tso_inner_bytes: 0
>      tx12_csum_partial: 2243809182
>      tx12_csum_partial_inner: 0
>      tx12_added_vlan_packets: 20980642456
>      tx12_nop: 330241007
>      tx12_csum_none: 18736833276
>      tx12_stopped: 10341
>      tx12_dropped: 0
>      tx12_xmit_more: 57834100
>      tx12_recover: 0
>      tx12_cqes: 20922815079
>      tx12_wake: 10342
>      tx12_cqe_err: 0
>      tx13_packets: 25332762261
>      tx13_bytes: 15353213283280
>      tx13_tso_packets: 1577433599
>      tx13_tso_bytes: 8785240284281
>      tx13_tso_inner_packets: 0
>      tx13_tso_inner_bytes: 0
>      tx13_csum_partial: 2110640515
>      tx13_csum_partial_inner: 0
>      tx13_added_vlan_packets: 20605670910
>      tx13_nop: 319805741
>      tx13_csum_none: 18495030395
>      tx13_stopped: 7006
>      tx13_dropped: 0
>      tx13_xmit_more: 58314402
>      tx13_recover: 0
>      tx13_cqes: 20547362770
>      tx13_wake: 7008
>      tx13_cqe_err: 0
>      tx14_packets: 26333743548
>      tx14_bytes: 16070719060573
>      tx14_tso_packets: 1677922970
>      tx14_tso_bytes: 9240299765487
>      tx14_tso_inner_packets: 0
>      tx14_tso_inner_bytes: 0
>      tx14_csum_partial: 2215668906
>      tx14_csum_partial_inner: 0
>      tx14_added_vlan_packets: 21384410786
>      tx14_nop: 332734939
>      tx14_csum_none: 19168741880
>      tx14_stopped: 13160
>      tx14_dropped: 0
>      tx14_xmit_more: 57650391
>      tx14_recover: 0
>      tx14_cqes: 21326767783
>      tx14_wake: 13161
>      tx14_cqe_err: 0
>      tx15_packets: 26824968971
>      tx15_bytes: 16687994233452
>      tx15_tso_packets: 1755745052
>      tx15_tso_bytes: 9533814012441
>      tx15_tso_inner_packets: 0
>      tx15_tso_inner_bytes: 0
>      tx15_csum_partial: 2304778064
>      tx15_csum_partial_inner: 0
>      tx15_added_vlan_packets: 21740906107
>      tx15_nop: 344143287
>      tx15_csum_none: 19436128058
>      tx15_stopped: 75
>      tx15_dropped: 0
>      tx15_xmit_more: 63325832
>      tx15_recover: 0
>      tx15_cqes: 21677585345
>      tx15_wake: 74
>      tx15_cqe_err: 0
>      tx16_packets: 24488158946
>      tx16_bytes: 15027415004570
>      tx16_tso_packets: 1559127391
>      tx16_tso_bytes: 8658691917845
>      tx16_tso_inner_packets: 0
>      tx16_tso_inner_bytes: 0
>      tx16_csum_partial: 2075856395
>      tx16_csum_partial_inner: 0
>      tx16_added_vlan_packets: 19835695731
>      tx16_nop: 308464189
>      tx16_csum_none: 17759839340
>      tx16_stopped: 4567
>      tx16_dropped: 0
>      tx16_xmit_more: 62631422
>      tx16_recover: 0
>      tx16_cqes: 19773070012
>      tx16_wake: 4568
>      tx16_cqe_err: 0
>      tx17_packets: 24700413784
>      tx17_bytes: 15216529713715
>      tx17_tso_packets: 1597555108
>      tx17_tso_bytes: 8773728661243
>      tx17_tso_inner_packets: 0
>      tx17_tso_inner_bytes: 0
>      tx17_csum_partial: 2127177297
>      tx17_csum_partial_inner: 0
>      tx17_added_vlan_packets: 20003144561
>      tx17_nop: 313356918
>      tx17_csum_none: 17875967264
>      tx17_stopped: 12572
>      tx17_dropped: 0
>      tx17_xmit_more: 62742980
>      tx17_recover: 0
>      tx17_cqes: 19940407615
>      tx17_wake: 12573
>      tx17_cqe_err: 0
>      tx18_packets: 24887710046
>      tx18_bytes: 15245034034664
>      tx18_tso_packets: 1582550520
>      tx18_tso_bytes: 8782692335483
>      tx18_tso_inner_packets: 0
>      tx18_tso_inner_bytes: 0
>      tx18_csum_partial: 2084514331
>      tx18_csum_partial_inner: 0
>      tx18_added_vlan_packets: 20173879181
>      tx18_nop: 314818702
>      tx18_csum_none: 18089364850
>      tx18_stopped: 21366
>      tx18_dropped: 0
>      tx18_xmit_more: 62485819
>      tx18_recover: 0
>      tx18_cqes: 20111400935
>      tx18_wake: 21366
>      tx18_cqe_err: 0
>      tx19_packets: 24831057648
>      tx19_bytes: 15164663890576
>      tx19_tso_packets: 1599135489
>      tx19_tso_bytes: 8756045449746
>      tx19_tso_inner_packets: 0
>      tx19_tso_inner_bytes: 0
>      tx19_csum_partial: 2119746608
>      tx19_csum_partial_inner: 0
>      tx19_added_vlan_packets: 20143573903
>      tx19_nop: 316966450
>      tx19_csum_none: 18023827295
>      tx19_stopped: 11431
>      tx19_dropped: 0
>      tx19_xmit_more: 57535904
>      tx19_recover: 0
>      tx19_cqes: 20086045325
>      tx19_wake: 11431
>      tx19_cqe_err: 0
>      tx20_packets: 21943735263
>      tx20_bytes: 13528749492187
>      tx20_tso_packets: 1390048103
>      tx20_tso_bytes: 7629058809637
>      tx20_tso_inner_packets: 0
>      tx20_tso_inner_bytes: 0
>      tx20_csum_partial: 1848533941
>      tx20_csum_partial_inner: 0
>      tx20_added_vlan_packets: 17861417651
>      tx20_nop: 276840365
>      tx20_csum_none: 16012883710
>      tx20_stopped: 38457
>      tx20_dropped: 0
>      tx20_xmit_more: 57042753
>      tx20_recover: 0
>      tx20_cqes: 17804384839
>      tx20_wake: 38457
>      tx20_cqe_err: 0
>      tx21_packets: 21476926958
>      tx21_bytes: 13096410597896
>      tx21_tso_packets: 1367724090
>      tx21_tso_bytes: 7568364585127
>      tx21_tso_inner_packets: 0
>      tx21_tso_inner_bytes: 0
>      tx21_csum_partial: 1830570727
>      tx21_csum_partial_inner: 0
>      tx21_added_vlan_packets: 17421087814
>      tx21_nop: 270611519
>      tx21_csum_none: 15590517087
>      tx21_stopped: 31213
>      tx21_dropped: 0
>      tx21_xmit_more: 60305389
>      tx21_recover: 0
>      tx21_cqes: 17360791205
>      tx21_wake: 31213
>      tx21_cqe_err: 0
>      tx22_packets: 21819106444
>      tx22_bytes: 13492871887100
>      tx22_tso_packets: 1387002018
>      tx22_tso_bytes: 7617705727669
>      tx22_tso_inner_packets: 0
>      tx22_tso_inner_bytes: 0
>      tx22_csum_partial: 1853632107
>      tx22_csum_partial_inner: 0
>      tx22_added_vlan_packets: 17743255447
>      tx22_nop: 274820992
>      tx22_csum_none: 15889623340
>      tx22_stopped: 24814
>      tx22_dropped: 0
>      tx22_xmit_more: 60811304
>      tx22_recover: 0
>      tx22_cqes: 17682451111
>      tx22_wake: 24815
>      tx22_cqe_err: 0
>      tx23_packets: 21830455800
>      tx23_bytes: 13427551902532
>      tx23_tso_packets: 1388556038
>      tx23_tso_bytes: 7604040587125
>      tx23_tso_inner_packets: 0
>      tx23_tso_inner_bytes: 0
>      tx23_csum_partial: 1850819694
>      tx23_csum_partial_inner: 0
>      tx23_added_vlan_packets: 17761271122
>      tx23_nop: 275142775
>      tx23_csum_none: 15910451428
>      tx23_stopped: 29899
>      tx23_dropped: 0
>      tx23_xmit_more: 58924909
>      tx23_recover: 0
>      tx23_cqes: 17702355187
>      tx23_wake: 29898
>      tx23_cqe_err: 0
>      tx24_packets: 21961484213
>      tx24_bytes: 13531373062497
>      tx24_tso_packets: 1394697504
>      tx24_tso_bytes: 7663866609308
>      tx24_tso_inner_packets: 0
>      tx24_tso_inner_bytes: 0
>      tx24_csum_partial: 1857072074
>      tx24_csum_partial_inner: 0
>      tx24_added_vlan_packets: 17856887568
>      tx24_nop: 276352855
>      tx24_csum_none: 15999815494
>      tx24_stopped: 33924
>      tx24_dropped: 0
>      tx24_xmit_more: 63992426
>      tx24_recover: 0
>      tx24_cqes: 17792905243
>      tx24_wake: 33923
>      tx24_cqe_err: 0
>      tx25_packets: 21853593838
>      tx25_bytes: 13357487830519
>      tx25_tso_packets: 1398822411
>      tx25_tso_bytes: 7691191518838
>      tx25_tso_inner_packets: 0
>      tx25_tso_inner_bytes: 0
>      tx25_csum_partial: 1869483109
>      tx25_csum_partial_inner: 0
>      tx25_added_vlan_packets: 17734634614
>      tx25_nop: 276327643
>      tx25_csum_none: 15865151505
>      tx25_stopped: 38651
>      tx25_dropped: 0
>      tx25_xmit_more: 56410535
>      tx25_recover: 0
>      tx25_cqes: 17678234537
>      tx25_wake: 38650
>      tx25_cqe_err: 0
>      tx26_packets: 21480261205
>      tx26_bytes: 13148973015935
>      tx26_tso_packets: 1348132284
>      tx26_tso_bytes: 7523489481775
>      tx26_tso_inner_packets: 0
>      tx26_tso_inner_bytes: 0
>      tx26_csum_partial: 1839740745
>      tx26_csum_partial_inner: 0
>      tx26_added_vlan_packets: 17430592911
>      tx26_nop: 270367836
>      tx26_csum_none: 15590852166
>      tx26_stopped: 34044
>      tx26_dropped: 0
>      tx26_xmit_more: 59870114
>      tx26_recover: 0
>      tx26_cqes: 17370736612
>      tx26_wake: 34043
>      tx26_cqe_err: 0
>      tx27_packets: 22694273108
>      tx27_bytes: 14135473431004
>      tx27_tso_packets: 1418371875
>      tx27_tso_bytes: 7784842263038
>      tx27_tso_inner_packets: 0
>      tx27_tso_inner_bytes: 0
>      tx27_csum_partial: 1919170584
>      tx27_csum_partial_inner: 0
>      tx27_added_vlan_packets: 18520826023
>      tx27_nop: 286296272
>      tx27_csum_none: 16601655439
>      tx27_stopped: 38125
>      tx27_dropped: 0
>      tx27_xmit_more: 72749775
>      tx27_recover: 0
>      tx27_cqes: 18448090270
>      tx27_wake: 38127
>      tx27_cqe_err: 0
>      tx28_packets: 0
>      tx28_bytes: 0
>      tx28_tso_packets: 0
>      tx28_tso_bytes: 0
>      tx28_tso_inner_packets: 0
>      tx28_tso_inner_bytes: 0
>      tx28_csum_partial: 0
>      tx28_csum_partial_inner: 0
>      tx28_added_vlan_packets: 0
>      tx28_nop: 0
>      tx28_csum_none: 0
>      tx28_stopped: 0
>      tx28_dropped: 0
>      tx28_xmit_more: 0
>      tx28_recover: 0
>      tx28_cqes: 0
>      tx28_wake: 0
>      tx28_cqe_err: 0
>      tx29_packets: 3
>      tx29_bytes: 266
>      tx29_tso_packets: 0
>      tx29_tso_bytes: 0
>      tx29_tso_inner_packets: 0
>      tx29_tso_inner_bytes: 0
>      tx29_csum_partial: 0
>      tx29_csum_partial_inner: 0
>      tx29_added_vlan_packets: 0
>      tx29_nop: 0
>      tx29_csum_none: 3
>      tx29_stopped: 0
>      tx29_dropped: 0
>      tx29_xmit_more: 1
>      tx29_recover: 0
>      tx29_cqes: 2
>      tx29_wake: 0
>      tx29_cqe_err: 0
>      tx30_packets: 0
>      tx30_bytes: 0
>      tx30_tso_packets: 0
>      tx30_tso_bytes: 0
>      tx30_tso_inner_packets: 0
>      tx30_tso_inner_bytes: 0
>      tx30_csum_partial: 0
>      tx30_csum_partial_inner: 0
>      tx30_added_vlan_packets: 0
>      tx30_nop: 0
>      tx30_csum_none: 0
>      tx30_stopped: 0
>      tx30_dropped: 0
>      tx30_xmit_more: 0
>      tx30_recover: 0
>      tx30_cqes: 0
>      tx30_wake: 0
>      tx30_cqe_err: 0
>      tx31_packets: 0
>      tx31_bytes: 0
>      tx31_tso_packets: 0
>      tx31_tso_bytes: 0
>      tx31_tso_inner_packets: 0
>      tx31_tso_inner_bytes: 0
>      tx31_csum_partial: 0
>      tx31_csum_partial_inner: 0
>      tx31_added_vlan_packets: 0
>      tx31_nop: 0
>      tx31_csum_none: 0
>      tx31_stopped: 0
>      tx31_dropped: 0
>      tx31_xmit_more: 0
>      tx31_recover: 0
>      tx31_cqes: 0
>      tx31_wake: 0
>      tx31_cqe_err: 0
>      tx32_packets: 0
>      tx32_bytes: 0
>      tx32_tso_packets: 0
>      tx32_tso_bytes: 0
>      tx32_tso_inner_packets: 0
>      tx32_tso_inner_bytes: 0
>      tx32_csum_partial: 0
>      tx32_csum_partial_inner: 0
>      tx32_added_vlan_packets: 0
>      tx32_nop: 0
>      tx32_csum_none: 0
>      tx32_stopped: 0
>      tx32_dropped: 0
>      tx32_xmit_more: 0
>      tx32_recover: 0
>      tx32_cqes: 0
>      tx32_wake: 0
>      tx32_cqe_err: 0
>      tx33_packets: 0
>      tx33_bytes: 0
>      tx33_tso_packets: 0
>      tx33_tso_bytes: 0
>      tx33_tso_inner_packets: 0
>      tx33_tso_inner_bytes: 0
>      tx33_csum_partial: 0
>      tx33_csum_partial_inner: 0
>      tx33_added_vlan_packets: 0
>      tx33_nop: 0
>      tx33_csum_none: 0
>      tx33_stopped: 0
>      tx33_dropped: 0
>      tx33_xmit_more: 0
>      tx33_recover: 0
>      tx33_cqes: 0
>      tx33_wake: 0
>      tx33_cqe_err: 0
>      tx34_packets: 0
>      tx34_bytes: 0
>      tx34_tso_packets: 0
>      tx34_tso_bytes: 0
>      tx34_tso_inner_packets: 0
>      tx34_tso_inner_bytes: 0
>      tx34_csum_partial: 0
>      tx34_csum_partial_inner: 0
>      tx34_added_vlan_packets: 0
>      tx34_nop: 0
>      tx34_csum_none: 0
>      tx34_stopped: 0
>      tx34_dropped: 0
>      tx34_xmit_more: 0
>      tx34_recover: 0
>      tx34_cqes: 0
>      tx34_wake: 0
>      tx34_cqe_err: 0
>      tx35_packets: 0
>      tx35_bytes: 0
>      tx35_tso_packets: 0
>      tx35_tso_bytes: 0
>      tx35_tso_inner_packets: 0
>      tx35_tso_inner_bytes: 0
>      tx35_csum_partial: 0
>      tx35_csum_partial_inner: 0
>      tx35_added_vlan_packets: 0
>      tx35_nop: 0
>      tx35_csum_none: 0
>      tx35_stopped: 0
>      tx35_dropped: 0
>      tx35_xmit_more: 0
>      tx35_recover: 0
>      tx35_cqes: 0
>      tx35_wake: 0
>      tx35_cqe_err: 0
>      tx36_packets: 0
>      tx36_bytes: 0
>      tx36_tso_packets: 0
>      tx36_tso_bytes: 0
>      tx36_tso_inner_packets: 0
>      tx36_tso_inner_bytes: 0
>      tx36_csum_partial: 0
>      tx36_csum_partial_inner: 0
>      tx36_added_vlan_packets: 0
>      tx36_nop: 0
>      tx36_csum_none: 0
>      tx36_stopped: 0
>      tx36_dropped: 0
>      tx36_xmit_more: 0
>      tx36_recover: 0
>      tx36_cqes: 0
>      tx36_wake: 0
>      tx36_cqe_err: 0
>      tx37_packets: 0
>      tx37_bytes: 0
>      tx37_tso_packets: 0
>      tx37_tso_bytes: 0
>      tx37_tso_inner_packets: 0
>      tx37_tso_inner_bytes: 0
>      tx37_csum_partial: 0
>      tx37_csum_partial_inner: 0
>      tx37_added_vlan_packets: 0
>      tx37_nop: 0
>      tx37_csum_none: 0
>      tx37_stopped: 0
>      tx37_dropped: 0
>      tx37_xmit_more: 0
>      tx37_recover: 0
>      tx37_cqes: 0
>      tx37_wake: 0
>      tx37_cqe_err: 0
>      tx38_packets: 0
>      tx38_bytes: 0
>      tx38_tso_packets: 0
>      tx38_tso_bytes: 0
>      tx38_tso_inner_packets: 0
>      tx38_tso_inner_bytes: 0
>      tx38_csum_partial: 0
>      tx38_csum_partial_inner: 0
>      tx38_added_vlan_packets: 0
>      tx38_nop: 0
>      tx38_csum_none: 0
>      tx38_stopped: 0
>      tx38_dropped: 0
>      tx38_xmit_more: 0
>      tx38_recover: 0
>      tx38_cqes: 0
>      tx38_wake: 0
>      tx38_cqe_err: 0
>      tx39_packets: 0
>      tx39_bytes: 0
>      tx39_tso_packets: 0
>      tx39_tso_bytes: 0
>      tx39_tso_inner_packets: 0
>      tx39_tso_inner_bytes: 0
>      tx39_csum_partial: 0
>      tx39_csum_partial_inner: 0
>      tx39_added_vlan_packets: 0
>      tx39_nop: 0
>      tx39_csum_none: 0
>      tx39_stopped: 0
>      tx39_dropped: 0
>      tx39_xmit_more: 0
>      tx39_recover: 0
>      tx39_cqes: 0
>      tx39_wake: 0
>      tx39_cqe_err: 0
>      tx40_packets: 0
>      tx40_bytes: 0
>      tx40_tso_packets: 0
>      tx40_tso_bytes: 0
>      tx40_tso_inner_packets: 0
>      tx40_tso_inner_bytes: 0
>      tx40_csum_partial: 0
>      tx40_csum_partial_inner: 0
>      tx40_added_vlan_packets: 0
>      tx40_nop: 0
>      tx40_csum_none: 0
>      tx40_stopped: 0
>      tx40_dropped: 0
>      tx40_xmit_more: 0
>      tx40_recover: 0
>      tx40_cqes: 0
>      tx40_wake: 0
>      tx40_cqe_err: 0
>      tx41_packets: 0
>      tx41_bytes: 0
>      tx41_tso_packets: 0
>      tx41_tso_bytes: 0
>      tx41_tso_inner_packets: 0
>      tx41_tso_inner_bytes: 0
>      tx41_csum_partial: 0
>      tx41_csum_partial_inner: 0
>      tx41_added_vlan_packets: 0
>      tx41_nop: 0
>      tx41_csum_none: 0
>      tx41_stopped: 0
>      tx41_dropped: 0
>      tx41_xmit_more: 0
>      tx41_recover: 0
>      tx41_cqes: 0
>      tx41_wake: 0
>      tx41_cqe_err: 0
>      tx42_packets: 0
>      tx42_bytes: 0
>      tx42_tso_packets: 0
>      tx42_tso_bytes: 0
>      tx42_tso_inner_packets: 0
>      tx42_tso_inner_bytes: 0
>      tx42_csum_partial: 0
>      tx42_csum_partial_inner: 0
>      tx42_added_vlan_packets: 0
>      tx42_nop: 0
>      tx42_csum_none: 0
>      tx42_stopped: 0
>      tx42_dropped: 0
>      tx42_xmit_more: 0
>      tx42_recover: 0
>      tx42_cqes: 0
>      tx42_wake: 0
>      tx42_cqe_err: 0
>      tx43_packets: 0
>      tx43_bytes: 0
>      tx43_tso_packets: 0
>      tx43_tso_bytes: 0
>      tx43_tso_inner_packets: 0
>      tx43_tso_inner_bytes: 0
>      tx43_csum_partial: 0
>      tx43_csum_partial_inner: 0
>      tx43_added_vlan_packets: 0
>      tx43_nop: 0
>      tx43_csum_none: 0
>      tx43_stopped: 0
>      tx43_dropped: 0
>      tx43_xmit_more: 0
>      tx43_recover: 0
>      tx43_cqes: 0
>      tx43_wake: 0
>      tx43_cqe_err: 0
>      tx44_packets: 0
>      tx44_bytes: 0
>      tx44_tso_packets: 0
>      tx44_tso_bytes: 0
>      tx44_tso_inner_packets: 0
>      tx44_tso_inner_bytes: 0
>      tx44_csum_partial: 0
>      tx44_csum_partial_inner: 0
>      tx44_added_vlan_packets: 0
>      tx44_nop: 0
>      tx44_csum_none: 0
>      tx44_stopped: 0
>      tx44_dropped: 0
>      tx44_xmit_more: 0
>      tx44_recover: 0
>      tx44_cqes: 0
>      tx44_wake: 0
>      tx44_cqe_err: 0
>      tx45_packets: 0
>      tx45_bytes: 0
>      tx45_tso_packets: 0
>      tx45_tso_bytes: 0
>      tx45_tso_inner_packets: 0
>      tx45_tso_inner_bytes: 0
>      tx45_csum_partial: 0
>      tx45_csum_partial_inner: 0
>      tx45_added_vlan_packets: 0
>      tx45_nop: 0
>      tx45_csum_none: 0
>      tx45_stopped: 0
>      tx45_dropped: 0
>      tx45_xmit_more: 0
>      tx45_recover: 0
>      tx45_cqes: 0
>      tx45_wake: 0
>      tx45_cqe_err: 0
>      tx46_packets: 0
>      tx46_bytes: 0
>      tx46_tso_packets: 0
>      tx46_tso_bytes: 0
>      tx46_tso_inner_packets: 0
>      tx46_tso_inner_bytes: 0
>      tx46_csum_partial: 0
>      tx46_csum_partial_inner: 0
>      tx46_added_vlan_packets: 0
>      tx46_nop: 0
>      tx46_csum_none: 0
>      tx46_stopped: 0
>      tx46_dropped: 0
>      tx46_xmit_more: 0
>      tx46_recover: 0
>      tx46_cqes: 0
>      tx46_wake: 0
>      tx46_cqe_err: 0
>      tx47_packets: 0
>      tx47_bytes: 0
>      tx47_tso_packets: 0
>      tx47_tso_bytes: 0
>      tx47_tso_inner_packets: 0
>      tx47_tso_inner_bytes: 0
>      tx47_csum_partial: 0
>      tx47_csum_partial_inner: 0
>      tx47_added_vlan_packets: 0
>      tx47_nop: 0
>      tx47_csum_none: 0
>      tx47_stopped: 0
>      tx47_dropped: 0
>      tx47_xmit_more: 0
>      tx47_recover: 0
>      tx47_cqes: 0
>      tx47_wake: 0
>      tx47_cqe_err: 0
>      tx48_packets: 0
>      tx48_bytes: 0
>      tx48_tso_packets: 0
>      tx48_tso_bytes: 0
>      tx48_tso_inner_packets: 0
>      tx48_tso_inner_bytes: 0
>      tx48_csum_partial: 0
>      tx48_csum_partial_inner: 0
>      tx48_added_vlan_packets: 0
>      tx48_nop: 0
>      tx48_csum_none: 0
>      tx48_stopped: 0
>      tx48_dropped: 0
>      tx48_xmit_more: 0
>      tx48_recover: 0
>      tx48_cqes: 0
>      tx48_wake: 0
>      tx48_cqe_err: 0
>      tx49_packets: 0
>      tx49_bytes: 0
>      tx49_tso_packets: 0
>      tx49_tso_bytes: 0
>      tx49_tso_inner_packets: 0
>      tx49_tso_inner_bytes: 0
>      tx49_csum_partial: 0
>      tx49_csum_partial_inner: 0
>      tx49_added_vlan_packets: 0
>      tx49_nop: 0
>      tx49_csum_none: 0
>      tx49_stopped: 0
>      tx49_dropped: 0
>      tx49_xmit_more: 0
>      tx49_recover: 0
>      tx49_cqes: 0
>      tx49_wake: 0
>      tx49_cqe_err: 0
>      tx50_packets: 0
>      tx50_bytes: 0
>      tx50_tso_packets: 0
>      tx50_tso_bytes: 0
>      tx50_tso_inner_packets: 0
>      tx50_tso_inner_bytes: 0
>      tx50_csum_partial: 0
>      tx50_csum_partial_inner: 0
>      tx50_added_vlan_packets: 0
>      tx50_nop: 0
>      tx50_csum_none: 0
>      tx50_stopped: 0
>      tx50_dropped: 0
>      tx50_xmit_more: 0
>      tx50_recover: 0
>      tx50_cqes: 0
>      tx50_wake: 0
>      tx50_cqe_err: 0
>      tx51_packets: 0
>      tx51_bytes: 0
>      tx51_tso_packets: 0
>      tx51_tso_bytes: 0
>      tx51_tso_inner_packets: 0
>      tx51_tso_inner_bytes: 0
>      tx51_csum_partial: 0
>      tx51_csum_partial_inner: 0
>      tx51_added_vlan_packets: 0
>      tx51_nop: 0
>      tx51_csum_none: 0
>      tx51_stopped: 0
>      tx51_dropped: 0
>      tx51_xmit_more: 0
>      tx51_recover: 0
>      tx51_cqes: 0
>      tx51_wake: 0
>      tx51_cqe_err: 0
>      tx52_packets: 0
>      tx52_bytes: 0
>      tx52_tso_packets: 0
>      tx52_tso_bytes: 0
>      tx52_tso_inner_packets: 0
>      tx52_tso_inner_bytes: 0
>      tx52_csum_partial: 0
>      tx52_csum_partial_inner: 0
>      tx52_added_vlan_packets: 0
>      tx52_nop: 0
>      tx52_csum_none: 0
>      tx52_stopped: 0
>      tx52_dropped: 0
>      tx52_xmit_more: 0
>      tx52_recover: 0
>      tx52_cqes: 0
>      tx52_wake: 0
>      tx52_cqe_err: 0
>      tx53_packets: 0
>      tx53_bytes: 0
>      tx53_tso_packets: 0
>      tx53_tso_bytes: 0
>      tx53_tso_inner_packets: 0
>      tx53_tso_inner_bytes: 0
>      tx53_csum_partial: 0
>      tx53_csum_partial_inner: 0
>      tx53_added_vlan_packets: 0
>      tx53_nop: 0
>      tx53_csum_none: 0
>      tx53_stopped: 0
>      tx53_dropped: 0
>      tx53_xmit_more: 0
>      tx53_recover: 0
>      tx53_cqes: 0
>      tx53_wake: 0
>      tx53_cqe_err: 0
>      tx54_packets: 0
>      tx54_bytes: 0
>      tx54_tso_packets: 0
>      tx54_tso_bytes: 0
>      tx54_tso_inner_packets: 0
>      tx54_tso_inner_bytes: 0
>      tx54_csum_partial: 0
>      tx54_csum_partial_inner: 0
>      tx54_added_vlan_packets: 0
>      tx54_nop: 0
>      tx54_csum_none: 0
>      tx54_stopped: 0
>      tx54_dropped: 0
>      tx54_xmit_more: 0
>      tx54_recover: 0
>      tx54_cqes: 0
>      tx54_wake: 0
>      tx54_cqe_err: 0
>      tx55_packets: 0
>      tx55_bytes: 0
>      tx55_tso_packets: 0
>      tx55_tso_bytes: 0
>      tx55_tso_inner_packets: 0
>      tx55_tso_inner_bytes: 0
>      tx55_csum_partial: 0
>      tx55_csum_partial_inner: 0
>      tx55_added_vlan_packets: 0
>      tx55_nop: 0
>      tx55_csum_none: 0
>      tx55_stopped: 0
>      tx55_dropped: 0
>      tx55_xmit_more: 0
>      tx55_recover: 0
>      tx55_cqes: 0
>      tx55_wake: 0
>      tx55_cqe_err: 0
>      tx0_xdp_xmit: 0
>      tx0_xdp_full: 0
>      tx0_xdp_err: 0
>      tx0_xdp_cqes: 0
>      tx1_xdp_xmit: 0
>      tx1_xdp_full: 0
>      tx1_xdp_err: 0
>      tx1_xdp_cqes: 0
>      tx2_xdp_xmit: 0
>      tx2_xdp_full: 0
>      tx2_xdp_err: 0
>      tx2_xdp_cqes: 0
>      tx3_xdp_xmit: 0
>      tx3_xdp_full: 0
>      tx3_xdp_err: 0
>      tx3_xdp_cqes: 0
>      tx4_xdp_xmit: 0
>      tx4_xdp_full: 0
>      tx4_xdp_err: 0
>      tx4_xdp_cqes: 0
>      tx5_xdp_xmit: 0
>      tx5_xdp_full: 0
>      tx5_xdp_err: 0
>      tx5_xdp_cqes: 0
>      tx6_xdp_xmit: 0
>      tx6_xdp_full: 0
>      tx6_xdp_err: 0
>      tx6_xdp_cqes: 0
>      tx7_xdp_xmit: 0
>      tx7_xdp_full: 0
>      tx7_xdp_err: 0
>      tx7_xdp_cqes: 0
>      tx8_xdp_xmit: 0
>      tx8_xdp_full: 0
>      tx8_xdp_err: 0
>      tx8_xdp_cqes: 0
>      tx9_xdp_xmit: 0
>      tx9_xdp_full: 0
>      tx9_xdp_err: 0
>      tx9_xdp_cqes: 0
>      tx10_xdp_xmit: 0
>      tx10_xdp_full: 0
>      tx10_xdp_err: 0
>      tx10_xdp_cqes: 0
>      tx11_xdp_xmit: 0
>      tx11_xdp_full: 0
>      tx11_xdp_err: 0
>      tx11_xdp_cqes: 0
>      tx12_xdp_xmit: 0
>      tx12_xdp_full: 0
>      tx12_xdp_err: 0
>      tx12_xdp_cqes: 0
>      tx13_xdp_xmit: 0
>      tx13_xdp_full: 0
>      tx13_xdp_err: 0
>      tx13_xdp_cqes: 0
>      tx14_xdp_xmit: 0
>      tx14_xdp_full: 0
>      tx14_xdp_err: 0
>      tx14_xdp_cqes: 0
>      tx15_xdp_xmit: 0
>      tx15_xdp_full: 0
>      tx15_xdp_err: 0
>      tx15_xdp_cqes: 0
>      tx16_xdp_xmit: 0
>      tx16_xdp_full: 0
>      tx16_xdp_err: 0
>      tx16_xdp_cqes: 0
>      tx17_xdp_xmit: 0
>      tx17_xdp_full: 0
>      tx17_xdp_err: 0
>      tx17_xdp_cqes: 0
>      tx18_xdp_xmit: 0
>      tx18_xdp_full: 0
>      tx18_xdp_err: 0
>      tx18_xdp_cqes: 0
>      tx19_xdp_xmit: 0
>      tx19_xdp_full: 0
>      tx19_xdp_err: 0
>      tx19_xdp_cqes: 0
>      tx20_xdp_xmit: 0
>      tx20_xdp_full: 0
>      tx20_xdp_err: 0
>      tx20_xdp_cqes: 0
>      tx21_xdp_xmit: 0
>      tx21_xdp_full: 0
>      tx21_xdp_err: 0
>      tx21_xdp_cqes: 0
>      tx22_xdp_xmit: 0
>      tx22_xdp_full: 0
>      tx22_xdp_err: 0
>      tx22_xdp_cqes: 0
>      tx23_xdp_xmit: 0
>      tx23_xdp_full: 0
>      tx23_xdp_err: 0
>      tx23_xdp_cqes: 0
>      tx24_xdp_xmit: 0
>      tx24_xdp_full: 0
>      tx24_xdp_err: 0
>      tx24_xdp_cqes: 0
>      tx25_xdp_xmit: 0
>      tx25_xdp_full: 0
>      tx25_xdp_err: 0
>      tx25_xdp_cqes: 0
>      tx26_xdp_xmit: 0
>      tx26_xdp_full: 0
>      tx26_xdp_err: 0
>      tx26_xdp_cqes: 0
>      tx27_xdp_xmit: 0
>      tx27_xdp_full: 0
>      tx27_xdp_err: 0
>      tx27_xdp_cqes: 0
>      tx28_xdp_xmit: 0
>      tx28_xdp_full: 0
>      tx28_xdp_err: 0
>      tx28_xdp_cqes: 0
>      tx29_xdp_xmit: 0
>      tx29_xdp_full: 0
>      tx29_xdp_err: 0
>      tx29_xdp_cqes: 0
>      tx30_xdp_xmit: 0
>      tx30_xdp_full: 0
>      tx30_xdp_err: 0
>      tx30_xdp_cqes: 0
>      tx31_xdp_xmit: 0
>      tx31_xdp_full: 0
>      tx31_xdp_err: 0
>      tx31_xdp_cqes: 0
>      tx32_xdp_xmit: 0
>      tx32_xdp_full: 0
>      tx32_xdp_err: 0
>      tx32_xdp_cqes: 0
>      tx33_xdp_xmit: 0
>      tx33_xdp_full: 0
>      tx33_xdp_err: 0
>      tx33_xdp_cqes: 0
>      tx34_xdp_xmit: 0
>      tx34_xdp_full: 0
>      tx34_xdp_err: 0
>      tx34_xdp_cqes: 0
>      tx35_xdp_xmit: 0
>      tx35_xdp_full: 0
>      tx35_xdp_err: 0
>      tx35_xdp_cqes: 0
>      tx36_xdp_xmit: 0
>      tx36_xdp_full: 0
>      tx36_xdp_err: 0
>      tx36_xdp_cqes: 0
>      tx37_xdp_xmit: 0
>      tx37_xdp_full: 0
>      tx37_xdp_err: 0
>      tx37_xdp_cqes: 0
>      tx38_xdp_xmit: 0
>      tx38_xdp_full: 0
>      tx38_xdp_err: 0
>      tx38_xdp_cqes: 0
>      tx39_xdp_xmit: 0
>      tx39_xdp_full: 0
>      tx39_xdp_err: 0
>      tx39_xdp_cqes: 0
>      tx40_xdp_xmit: 0
>      tx40_xdp_full: 0
>      tx40_xdp_err: 0
>      tx40_xdp_cqes: 0
>      tx41_xdp_xmit: 0
>      tx41_xdp_full: 0
>      tx41_xdp_err: 0
>      tx41_xdp_cqes: 0
>      tx42_xdp_xmit: 0
>      tx42_xdp_full: 0
>      tx42_xdp_err: 0
>      tx42_xdp_cqes: 0
>      tx43_xdp_xmit: 0
>      tx43_xdp_full: 0
>      tx43_xdp_err: 0
>      tx43_xdp_cqes: 0
>      tx44_xdp_xmit: 0
>      tx44_xdp_full: 0
>      tx44_xdp_err: 0
>      tx44_xdp_cqes: 0
>      tx45_xdp_xmit: 0
>      tx45_xdp_full: 0
>      tx45_xdp_err: 0
>      tx45_xdp_cqes: 0
>      tx46_xdp_xmit: 0
>      tx46_xdp_full: 0
>      tx46_xdp_err: 0
>      tx46_xdp_cqes: 0
>      tx47_xdp_xmit: 0
>      tx47_xdp_full: 0
>      tx47_xdp_err: 0
>      tx47_xdp_cqes: 0
>      tx48_xdp_xmit: 0
>      tx48_xdp_full: 0
>      tx48_xdp_err: 0
>      tx48_xdp_cqes: 0
>      tx49_xdp_xmit: 0
>      tx49_xdp_full: 0
>      tx49_xdp_err: 0
>      tx49_xdp_cqes: 0
>      tx50_xdp_xmit: 0
>      tx50_xdp_full: 0
>      tx50_xdp_err: 0
>      tx50_xdp_cqes: 0
>      tx51_xdp_xmit: 0
>      tx51_xdp_full: 0
>      tx51_xdp_err: 0
>      tx51_xdp_cqes: 0
>      tx52_xdp_xmit: 0
>      tx52_xdp_full: 0
>      tx52_xdp_err: 0
>      tx52_xdp_cqes: 0
>      tx53_xdp_xmit: 0
>      tx53_xdp_full: 0
>      tx53_xdp_err: 0
>      tx53_xdp_cqes: 0
>      tx54_xdp_xmit: 0
>      tx54_xdp_full: 0
>      tx54_xdp_err: 0
>      tx54_xdp_cqes: 0
>      tx55_xdp_xmit: 0
>      tx55_xdp_full: 0
>      tx55_xdp_err: 0
>      tx55_xdp_cqes: 0
>
>
>> [...]
>>
>>>>> ethtool -S enp175s0f0
>>>>> NIC statistics:
>>>>>         rx_packets: 141574897253
>>>>>         rx_bytes: 184445040406258
>>>>>         tx_packets: 172569543894
>>>>>         tx_bytes: 99486882076365
>>>>>         tx_tso_packets: 9367664195
>>>>>         tx_tso_bytes: 56435233992948
>>>>>         tx_tso_inner_packets: 0
>>>>>         tx_tso_inner_bytes: 0
>>>>>         tx_added_vlan_packets: 141297671626
>>>>>         tx_nop: 2102916272
>>>>>         rx_lro_packets: 0
>>>>>         rx_lro_bytes: 0
>>>>>         rx_ecn_mark: 0
>>>>>         rx_removed_vlan_packets: 141574897252
>>>>>         rx_csum_unnecessary: 0
>>>>>         rx_csum_none: 23135854
>>>>>         rx_csum_complete: 141551761398
>>>>>         rx_csum_unnecessary_inner: 0
>>>>>         rx_xdp_drop: 0
>>>>>         rx_xdp_redirect: 0
>>>>>         rx_xdp_tx_xmit: 0
>>>>>         rx_xdp_tx_full: 0
>>>>>         rx_xdp_tx_err: 0
>>>>>         rx_xdp_tx_cqe: 0
>>>>>         tx_csum_none: 127934791664
>>>> It is a good idea to look into this, tx is not requesting hw tx
>>>> csumming for a lot of packets, maybe you are wasting a lot of cpu
>>>> on
>>>> calculating csum, or maybe this is just the rx csum complete..
>>>>
>>>>>         tx_csum_partial: 13362879974
>>>>>         tx_csum_partial_inner: 0
>>>>>         tx_queue_stopped: 232561
>>>> TX queues are stalling, could be an indentation for the pcie
>>>> bottelneck.
>>>>
>>>>>         tx_queue_dropped: 0
>>>>>         tx_xmit_more: 1266021946
>>>>>         tx_recover: 0
>>>>>         tx_cqes: 140031716469
>>>>>         tx_queue_wake: 232561
>>>>>         tx_udp_seg_rem: 0
>>>>>         tx_cqe_err: 0
>>>>>         tx_xdp_xmit: 0
>>>>>         tx_xdp_full: 0
>>>>>         tx_xdp_err: 0
>>>>>         tx_xdp_cqes: 0
>>>>>         rx_wqe_err: 0
>>>>>         rx_mpwqe_filler_cqes: 0
>>>>>         rx_mpwqe_filler_strides: 0
>>>>>         rx_buff_alloc_err: 0
>>>>>         rx_cqe_compress_blks: 0
>>>>>         rx_cqe_compress_pkts: 0
>>>>>         rx_page_reuse: 0
>>>>>         rx_cache_reuse: 16625975793
>>>>>         rx_cache_full: 54161465914
>>>>>         rx_cache_empty: 258048
>>>>>         rx_cache_busy: 54161472735
>>>>>         rx_cache_waive: 0
>>>>>         rx_congst_umr: 0
>>>>>         rx_arfs_err: 0
>>>>>         ch_events: 40572621887
>>>>>         ch_poll: 40885650979
>>>>>         ch_arm: 40429276692
>>>>>         ch_aff_change: 0
>>>>>         ch_eq_rearm: 0
>>>>>         rx_out_of_buffer: 2791690
>>>>>         rx_if_down_packets: 74
>>>>>         rx_vport_unicast_packets: 141843476308
>>>>>         rx_vport_unicast_bytes: 185421265403318
>>>>>         tx_vport_unicast_packets: 172569484005
>>>>>         tx_vport_unicast_bytes: 100019940094298
>>>>>         rx_vport_multicast_packets: 85122935
>>>>>         rx_vport_multicast_bytes: 5761316431
>>>>>         tx_vport_multicast_packets: 6452
>>>>>         tx_vport_multicast_bytes: 643540
>>>>>         rx_vport_broadcast_packets: 22423624
>>>>>         rx_vport_broadcast_bytes: 1390127090
>>>>>         tx_vport_broadcast_packets: 22024
>>>>>         tx_vport_broadcast_bytes: 1321440
>>>>>         rx_vport_rdma_unicast_packets: 0
>>>>>         rx_vport_rdma_unicast_bytes: 0
>>>>>         tx_vport_rdma_unicast_packets: 0
>>>>>         tx_vport_rdma_unicast_bytes: 0
>>>>>         rx_vport_rdma_multicast_packets: 0
>>>>>         rx_vport_rdma_multicast_bytes: 0
>>>>>         tx_vport_rdma_multicast_packets: 0
>>>>>         tx_vport_rdma_multicast_bytes: 0
>>>>>         tx_packets_phy: 172569501577
>>>>>         rx_packets_phy: 142871314588
>>>>>         rx_crc_errors_phy: 0
>>>>>         tx_bytes_phy: 100710212814151
>>>>>         rx_bytes_phy: 187209224289564
>>>>>         tx_multicast_phy: 6452
>>>>>         tx_broadcast_phy: 22024
>>>>>         rx_multicast_phy: 85122933
>>>>>         rx_broadcast_phy: 22423623
>>>>>         rx_in_range_len_errors_phy: 2
>>>>>         rx_out_of_range_len_phy: 0
>>>>>         rx_oversize_pkts_phy: 0
>>>>>         rx_symbol_err_phy: 0
>>>>>         tx_mac_control_phy: 0
>>>>>         rx_mac_control_phy: 0
>>>>>         rx_unsupported_op_phy: 0
>>>>>         rx_pause_ctrl_phy: 0
>>>>>         tx_pause_ctrl_phy: 0
>>>>>         rx_discards_phy: 920161423
>>>> Ok, this port seem to be suffering more, RX is congested, maybe due
>>>> to
>>>> the pcie bottleneck.
>>> Yes this side is receiving more traffic - second port is +10G more tx
>>>
>> [...]
>>
>>
>>>>> Average:      17    0.00    0.00 16.60    0.00    0.00 52.10
>>>>> 0.00    0.00    0.00   31.30
>>>>> Average:      18    0.00    0.00   13.90    0.00    0.00 61.20
>>>>> 0.00    0.00    0.00   24.90
>>>>> Average:      19    0.00    0.00    9.99    0.00    0.00 70.33
>>>>> 0.00    0.00    0.00   19.68
>>>>> Average:      20    0.00    0.00    9.00    0.00    0.00 73.00
>>>>> 0.00    0.00    0.00   18.00
>>>>> Average:      21    0.00    0.00    8.70    0.00    0.00 73.90
>>>>> 0.00    0.00    0.00   17.40
>>>>> Average:      22    0.00    0.00   15.42    0.00    0.00 58.56
>>>>> 0.00    0.00    0.00   26.03
>>>>> Average:      23    0.00    0.00   10.81    0.00    0.00 71.67
>>>>> 0.00    0.00    0.00   17.52
>>>>> Average:      24    0.00    0.00   10.00    0.00    0.00 71.80
>>>>> 0.00    0.00    0.00   18.20
>>>>> Average:      25    0.00    0.00   11.19    0.00    0.00 71.13
>>>>> 0.00    0.00    0.00   17.68
>>>>> Average:      26    0.00    0.00   11.00    0.00    0.00 70.80
>>>>> 0.00    0.00    0.00   18.20
>>>>> Average:      27    0.00    0.00   10.01    0.00    0.00 69.57
>>>>> 0.00    0.00    0.00   20.42
>>>> The numa cores are not at 100% util, you have around 20% of idle on
>>>> each one.
>>> Yes - no 100% cpu - but the difference between 80% and 100% is like
>>> push
>>> aditional 1-2Gbit/s
>>>
>> yes but, it doens't look like the bottleneck is the cpu, although it is
>> close to be :)..
>>
>>>>> Average:      28    0.00    0.00 0.00    0.00    0.00 0.00
>>>>> 0.00
>>>>> 0.00    0.00  100.00
>>>>> Average:      29    0.00    0.00    0.00    0.00    0.00 0.00
>>>>> 0.00
>>>>> 0.00    0.00  100.00
>>>>> Average:      30    0.00    0.00    0.00    0.00    0.00 0.00
>>>>> 0.00
>>>>> 0.00    0.00  100.00
>>>>> Average:      31    0.00    0.00    0.00    0.00    0.00 0.00
>>>>> 0.00
>>>>> 0.00    0.00  100.00
>>>>> Average:      32    0.00    0.00    0.00    0.00    0.00 0.00
>>>>> 0.00
>>>>> 0.00    0.00  100.00
>>>>> Average:      33    0.00    0.00    3.90    0.00    0.00 0.00
>>>>> 0.00
>>>>> 0.00    0.00   96.10
>>>>> Average:      34    0.00    0.00    0.00    0.00    0.00 0.00
>>>>> 0.00
>>>>> 0.00    0.00  100.00
>>>>> Average:      35    0.00    0.00    0.00    0.00    0.00 0.00
>>>>> 0.00
>>>>> 0.00    0.00  100.00
>>>>> Average:      36    0.10    0.00    0.20    0.00    0.00 0.00
>>>>> 0.00
>>>>> 0.00    0.00   99.70
>>>>> Average:      37    0.20    0.00    0.30    0.00    0.00 0.00
>>>>> 0.00
>>>>> 0.00    0.00   99.50
>>>>> Average:      38    0.00    0.00    0.00    0.00    0.00 0.00
>>>>> 0.00
>>>>> 0.00    0.00  100.00
>>>>> Average:      39    0.00    0.00    2.60    0.00    0.00 0.00
>>>>> 0.00
>>>>> 0.00    0.00   97.40
>>>>> Average:      40    0.00    0.00    0.90    0.00    0.00 0.00
>>>>> 0.00
>>>>> 0.00    0.00   99.10
>>>>> Average:      41    0.10    0.00    0.50    0.00    0.00 0.00
>>>>> 0.00
>>>>> 0.00    0.00   99.40
>>>>> Average:      42    0.00    0.00    9.91    0.00    0.00 70.67
>>>>> 0.00    0.00    0.00   19.42
>>>>> Average:      43    0.00    0.00   15.90    0.00    0.00 57.50
>>>>> 0.00    0.00    0.00   26.60
>>>>> Average:      44    0.00    0.00   12.20    0.00    0.00 66.20
>>>>> 0.00    0.00    0.00   21.60
>>>>> Average:      45    0.00    0.00   12.00    0.00    0.00 67.50
>>>>> 0.00    0.00    0.00   20.50
>>>>> Average:      46    0.00    0.00   12.90    0.00    0.00 65.50
>>>>> 0.00    0.00    0.00   21.60
>>>>> Average:      47    0.00    0.00   14.59    0.00    0.00 60.84
>>>>> 0.00    0.00    0.00   24.58
>>>>> Average:      48    0.00    0.00   13.59    0.00    0.00 61.74
>>>>> 0.00    0.00    0.00   24.68
>>>>> Average:      49    0.00    0.00   18.36    0.00    0.00 53.29
>>>>> 0.00    0.00    0.00   28.34
>>>>> Average:      50    0.00    0.00   15.32    0.00    0.00 58.86
>>>>> 0.00    0.00    0.00   25.83
>>>>> Average:      51    0.00    0.00   17.60    0.00    0.00 55.20
>>>>> 0.00    0.00    0.00   27.20
>>>>> Average:      52    0.00    0.00   15.92    0.00    0.00 56.06
>>>>> 0.00    0.00    0.00   28.03
>>>>> Average:      53    0.00    0.00   13.00    0.00    0.00 62.30
>>>>> 0.00    0.00    0.00   24.70
>>>>> Average:      54    0.00    0.00   13.20    0.00    0.00 61.50
>>>>> 0.00    0.00    0.00   25.30
>>>>> Average:      55    0.00    0.00   14.59    0.00    0.00 58.64
>>>>> 0.00    0.00    0.00   26.77
>>>>>
>>>>>
>>>>> ethtool -k enp175s0f0
>>>>> Features for enp175s0f0:
>>>>> rx-checksumming: on
>>>>> tx-checksumming: on
>>>>>            tx-checksum-ipv4: on
>>>>>            tx-checksum-ip-generic: off [fixed]
>>>>>            tx-checksum-ipv6: on
>>>>>            tx-checksum-fcoe-crc: off [fixed]
>>>>>            tx-checksum-sctp: off [fixed]
>>>>> scatter-gather: on
>>>>>            tx-scatter-gather: on
>>>>>            tx-scatter-gather-fraglist: off [fixed]
>>>>> tcp-segmentation-offload: on
>>>>>            tx-tcp-segmentation: on
>>>>>            tx-tcp-ecn-segmentation: off [fixed]
>>>>>            tx-tcp-mangleid-segmentation: off
>>>>>            tx-tcp6-segmentation: on
>>>>> udp-fragmentation-offload: off
>>>>> generic-segmentation-offload: on
>>>>> generic-receive-offload: on
>>>>> large-receive-offload: off [fixed]
>>>>> rx-vlan-offload: on
>>>>> tx-vlan-offload: on
>>>>> ntuple-filters: off
>>>>> receive-hashing: on
>>>>> highdma: on [fixed]
>>>>> rx-vlan-filter: on
>>>>> vlan-challenged: off [fixed]
>>>>> tx-lockless: off [fixed]
>>>>> netns-local: off [fixed]
>>>>> tx-gso-robust: off [fixed]
>>>>> tx-fcoe-segmentation: off [fixed]
>>>>> tx-gre-segmentation: on
>>>>> tx-gre-csum-segmentation: on
>>>>> tx-ipxip4-segmentation: off [fixed]
>>>>> tx-ipxip6-segmentation: off [fixed]
>>>>> tx-udp_tnl-segmentation: on
>>>>> tx-udp_tnl-csum-segmentation: on
>>>>> tx-gso-partial: on
>>>>> tx-sctp-segmentation: off [fixed]
>>>>> tx-esp-segmentation: off [fixed]
>>>>> tx-udp-segmentation: on
>>>>> fcoe-mtu: off [fixed]
>>>>> tx-nocache-copy: off
>>>>> loopback: off [fixed]
>>>>> rx-fcs: off
>>>>> rx-all: off
>>>>> tx-vlan-stag-hw-insert: on
>>>>> rx-vlan-stag-hw-parse: off [fixed]
>>>>> rx-vlan-stag-filter: on [fixed]
>>>>> l2-fwd-offload: off [fixed]
>>>>> hw-tc-offload: off
>>>>> esp-hw-offload: off [fixed]
>>>>> esp-tx-csum-hw-offload: off [fixed]
>>>>> rx-udp_tunnel-port-offload: on
>>>>> tls-hw-tx-offload: off [fixed]
>>>>> tls-hw-rx-offload: off [fixed]
>>>>> rx-gro-hw: off [fixed]
>>>>> tls-hw-record: off [fixed]
>>>>>
>>>>> ethtool -c enp175s0f0
>>>>> Coalesce parameters for enp175s0f0:
>>>>> Adaptive RX: off  TX: on
>>>>> stats-block-usecs: 0
>>>>> sample-interval: 0
>>>>> pkt-rate-low: 0
>>>>> pkt-rate-high: 0
>>>>> dmac: 32703
>>>>>
>>>>> rx-usecs: 256
>>>>> rx-frames: 128
>>>>> rx-usecs-irq: 0
>>>>> rx-frames-irq: 0
>>>>>
>>>>> tx-usecs: 8
>>>>> tx-frames: 128
>>>>> tx-usecs-irq: 0
>>>>> tx-frames-irq: 0
>>>>>
>>>>> rx-usecs-low: 0
>>>>> rx-frame-low: 0
>>>>> tx-usecs-low: 0
>>>>> tx-frame-low: 0
>>>>>
>>>>> rx-usecs-high: 0
>>>>> rx-frame-high: 0
>>>>> tx-usecs-high: 0
>>>>> tx-frame-high: 0
>>>>>
>>>>> ethtool -g enp175s0f0
>>>>> Ring parameters for enp175s0f0:
>>>>> Pre-set maximums:
>>>>> RX:             8192
>>>>> RX Mini:        0
>>>>> RX Jumbo:       0
>>>>> TX:             8192
>>>>> Current hardware settings:
>>>>> RX:             4096
>>>>> RX Mini:        0
>>>>> RX Jumbo:       0
>>>>> TX:             4096
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>> Also changed a little coalesce params - and best for this config are:
>>> ethtool -c enp175s0f0
>>> Coalesce parameters for enp175s0f0:
>>> Adaptive RX: off  TX: off
>>> stats-block-usecs: 0
>>> sample-interval: 0
>>> pkt-rate-low: 0
>>> pkt-rate-high: 0
>>> dmac: 32573
>>>
>>> rx-usecs: 40
>>> rx-frames: 128
>>> rx-usecs-irq: 0
>>> rx-frames-irq: 0
>>>
>>> tx-usecs: 8
>>> tx-frames: 8
>>> tx-usecs-irq: 0
>>> tx-frames-irq: 0
>>>
>>> rx-usecs-low: 0
>>> rx-frame-low: 0
>>> tx-usecs-low: 0
>>> tx-frame-low: 0
>>>
>>> rx-usecs-high: 0
>>> rx-frame-high: 0
>>> tx-usecs-high: 0
>>> tx-frame-high: 0
>>>
>>>
>>> Less drops on RX side - and more pps in overall forwarded.
>>>
>> how much improvement ? maybe we can improve our adaptive rx coal to be
>> efficient for this work load.
>>
>>
> I can prepare more stats with ethtool maybee to compare - but normally 
> tested with simple icmp forwarded from interface to interface
> - before change  coalescence params:
> adaptive-rx off rx-usecs 384 rx-frames 128
> 3% loss for icmp
> - after change to:
> adaptive-rx off rx-usecs 40 rx-frames 128 adaptive-tx off tx-usecs 8 
> tx-frames 8
> 2% loss for icmp
>
> But yes - to know better will need to compare rx/tx counters from 
> ethtool + /proc/net/dev
>
>
> Was trying to turn on adaptative-tx+rx - but 100% saturation at 
> 43Gbit/s RX / 43Gbit/s TX
>
>
>
>
>
>
>