netdev - Re: Kernel 4.19 network performance - forwarding/routing normal users traffic

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives
Hash Suite: Windows password security audit tool. GUI, reports in PDF.
[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]
Message-ID: <54a9f3fb-50a7-e12d-fa57-b01d186b26e9@itcare.pl>
Date:   Thu, 1 Nov 2018 22:34:20 +0100
From:   Paweł Staszewski <pstaszewski@...are.pl>
To:     Saeed Mahameed <saeedm@...lanox.com>,
        "netdev@...r.kernel.org" <netdev@...r.kernel.org>
Subject: Re: Kernel 4.19 network performance - forwarding/routing normal users
 traffic



W dniu 01.11.2018 o 22:24, Paweł Staszewski pisze:
>
>
> W dniu 01.11.2018 o 22:18, Paweł Staszewski pisze:
>>
>>
>> W dniu 01.11.2018 o 21:37, Saeed Mahameed pisze:
>>> On Thu, 2018-11-01 at 12:09 +0100, Paweł Staszewski wrote:
>>>> W dniu 01.11.2018 o 10:50, Saeed Mahameed pisze:
>>>>> On Wed, 2018-10-31 at 22:57 +0100, Paweł Staszewski wrote:
>>>>>> Hi
>>>>>>
>>>>>> So maybee someone will be interested how linux kernel handles
>>>>>> normal
>>>>>> traffic (not pktgen :) )
>>>>>>
>>>>>>
>>>>>> Server HW configuration:
>>>>>>
>>>>>> CPU : Intel(R) Xeon(R) Gold 6132 CPU @ 2.60GHz
>>>>>>
>>>>>> NIC's: 2x 100G Mellanox ConnectX-4 (connected to x16 pcie 8GT)
>>>>>>
>>>>>>
>>>>>> Server software:
>>>>>>
>>>>>> FRR - as routing daemon
>>>>>>
>>>>>> enp175s0f0 (100G) - 16 vlans from upstreams (28 RSS binded to
>>>>>> local
>>>>>> numa
>>>>>> node)
>>>>>>
>>>>>> enp175s0f1 (100G) - 343 vlans to clients (28 RSS binded to local
>>>>>> numa
>>>>>> node)
>>>>>>
>>>>>>
>>>>>> Maximum traffic that server can handle:
>>>>>>
>>>>>> Bandwidth
>>>>>>
>>>>>>     bwm-ng v0.6.1 (probing every 1.000s), press 'h' for help
>>>>>>      input: /proc/net/dev type: rate
>>>>>>      \         iface                   Rx Tx                Total
>>>>>> =================================================================
>>>>>> ====
>>>>>> =========
>>>>>>           enp175s0f1:          28.51 Gb/s 37.24
>>>>>> Gb/s
>>>>>> 65.74 Gb/s
>>>>>>           enp175s0f0:          38.07 Gb/s 28.44
>>>>>> Gb/s
>>>>>> 66.51 Gb/s
>>>>>> ---------------------------------------------------------------
>>>>>> ----
>>>>>> -----------
>>>>>>                total:          66.58 Gb/s 65.67
>>>>>> Gb/s
>>>>>> 132.25 Gb/s
>>>>>>
>>>>>>
>>>>>> Packets per second:
>>>>>>
>>>>>>     bwm-ng v0.6.1 (probing every 1.000s), press 'h' for help
>>>>>>      input: /proc/net/dev type: rate
>>>>>>      -         iface                   Rx Tx                Total
>>>>>> =================================================================
>>>>>> ====
>>>>>> =========
>>>>>>           enp175s0f1:      5248589.00 P/s 3486617.75 P/s
>>>>>> 8735207.00 P/s
>>>>>>           enp175s0f0:      3557944.25 P/s 5232516.00 P/s
>>>>>> 8790460.00 P/s
>>>>>> ---------------------------------------------------------------
>>>>>> ----
>>>>>> -----------
>>>>>>                total:      8806533.00 P/s 8719134.00 P/s
>>>>>> 17525668.00 P/s
>>>>>>
>>>>>>
>>>>>> After reaching that limits nics on the upstream side (more RX
>>>>>> traffic)
>>>>>> start to drop packets
>>>>>>
>>>>>>
>>>>>> I just dont understand that server can't handle more bandwidth
>>>>>> (~40Gbit/s is limit where all cpu's are 100% util) - where pps on
>>>>>> RX
>>>>>> side are increasing.
>>>>>>
>>>>> Where do you see 40 Gb/s ? you showed that both ports on the same
>>>>> NIC (
>>>>> same pcie link) are doing  66.58 Gb/s (RX) + 65.67 Gb/s (TX) =
>>>>> 132.25
>>>>> Gb/s which aligns with your pcie link limit, what am i missing ?
>>>> hmm yes that was my concern also - cause cant find anywhere
>>>> informations
>>>> about that bandwidth is uni or bidirectional - so if 126Gbit for x16
>>>> 8GT
>>>> is unidir - then bidir will be 126/2 ~68Gbit - which will fit total
>>>> bw
>>>> on both ports
>>> i think it is bidir
>>>
>>>> This can explain maybee also why cpuload is rising rapidly from
>>>> 120Gbit/s in total to 132Gbit (counters of bwmng are from /proc/net -
>>>> so
>>>> there can be some error in reading them when offloading (gro/gso/tso)
>>>> on
>>>> nic's is enabled that is why
>>>>
>>>>>> Was thinking that maybee reached some pcie x16 limit - but x16
>>>>>> 8GT
>>>>>> is
>>>>>> 126Gbit - and also when testing with pktgen i can reach more bw
>>>>>> and
>>>>>> pps
>>>>>> (like 4x more comparing to normal internet traffic)
>>>>>>
>>>>> Are you forwarding when using pktgen as well or you just testing
>>>>> the RX
>>>>> side pps ?
>>>> Yes pktgen was tested on single port RX
>>>> Can check also forwarding to eliminate pciex limits
>>>>
>>> So this explains why you have more RX pps, since tx is idle and pcie
>>> will be free to do only rx.
>>>
>>> [...]
>>>
>>>
>>>>>> ethtool -S enp175s0f1
>>>>>> NIC statistics:
>>>>>>         rx_packets: 173730800927
>>>>>>         rx_bytes: 99827422751332
>>>>>>         tx_packets: 142532009512
>>>>>>         tx_bytes: 184633045911222
>>>>>>         tx_tso_packets: 25989113891
>>>>>>         tx_tso_bytes: 132933363384458
>>>>>>         tx_tso_inner_packets: 0
>>>>>>         tx_tso_inner_bytes: 0
>>>>>>         tx_added_vlan_packets: 74630239613
>>>>>>         tx_nop: 2029817748
>>>>>>         rx_lro_packets: 0
>>>>>>         rx_lro_bytes: 0
>>>>>>         rx_ecn_mark: 0
>>>>>>         rx_removed_vlan_packets: 173730800927
>>>>>>         rx_csum_unnecessary: 0
>>>>>>         rx_csum_none: 434357
>>>>>>         rx_csum_complete: 173730366570
>>>>>>         rx_csum_unnecessary_inner: 0
>>>>>>         rx_xdp_drop: 0
>>>>>>         rx_xdp_redirect: 0
>>>>>>         rx_xdp_tx_xmit: 0
>>>>>>         rx_xdp_tx_full: 0
>>>>>>         rx_xdp_tx_err: 0
>>>>>>         rx_xdp_tx_cqe: 0
>>>>>>         tx_csum_none: 38260960853
>>>>>>         tx_csum_partial: 36369278774
>>>>>>         tx_csum_partial_inner: 0
>>>>>>         tx_queue_stopped: 1
>>>>>>         tx_queue_dropped: 0
>>>>>>         tx_xmit_more: 748638099
>>>>>>         tx_recover: 0
>>>>>>         tx_cqes: 73881645031
>>>>>>         tx_queue_wake: 1
>>>>>>         tx_udp_seg_rem: 0
>>>>>>         tx_cqe_err: 0
>>>>>>         tx_xdp_xmit: 0
>>>>>>         tx_xdp_full: 0
>>>>>>         tx_xdp_err: 0
>>>>>>         tx_xdp_cqes: 0
>>>>>>         rx_wqe_err: 0
>>>>>>         rx_mpwqe_filler_cqes: 0
>>>>>>         rx_mpwqe_filler_strides: 0
>>>>>>         rx_buff_alloc_err: 0
>>>>>>         rx_cqe_compress_blks: 0
>>>>>>         rx_cqe_compress_pkts: 0
>>>>> If this is a pcie bottleneck it might be useful to  enable CQE
>>>>> compression (to reduce PCIe completion descriptors transactions)
>>>>> you should see the above rx_cqe_compress_pkts increasing when
>>>>> enabled.
>>>>>
>>>>> $ ethtool  --set-priv-flags enp175s0f1 rx_cqe_compress on
>>>>> $ ethtool --show-priv-flags enp175s0f1
>>>>> Private flags for p6p1:
>>>>> rx_cqe_moder       : on
>>>>> cqe_moder          : off
>>>>> rx_cqe_compress    : on
>>>>> ...
>>>>>
>>>>> try this on both interfaces.
>>>> Done
>>>> ethtool --show-priv-flags enp175s0f1
>>>> Private flags for enp175s0f1:
>>>> rx_cqe_moder       : on
>>>> tx_cqe_moder       : off
>>>> rx_cqe_compress    : on
>>>> rx_striding_rq     : off
>>>> rx_no_csum_complete: off
>>>>
>>>> ethtool --show-priv-flags enp175s0f0
>>>> Private flags for enp175s0f0:
>>>> rx_cqe_moder       : on
>>>> tx_cqe_moder       : off
>>>> rx_cqe_compress    : on
>>>> rx_striding_rq     : off
>>>> rx_no_csum_complete: off
>>>>
>>> did it help reduce the load on the pcie  ? do you see more pps ?
>>> what is the ratio between rx_cqe_compress_pkts and over all rx packets
>>> ?
>> So - a little more pps
>> Before change top - graph / after bottom -> image with graph stats 
>> from proc/net/dev
> Attached link to graph
> https://uploadfiles.io/5vgbh
>
>
Enabling tx_cqe_moder for both ports
helps alot with cpu load at same traffic lvl - the change is about -20% 
on all 28 cores






>> cqe_compress enabled at 11:55
>>
>> Sorry - but for real life traffic it is hard to do any counter 
>> differences - cause traffic just rising alone from minute to minute :)
>> But for that time the change is visible on graph - cause was almost 
>> same for past 20minutes before change.
>>
>>
>> full ethtool below:
>> NIC statistics:
>>      rx_packets: 516522465438
>>      rx_bytes: 680052911258729
>>      tx_packets: 677697545586
>>      tx_bytes: 413647643141709
>>      tx_tso_packets: 42530913279
>>      tx_tso_bytes: 235655668554142
>>      tx_tso_inner_packets: 0
>>      tx_tso_inner_bytes: 0
>>      tx_added_vlan_packets: 551156530885
>>      tx_nop: 8536823558
>>      rx_lro_packets: 0
>>      rx_lro_bytes: 0
>>      rx_ecn_mark: 0
>>      rx_removed_vlan_packets: 516522465438
>>      rx_csum_unnecessary: 0
>>      rx_csum_none: 50382868
>>      rx_csum_complete: 516472082570
>>      rx_csum_unnecessary_inner: 0
>>      rx_xdp_drop: 0
>>      rx_xdp_redirect: 0
>>      rx_xdp_tx_xmit: 0
>>      rx_xdp_tx_full: 0
>>      rx_xdp_tx_err: 0
>>      rx_xdp_tx_cqe: 0
>>      tx_csum_none: 494075047017
>>      tx_csum_partial: 57081483898
>>      tx_csum_partial_inner: 0
>>      tx_queue_stopped: 518624
>>      tx_queue_dropped: 0
>>      tx_xmit_more: 1717880628
>>      tx_recover: 0
>>      tx_cqes: 549438869029
>>      tx_queue_wake: 518627
>>      tx_udp_seg_rem: 0
>>      tx_cqe_err: 0
>>      tx_xdp_xmit: 0
>>      tx_xdp_full: 0
>>      tx_xdp_err: 0
>>      tx_xdp_cqes: 0
>>      rx_wqe_err: 0
>>      rx_mpwqe_filler_cqes: 0
>>      rx_mpwqe_filler_strides: 0
>>      rx_buff_alloc_err: 0
>>      rx_cqe_compress_blks: 11483228712
>>      rx_cqe_compress_pkts: 25794213324
>>      rx_page_reuse: 0
>>      rx_cache_reuse: 63610249810
>>      rx_cache_full: 194650916511
>>      rx_cache_empty: 1118208
>>      rx_cache_busy: 194650982430
>>      rx_cache_waive: 0
>>      rx_congst_umr: 0
>>      rx_arfs_err: 0
>>      ch_events: 119556002196
>>      ch_poll: 121107424977
>>      ch_arm: 115856746008
>>      ch_aff_change: 31
>>      ch_eq_rearm: 0
>>      rx_out_of_buffer: 6880325
>>      rx_if_down_packets: 2062529
>>      rx_vport_unicast_packets: 517433716795
>>      rx_vport_unicast_bytes: 683464347301443
>>      tx_vport_unicast_packets: 677697453738
>>      tx_vport_unicast_bytes: 415788589663315
>>      rx_vport_multicast_packets: 208258309
>>      rx_vport_multicast_bytes: 14224046052
>>      tx_vport_multicast_packets: 21689
>>      tx_vport_multicast_bytes: 2158334
>>      rx_vport_broadcast_packets: 75838646
>>      rx_vport_broadcast_bytes: 4697944695
>>      tx_vport_broadcast_packets: 68730
>>      tx_vport_broadcast_bytes: 4123800
>>      rx_vport_rdma_unicast_packets: 0
>>      rx_vport_rdma_unicast_bytes: 0
>>      tx_vport_rdma_unicast_packets: 0
>>      tx_vport_rdma_unicast_bytes: 0
>>      rx_vport_rdma_multicast_packets: 0
>>      rx_vport_rdma_multicast_bytes: 0
>>      tx_vport_rdma_multicast_packets: 0
>>      tx_vport_rdma_multicast_bytes: 0
>>      tx_packets_phy: 677697543252
>>      rx_packets_phy: 521319491878
>>      rx_crc_errors_phy: 0
>>      tx_bytes_phy: 418499385791411
>>      rx_bytes_phy: 690322537017274
>>      tx_multicast_phy: 21689
>>      tx_broadcast_phy: 68730
>>      rx_multicast_phy: 208258305
>>      rx_broadcast_phy: 75838646
>>      rx_in_range_len_errors_phy: 4
>>      rx_out_of_range_len_phy: 0
>>      rx_oversize_pkts_phy: 0
>>      rx_symbol_err_phy: 0
>>      tx_mac_control_phy: 0
>>      rx_mac_control_phy: 0
>>      rx_unsupported_op_phy: 0
>>      rx_pause_ctrl_phy: 0
>>      tx_pause_ctrl_phy: 0
>>      rx_discards_phy: 3601449265
>>      tx_discards_phy: 0
>>      tx_errors_phy: 0
>>      rx_undersize_pkts_phy: 0
>>      rx_fragments_phy: 0
>>      rx_jabbers_phy: 0
>>      rx_64_bytes_phy: 1416456771
>>      rx_65_to_127_bytes_phy: 40750434737
>>      rx_128_to_255_bytes_phy: 11518110310
>>      rx_256_to_511_bytes_phy: 7055850637
>>      rx_512_to_1023_bytes_phy: 7811550424
>>      rx_1024_to_1518_bytes_phy: 265547564845
>>      rx_1519_to_2047_bytes_phy: 187219522899
>>      rx_2048_to_4095_bytes_phy: 0
>>      rx_4096_to_8191_bytes_phy: 0
>>      rx_8192_to_10239_bytes_phy: 0
>>      link_down_events_phy: 0
>>      rx_pcs_symbol_err_phy: 0
>>      rx_corrected_bits_phy: 0
>>      rx_pci_signal_integrity: 0
>>      tx_pci_signal_integrity: 48
>>      rx_prio0_bytes: 688807632117485
>>      rx_prio0_packets: 516310309931
>>      tx_prio0_bytes: 418499382756025
>>      tx_prio0_packets: 677697534982
>>      rx_prio1_bytes: 1497701612877
>>      rx_prio1_packets: 1206768094
>>      tx_prio1_bytes: 0
>>      tx_prio1_packets: 0
>>      rx_prio2_bytes: 112271227
>>      rx_prio2_packets: 337295
>>      tx_prio2_bytes: 0
>>      tx_prio2_packets: 0
>>      rx_prio3_bytes: 1165455555
>>      rx_prio3_packets: 1544310
>>      tx_prio3_bytes: 0
>>      tx_prio3_packets: 0
>>      rx_prio4_bytes: 161857240
>>      rx_prio4_packets: 341392
>>      tx_prio4_bytes: 0
>>      tx_prio4_packets: 0
>>      rx_prio5_bytes: 455031612
>>      rx_prio5_packets: 2861469
>>      tx_prio5_bytes: 0
>>      tx_prio5_packets: 0
>>      rx_prio6_bytes: 1873928697
>>      rx_prio6_packets: 5146981
>>      tx_prio6_bytes: 0
>>      tx_prio6_packets: 0
>>      rx_prio7_bytes: 13423452430
>>      rx_prio7_packets: 190724796
>>      tx_prio7_bytes: 0
>>      tx_prio7_packets: 0
>>      module_unplug: 0
>>      module_bus_stuck: 0
>>      module_high_temp: 0
>>      module_bad_shorted: 0
>>      ch0_events: 4252266777
>>      ch0_poll: 4330804273
>>      ch0_arm: 4120233182
>>      ch0_aff_change: 2
>>      ch0_eq_rearm: 0
>>      ch1_events: 3938415938
>>      ch1_poll: 4012621322
>>      ch1_arm: 3810131188
>>      ch1_aff_change: 2
>>      ch1_eq_rearm: 0
>>      ch2_events: 3897428860
>>      ch2_poll: 3973886848
>>      ch2_arm: 3773019397
>>      ch2_aff_change: 1
>>      ch2_eq_rearm: 0
>>      ch3_events: 4108000541
>>      ch3_poll: 4180139872
>>      ch3_arm: 3982093366
>>      ch3_aff_change: 1
>>      ch3_eq_rearm: 0
>>      ch4_events: 4652570079
>>      ch4_poll: 4720541090
>>      ch4_arm: 4524475054
>>      ch4_aff_change: 2
>>      ch4_eq_rearm: 0
>>      ch5_events: 3899177385
>>      ch5_poll: 3974274186
>>      ch5_arm: 3772299186
>>      ch5_aff_change: 2
>>      ch5_eq_rearm: 0
>>      ch6_events: 3915161350
>>      ch6_poll: 3992338199
>>      ch6_arm: 3794710989
>>      ch6_aff_change: 0
>>      ch6_eq_rearm: 0
>>      ch7_events: 4008175631
>>      ch7_poll: 4081321248
>>      ch7_arm: 3882263723
>>      ch7_aff_change: 0
>>      ch7_eq_rearm: 0
>>      ch8_events: 4207422352
>>      ch8_poll: 4276465449
>>      ch8_arm: 4077650366
>>      ch8_aff_change: 0
>>      ch8_eq_rearm: 0
>>      ch9_events: 4036491879
>>      ch9_poll: 4108975987
>>      ch9_arm: 3914493694
>>      ch9_aff_change: 0
>>      ch9_eq_rearm: 0
>>      ch10_events: 4066261595
>>      ch10_poll: 4134419606
>>      ch10_arm: 3936637711
>>      ch10_aff_change: 1
>>      ch10_eq_rearm: 0
>>      ch11_events: 4440494043
>>      ch11_poll: 4507578730
>>      ch11_arm: 4318629438
>>      ch11_aff_change: 0
>>      ch11_eq_rearm: 0
>>      ch12_events: 4066958252
>>      ch12_poll: 4130191506
>>      ch12_arm: 3934337782
>>      ch12_aff_change: 0
>>      ch12_eq_rearm: 0
>>      ch13_events: 4051309159
>>      ch13_poll: 4118864120
>>      ch13_arm: 3921011919
>>      ch13_aff_change: 0
>>      ch13_eq_rearm: 0
>>      ch14_events: 4321664800
>>      ch14_poll: 4382433680
>>      ch14_arm: 4186130552
>>      ch14_aff_change: 0
>>      ch14_eq_rearm: 0
>>      ch15_events: 4701102075
>>      ch15_poll: 4760373932
>>      ch15_arm: 4570151468
>>      ch15_aff_change: 0
>>      ch15_eq_rearm: 0
>>      ch16_events: 4311052687
>>      ch16_poll: 4345937129
>>      ch16_arm: 4170883819
>>      ch16_aff_change: 0
>>      ch16_eq_rearm: 0
>>      ch17_events: 4647570931
>>      ch17_poll: 4680218533
>>      ch17_arm: 4509426288
>>      ch17_aff_change: 0
>>      ch17_eq_rearm: 0
>>      ch18_events: 4598195702
>>      ch18_poll: 4631314898
>>      ch18_arm: 4457267084
>>      ch18_aff_change: 0
>>      ch18_eq_rearm: 0
>>      ch19_events: 4808094560
>>      ch19_poll: 4841368340
>>      ch19_arm: 4670604358
>>      ch19_aff_change: 0
>>      ch19_eq_rearm: 0
>>      ch20_events: 4240910605
>>      ch20_poll: 4276531502
>>      ch20_arm: 4101767278
>>      ch20_aff_change: 1
>>      ch20_eq_rearm: 0
>>      ch21_events: 4389371472
>>      ch21_poll: 4426870311
>>      ch21_arm: 4249339045
>>      ch21_aff_change: 2
>>      ch21_eq_rearm: 0
>>      ch22_events: 4282958754
>>      ch22_poll: 4319228073
>>      ch22_arm: 4145102991
>>      ch22_aff_change: 2
>>      ch22_eq_rearm: 0
>>      ch23_events: 4440196528
>>      ch23_poll: 4474090188
>>      ch23_arm: 4300837147
>>      ch23_aff_change: 2
>>      ch23_eq_rearm: 0
>>      ch24_events: 4326875785
>>      ch24_poll: 4364971263
>>      ch24_arm: 4186404526
>>      ch24_aff_change: 2
>>      ch24_eq_rearm: 0
>>      ch25_events: 4286528453
>>      ch25_poll: 4324089445
>>      ch25_arm: 4147222616
>>      ch25_aff_change: 3
>>      ch25_eq_rearm: 0
>>      ch26_events: 4098043104
>>      ch26_poll: 4138133745
>>      ch26_arm: 3967438971
>>      ch26_aff_change: 4
>>      ch26_eq_rearm: 0
>>      ch27_events: 4563302840
>>      ch27_poll: 4599441446
>>      ch27_arm: 4432182806
>>      ch27_aff_change: 4
>>      ch27_eq_rearm: 0
>>      ch28_events: 4
>>      ch28_poll: 4
>>      ch28_arm: 4
>>      ch28_aff_change: 0
>>      ch28_eq_rearm: 0
>>      ch29_events: 6
>>      ch29_poll: 6
>>      ch29_arm: 6
>>      ch29_aff_change: 0
>>      ch29_eq_rearm: 0
>>      ch30_events: 4
>>      ch30_poll: 4
>>      ch30_arm: 4
>>      ch30_aff_change: 0
>>      ch30_eq_rearm: 0
>>      ch31_events: 4
>>      ch31_poll: 4
>>      ch31_arm: 4
>>      ch31_aff_change: 0
>>      ch31_eq_rearm: 0
>>      ch32_events: 4
>>      ch32_poll: 4
>>      ch32_arm: 4
>>      ch32_aff_change: 0
>>      ch32_eq_rearm: 0
>>      ch33_events: 4
>>      ch33_poll: 4
>>      ch33_arm: 4
>>      ch33_aff_change: 0
>>      ch33_eq_rearm: 0
>>      ch34_events: 4
>>      ch34_poll: 4
>>      ch34_arm: 4
>>      ch34_aff_change: 0
>>      ch34_eq_rearm: 0
>>      ch35_events: 4
>>      ch35_poll: 4
>>      ch35_arm: 4
>>      ch35_aff_change: 0
>>      ch35_eq_rearm: 0
>>      ch36_events: 4
>>      ch36_poll: 4
>>      ch36_arm: 4
>>      ch36_aff_change: 0
>>      ch36_eq_rearm: 0
>>      ch37_events: 4
>>      ch37_poll: 4
>>      ch37_arm: 4
>>      ch37_aff_change: 0
>>      ch37_eq_rearm: 0
>>      ch38_events: 4
>>      ch38_poll: 4
>>      ch38_arm: 4
>>      ch38_aff_change: 0
>>      ch38_eq_rearm: 0
>>      ch39_events: 4
>>      ch39_poll: 4
>>      ch39_arm: 4
>>      ch39_aff_change: 0
>>      ch39_eq_rearm: 0
>>      ch40_events: 4
>>      ch40_poll: 4
>>      ch40_arm: 4
>>      ch40_aff_change: 0
>>      ch40_eq_rearm: 0
>>      ch41_events: 4
>>      ch41_poll: 4
>>      ch41_arm: 4
>>      ch41_aff_change: 0
>>      ch41_eq_rearm: 0
>>      ch42_events: 4
>>      ch42_poll: 4
>>      ch42_arm: 4
>>      ch42_aff_change: 0
>>      ch42_eq_rearm: 0
>>      ch43_events: 4
>>      ch43_poll: 4
>>      ch43_arm: 4
>>      ch43_aff_change: 0
>>      ch43_eq_rearm: 0
>>      ch44_events: 4
>>      ch44_poll: 4
>>      ch44_arm: 4
>>      ch44_aff_change: 0
>>      ch44_eq_rearm: 0
>>      ch45_events: 4
>>      ch45_poll: 4
>>      ch45_arm: 4
>>      ch45_aff_change: 0
>>      ch45_eq_rearm: 0
>>      ch46_events: 4
>>      ch46_poll: 4
>>      ch46_arm: 4
>>      ch46_aff_change: 0
>>      ch46_eq_rearm: 0
>>      ch47_events: 4
>>      ch47_poll: 4
>>      ch47_arm: 4
>>      ch47_aff_change: 0
>>      ch47_eq_rearm: 0
>>      ch48_events: 4
>>      ch48_poll: 4
>>      ch48_arm: 4
>>      ch48_aff_change: 0
>>      ch48_eq_rearm: 0
>>      ch49_events: 4
>>      ch49_poll: 4
>>      ch49_arm: 4
>>      ch49_aff_change: 0
>>      ch49_eq_rearm: 0
>>      ch50_events: 4
>>      ch50_poll: 4
>>      ch50_arm: 4
>>      ch50_aff_change: 0
>>      ch50_eq_rearm: 0
>>      ch51_events: 4
>>      ch51_poll: 4
>>      ch51_arm: 4
>>      ch51_aff_change: 0
>>      ch51_eq_rearm: 0
>>      ch52_events: 4
>>      ch52_poll: 4
>>      ch52_arm: 4
>>      ch52_aff_change: 0
>>      ch52_eq_rearm: 0
>>      ch53_events: 4
>>      ch53_poll: 4
>>      ch53_arm: 4
>>      ch53_aff_change: 0
>>      ch53_eq_rearm: 0
>>      ch54_events: 4
>>      ch54_poll: 4
>>      ch54_arm: 4
>>      ch54_aff_change: 0
>>      ch54_eq_rearm: 0
>>      ch55_events: 4
>>      ch55_poll: 4
>>      ch55_arm: 4
>>      ch55_aff_change: 0
>>      ch55_eq_rearm: 0
>>      rx0_packets: 21390033774
>>      rx0_bytes: 27326856299122
>>      rx0_csum_complete: 21339650906
>>      rx0_csum_unnecessary: 0
>>      rx0_csum_unnecessary_inner: 0
>>      rx0_csum_none: 50382868
>>      rx0_xdp_drop: 0
>>      rx0_xdp_redirect: 0
>>      rx0_lro_packets: 0
>>      rx0_lro_bytes: 0
>>      rx0_ecn_mark: 0
>>      rx0_removed_vlan_packets: 21390033774
>>      rx0_wqe_err: 0
>>      rx0_mpwqe_filler_cqes: 0
>>      rx0_mpwqe_filler_strides: 0
>>      rx0_buff_alloc_err: 0
>>      rx0_cqe_compress_blks: 481077641
>>      rx0_cqe_compress_pkts: 1085647489
>>      rx0_page_reuse: 0
>>      rx0_cache_reuse: 19050049
>>      rx0_cache_full: 10675964285
>>      rx0_cache_empty: 37376
>>      rx0_cache_busy: 10675966819
>>      rx0_cache_waive: 0
>>      rx0_congst_umr: 0
>>      rx0_arfs_err: 0
>>      rx0_xdp_tx_xmit: 0
>>      rx0_xdp_tx_full: 0
>>      rx0_xdp_tx_err: 0
>>      rx0_xdp_tx_cqes: 0
>>      rx1_packets: 19868919527
>>      rx1_bytes: 26149716991561
>>      rx1_csum_complete: 19868919527
>>      rx1_csum_unnecessary: 0
>>      rx1_csum_unnecessary_inner: 0
>>      rx1_csum_none: 0
>>      rx1_xdp_drop: 0
>>      rx1_xdp_redirect: 0
>>      rx1_lro_packets: 0
>>      rx1_lro_bytes: 0
>>      rx1_ecn_mark: 0
>>      rx1_removed_vlan_packets: 19868919527
>>      rx1_wqe_err: 0
>>      rx1_mpwqe_filler_cqes: 0
>>      rx1_mpwqe_filler_strides: 0
>>      rx1_buff_alloc_err: 0
>>      rx1_cqe_compress_blks: 420210560
>>      rx1_cqe_compress_pkts: 941233388
>>      rx1_page_reuse: 0
>>      rx1_cache_reuse: 46200002
>>      rx1_cache_full: 9888257242
>>      rx1_cache_empty: 37376
>>      rx1_cache_busy: 9888259746
>>      rx1_cache_waive: 0
>>      rx1_congst_umr: 0
>>      rx1_arfs_err: 0
>>      rx1_xdp_tx_xmit: 0
>>      rx1_xdp_tx_full: 0
>>      rx1_xdp_tx_err: 0
>>      rx1_xdp_tx_cqes: 0
>>      rx2_packets: 19575013662
>>      rx2_bytes: 25759818417945
>>      rx2_csum_complete: 19575013662
>>      rx2_csum_unnecessary: 0
>>      rx2_csum_unnecessary_inner: 0
>>      rx2_csum_none: 0
>>      rx2_xdp_drop: 0
>>      rx2_xdp_redirect: 0
>>      rx2_lro_packets: 0
>>      rx2_lro_bytes: 0
>>      rx2_ecn_mark: 0
>>      rx2_removed_vlan_packets: 19575013662
>>      rx2_wqe_err: 0
>>      rx2_mpwqe_filler_cqes: 0
>>      rx2_mpwqe_filler_strides: 0
>>      rx2_buff_alloc_err: 0
>>      rx2_cqe_compress_blks: 412345511
>>      rx2_cqe_compress_pkts: 923376167
>>      rx2_page_reuse: 0
>>      rx2_cache_reuse: 38837731
>>      rx2_cache_full: 9748666548
>>      rx2_cache_empty: 37376
>>      rx2_cache_busy: 9748669093
>>      rx2_cache_waive: 0
>>      rx2_congst_umr: 0
>>      rx2_arfs_err: 0
>>      rx2_xdp_tx_xmit: 0
>>      rx2_xdp_tx_full: 0
>>      rx2_xdp_tx_err: 0
>>      rx2_xdp_tx_cqes: 0
>>      rx3_packets: 19795911749
>>      rx3_bytes: 25969475566905
>>      rx3_csum_complete: 19795911749
>>      rx3_csum_unnecessary: 0
>>      rx3_csum_unnecessary_inner: 0
>>      rx3_csum_none: 0
>>      rx3_xdp_drop: 0
>>      rx3_xdp_redirect: 0
>>      rx3_lro_packets: 0
>>      rx3_lro_bytes: 0
>>      rx3_ecn_mark: 0
>>      rx3_removed_vlan_packets: 19795911749
>>      rx3_wqe_err: 0
>>      rx3_mpwqe_filler_cqes: 0
>>      rx3_mpwqe_filler_strides: 0
>>      rx3_buff_alloc_err: 0
>>      rx3_cqe_compress_blks: 416658765
>>      rx3_cqe_compress_pkts: 934986266
>>      rx3_page_reuse: 0
>>      rx3_cache_reuse: 34542124
>>      rx3_cache_full: 9863411232
>>      rx3_cache_empty: 37376
>>      rx3_cache_busy: 9863413732
>>      rx3_cache_waive: 0
>>      rx3_congst_umr: 0
>>      rx3_arfs_err: 0
>>      rx3_xdp_tx_xmit: 0
>>      rx3_xdp_tx_full: 0
>>      rx3_xdp_tx_err: 0
>>      rx3_xdp_tx_cqes: 0
>>      rx4_packets: 20445652378
>>      rx4_bytes: 26949065110265
>>      rx4_csum_complete: 20445652378
>>      rx4_csum_unnecessary: 0
>>      rx4_csum_unnecessary_inner: 0
>>      rx4_csum_none: 0
>>      rx4_xdp_drop: 0
>>      rx4_xdp_redirect: 0
>>      rx4_lro_packets: 0
>>      rx4_lro_bytes: 0
>>      rx4_ecn_mark: 0
>>      rx4_removed_vlan_packets: 20445652378
>>      rx4_wqe_err: 0
>>      rx4_mpwqe_filler_cqes: 0
>>      rx4_mpwqe_filler_strides: 0
>>      rx4_buff_alloc_err: 0
>>      rx4_cqe_compress_blks: 506085858
>>      rx4_cqe_compress_pkts: 1147860328
>>      rx4_page_reuse: 0
>>      rx4_cache_reuse: 10122542864
>>      rx4_cache_full: 100281206
>>      rx4_cache_empty: 37376
>>      rx4_cache_busy: 100283304
>>      rx4_cache_waive: 0
>>      rx4_congst_umr: 0
>>      rx4_arfs_err: 0
>>      rx4_xdp_tx_xmit: 0
>>      rx4_xdp_tx_full: 0
>>      rx4_xdp_tx_err: 0
>>      rx4_xdp_tx_cqes: 0
>>      rx5_packets: 19622362246
>>      rx5_bytes: 25843450982982
>>      rx5_csum_complete: 19622362246
>>      rx5_csum_unnecessary: 0
>>      rx5_csum_unnecessary_inner: 0
>>      rx5_csum_none: 0
>>      rx5_xdp_drop: 0
>>      rx5_xdp_redirect: 0
>>      rx5_lro_packets: 0
>>      rx5_lro_bytes: 0
>>      rx5_ecn_mark: 0
>>      rx5_removed_vlan_packets: 19622362246
>>      rx5_wqe_err: 0
>>      rx5_mpwqe_filler_cqes: 0
>>      rx5_mpwqe_filler_strides: 0
>>      rx5_buff_alloc_err: 0
>>      rx5_cqe_compress_blks: 422840924
>>      rx5_cqe_compress_pkts: 948005878
>>      rx5_page_reuse: 0
>>      rx5_cache_reuse: 31285453
>>      rx5_cache_full: 9779893117
>>      rx5_cache_empty: 37376
>>      rx5_cache_busy: 9779895647
>>      rx5_cache_waive: 0
>>      rx5_congst_umr: 0
>>      rx5_arfs_err: 0
>>      rx5_xdp_tx_xmit: 0
>>      rx5_xdp_tx_full: 0
>>      rx5_xdp_tx_err: 0
>>      rx5_xdp_tx_cqes: 0
>>      rx6_packets: 19788231278
>>      rx6_bytes: 25985783006486
>>      rx6_csum_complete: 19788231278
>>      rx6_csum_unnecessary: 0
>>      rx6_csum_unnecessary_inner: 0
>>      rx6_csum_none: 0
>>      rx6_xdp_drop: 0
>>      rx6_xdp_redirect: 0
>>      rx6_lro_packets: 0
>>      rx6_lro_bytes: 0
>>      rx6_ecn_mark: 0
>>      rx6_removed_vlan_packets: 19788231278
>>      rx6_wqe_err: 0
>>      rx6_mpwqe_filler_cqes: 0
>>      rx6_mpwqe_filler_strides: 0
>>      rx6_buff_alloc_err: 0
>>      rx6_cqe_compress_blks: 418799056
>>      rx6_cqe_compress_pkts: 938282685
>>      rx6_page_reuse: 0
>>      rx6_cache_reuse: 18114793
>>      rx6_cache_full: 9875998295
>>      rx6_cache_empty: 37376
>>      rx6_cache_busy: 9876000831
>>      rx6_cache_waive: 0
>>      rx6_congst_umr: 0
>>      rx6_arfs_err: 0
>>      rx6_xdp_tx_xmit: 0
>>      rx6_xdp_tx_full: 0
>>      rx6_xdp_tx_err: 0
>>      rx6_xdp_tx_cqes: 0
>>      rx7_packets: 19795759168
>>      rx7_bytes: 26085056586860
>>      rx7_csum_complete: 19795759168
>>      rx7_csum_unnecessary: 0
>>      rx7_csum_unnecessary_inner: 0
>>      rx7_csum_none: 0
>>      rx7_xdp_drop: 0
>>      rx7_xdp_redirect: 0
>>      rx7_lro_packets: 0
>>      rx7_lro_bytes: 0
>>      rx7_ecn_mark: 0
>>      rx7_removed_vlan_packets: 19795759168
>>      rx7_wqe_err: 0
>>      rx7_mpwqe_filler_cqes: 0
>>      rx7_mpwqe_filler_strides: 0
>>      rx7_buff_alloc_err: 0
>>      rx7_cqe_compress_blks: 413959224
>>      rx7_cqe_compress_pkts: 927675936
>>      rx7_page_reuse: 0
>>      rx7_cache_reuse: 23902990
>>      rx7_cache_full: 9873974042
>>      rx7_cache_empty: 37376
>>      rx7_cache_busy: 9873976574
>>      rx7_cache_waive: 0
>>      rx7_congst_umr: 0
>>      rx7_arfs_err: 0
>>      rx7_xdp_tx_xmit: 0
>>      rx7_xdp_tx_full: 0
>>      rx7_xdp_tx_err: 0
>>      rx7_xdp_tx_cqes: 0
>>      rx8_packets: 19963477439
>>      rx8_bytes: 26384640501789
>>      rx8_csum_complete: 19963477439
>>      rx8_csum_unnecessary: 0
>>      rx8_csum_unnecessary_inner: 0
>>      rx8_csum_none: 0
>>      rx8_xdp_drop: 0
>>      rx8_xdp_redirect: 0
>>      rx8_lro_packets: 0
>>      rx8_lro_bytes: 0
>>      rx8_ecn_mark: 0
>>      rx8_removed_vlan_packets: 19963477439
>>      rx8_wqe_err: 0
>>      rx8_mpwqe_filler_cqes: 0
>>      rx8_mpwqe_filler_strides: 0
>>      rx8_buff_alloc_err: 0
>>      rx8_cqe_compress_blks: 420422857
>>      rx8_cqe_compress_pkts: 942720292
>>      rx8_page_reuse: 0
>>      rx8_cache_reuse: 88181713
>>      rx8_cache_full: 9893554525
>>      rx8_cache_empty: 37376
>>      rx8_cache_busy: 9893556983
>>      rx8_cache_waive: 0
>>      rx8_congst_umr: 0
>>      rx8_arfs_err: 0
>>      rx8_xdp_tx_xmit: 0
>>      rx8_xdp_tx_full: 0
>>      rx8_xdp_tx_err: 0
>>      rx8_xdp_tx_cqes: 0
>>      rx9_packets: 19726642138
>>      rx9_bytes: 26063924286499
>>      rx9_csum_complete: 19726642138
>>      rx9_csum_unnecessary: 0
>>      rx9_csum_unnecessary_inner: 0
>>      rx9_csum_none: 0
>>      rx9_xdp_drop: 0
>>      rx9_xdp_redirect: 0
>>      rx9_lro_packets: 0
>>      rx9_lro_bytes: 0
>>      rx9_ecn_mark: 0
>>      rx9_removed_vlan_packets: 19726642138
>>      rx9_wqe_err: 0
>>      rx9_mpwqe_filler_cqes: 0
>>      rx9_mpwqe_filler_strides: 0
>>      rx9_buff_alloc_err: 0
>>      rx9_cqe_compress_blks: 424227411
>>      rx9_cqe_compress_pkts: 951534873
>>      rx9_page_reuse: 0
>>      rx9_cache_reuse: 482901440
>>      rx9_cache_full: 9380417487
>>      rx9_cache_empty: 37376
>>      rx9_cache_busy: 9380419608
>>      rx9_cache_waive: 0
>>      rx9_congst_umr: 0
>>      rx9_arfs_err: 0
>>      rx9_xdp_tx_xmit: 0
>>      rx9_xdp_tx_full: 0
>>      rx9_xdp_tx_err: 0
>>      rx9_xdp_tx_cqes: 0
>>      rx10_packets: 19901229170
>>      rx10_bytes: 26300854495044
>>      rx10_csum_complete: 19901229170
>>      rx10_csum_unnecessary: 0
>>      rx10_csum_unnecessary_inner: 0
>>      rx10_csum_none: 0
>>      rx10_xdp_drop: 0
>>      rx10_xdp_redirect: 0
>>      rx10_lro_packets: 0
>>      rx10_lro_bytes: 0
>>      rx10_ecn_mark: 0
>>      rx10_removed_vlan_packets: 19901229170
>>      rx10_wqe_err: 0
>>      rx10_mpwqe_filler_cqes: 0
>>      rx10_mpwqe_filler_strides: 0
>>      rx10_buff_alloc_err: 0
>>      rx10_cqe_compress_blks: 419082938
>>      rx10_cqe_compress_pkts: 940791347
>>      rx10_page_reuse: 0
>>      rx10_cache_reuse: 14896055
>>      rx10_cache_full: 9935715977
>>      rx10_cache_empty: 37376
>>      rx10_cache_busy: 9935718513
>>      rx10_cache_waive: 0
>>      rx10_congst_umr: 0
>>      rx10_arfs_err: 0
>>      rx10_xdp_tx_xmit: 0
>>      rx10_xdp_tx_full: 0
>>      rx10_xdp_tx_err: 0
>>      rx10_xdp_tx_cqes: 0
>>      rx11_packets: 20352190494
>>      rx11_bytes: 26851034425372
>>      rx11_csum_complete: 20352190494
>>      rx11_csum_unnecessary: 0
>>      rx11_csum_unnecessary_inner: 0
>>      rx11_csum_none: 0
>>      rx11_xdp_drop: 0
>>      rx11_xdp_redirect: 0
>>      rx11_lro_packets: 0
>>      rx11_lro_bytes: 0
>>      rx11_ecn_mark: 0
>>      rx11_removed_vlan_packets: 20352190494
>>      rx11_wqe_err: 0
>>      rx11_mpwqe_filler_cqes: 0
>>      rx11_mpwqe_filler_strides: 0
>>      rx11_buff_alloc_err: 0
>>      rx11_cqe_compress_blks: 501992147
>>      rx11_cqe_compress_pkts: 1140398610
>>      rx11_page_reuse: 0
>>      rx11_cache_reuse: 10071721531
>>      rx11_cache_full: 104371621
>>      rx11_cache_empty: 37376
>>      rx11_cache_busy: 104373697
>>      rx11_cache_waive: 0
>>      rx11_congst_umr: 0
>>      rx11_arfs_err: 0
>>      rx11_xdp_tx_xmit: 0
>>      rx11_xdp_tx_full: 0
>>      rx11_xdp_tx_err: 0
>>      rx11_xdp_tx_cqes: 0
>>      rx12_packets: 19934747149
>>      rx12_bytes: 26296478787829
>>      rx12_csum_complete: 19934747149
>>      rx12_csum_unnecessary: 0
>>      rx12_csum_unnecessary_inner: 0
>>      rx12_csum_none: 0
>>      rx12_xdp_drop: 0
>>      rx12_xdp_redirect: 0
>>      rx12_lro_packets: 0
>>      rx12_lro_bytes: 0
>>      rx12_ecn_mark: 0
>>      rx12_removed_vlan_packets: 19934747149
>>      rx12_wqe_err: 0
>>      rx12_mpwqe_filler_cqes: 0
>>      rx12_mpwqe_filler_strides: 0
>>      rx12_buff_alloc_err: 0
>>      rx12_cqe_compress_blks: 443350570
>>      rx12_cqe_compress_pkts: 995997220
>>      rx12_page_reuse: 0
>>      rx12_cache_reuse: 9864934174
>>      rx12_cache_full: 102437428
>>      rx12_cache_empty: 37376
>>      rx12_cache_busy: 102439382
>>      rx12_cache_waive: 0
>>      rx12_congst_umr: 0
>>      rx12_arfs_err: 0
>>      rx12_xdp_tx_xmit: 0
>>      rx12_xdp_tx_full: 0
>>      rx12_xdp_tx_err: 0
>>      rx12_xdp_tx_cqes: 0
>>      rx13_packets: 19866908096
>>      rx13_bytes: 26160931936186
>>      rx13_csum_complete: 19866908096
>>      rx13_csum_unnecessary: 0
>>      rx13_csum_unnecessary_inner: 0
>>      rx13_csum_none: 0
>>      rx13_xdp_drop: 0
>>      rx13_xdp_redirect: 0
>>      rx13_lro_packets: 0
>>      rx13_lro_bytes: 0
>>      rx13_ecn_mark: 0
>>      rx13_removed_vlan_packets: 19866908096
>>      rx13_wqe_err: 0
>>      rx13_mpwqe_filler_cqes: 0
>>      rx13_mpwqe_filler_strides: 0
>>      rx13_buff_alloc_err: 0
>>      rx13_cqe_compress_blks: 413640141
>>      rx13_cqe_compress_pkts: 926175066
>>      rx13_page_reuse: 0
>>      rx13_cache_reuse: 36358610
>>      rx13_cache_full: 9897092921
>>      rx13_cache_empty: 37376
>>      rx13_cache_busy: 9897095422
>>      rx13_cache_waive: 0
>>      rx13_congst_umr: 0
>>      rx13_arfs_err: 0
>>      rx13_xdp_tx_xmit: 0
>>      rx13_xdp_tx_full: 0
>>      rx13_xdp_tx_err: 0
>>      rx13_xdp_tx_cqes: 0
>>      rx14_packets: 20229035746
>>      rx14_bytes: 26655092809172
>>      rx14_csum_complete: 20229035746
>>      rx14_csum_unnecessary: 0
>>      rx14_csum_unnecessary_inner: 0
>>      rx14_csum_none: 0
>>      rx14_xdp_drop: 0
>>      rx14_xdp_redirect: 0
>>      rx14_lro_packets: 0
>>      rx14_lro_bytes: 0
>>      rx14_ecn_mark: 0
>>      rx14_removed_vlan_packets: 20229035746
>>      rx14_wqe_err: 0
>>      rx14_mpwqe_filler_cqes: 0
>>      rx14_mpwqe_filler_strides: 0
>>      rx14_buff_alloc_err: 0
>>      rx14_cqe_compress_blks: 460990337
>>      rx14_cqe_compress_pkts: 1041287948
>>      rx14_page_reuse: 0
>>      rx14_cache_reuse: 25649275
>>      rx14_cache_full: 10088866045
>>      rx14_cache_empty: 37376
>>      rx14_cache_busy: 10088868574
>>      rx14_cache_waive: 0
>>      rx14_congst_umr: 0
>>      rx14_arfs_err: 0
>>      rx14_xdp_tx_xmit: 0
>>      rx14_xdp_tx_full: 0
>>      rx14_xdp_tx_err: 0
>>      rx14_xdp_tx_cqes: 0
>>      rx15_packets: 20528177154
>>      rx15_bytes: 27029263893264
>>      rx15_csum_complete: 20528177154
>>      rx15_csum_unnecessary: 0
>>      rx15_csum_unnecessary_inner: 0
>>      rx15_csum_none: 0
>>      rx15_xdp_drop: 0
>>      rx15_xdp_redirect: 0
>>      rx15_lro_packets: 0
>>      rx15_lro_bytes: 0
>>      rx15_ecn_mark: 0
>>      rx15_removed_vlan_packets: 20528177154
>>      rx15_wqe_err: 0
>>      rx15_mpwqe_filler_cqes: 0
>>      rx15_mpwqe_filler_strides: 0
>>      rx15_buff_alloc_err: 0
>>      rx15_cqe_compress_blks: 476776176
>>      rx15_cqe_compress_pkts: 1076153263
>>      rx15_page_reuse: 0
>>      rx15_cache_reuse: 48426735
>>      rx15_cache_full: 10215659289
>>      rx15_cache_empty: 37376
>>      rx15_cache_busy: 10215661817
>>      rx15_cache_waive: 0
>>      rx15_congst_umr: 0
>>      rx15_arfs_err: 0
>>      rx15_xdp_tx_xmit: 0
>>      rx15_xdp_tx_full: 0
>>      rx15_xdp_tx_err: 0
>>      rx15_xdp_tx_cqes: 0
>>      rx16_packets: 16104078098
>>      rx16_bytes: 21256361789679
>>      rx16_csum_complete: 16104078098
>>      rx16_csum_unnecessary: 0
>>      rx16_csum_unnecessary_inner: 0
>>      rx16_csum_none: 0
>>      rx16_xdp_drop: 0
>>      rx16_xdp_redirect: 0
>>      rx16_lro_packets: 0
>>      rx16_lro_bytes: 0
>>      rx16_ecn_mark: 0
>>      rx16_removed_vlan_packets: 16104078098
>>      rx16_wqe_err: 0
>>      rx16_mpwqe_filler_cqes: 0
>>      rx16_mpwqe_filler_strides: 0
>>      rx16_buff_alloc_err: 0
>>      rx16_cqe_compress_blks: 352082054
>>      rx16_cqe_compress_pkts: 787161670
>>      rx16_page_reuse: 0
>>      rx16_cache_reuse: 25912567
>>      rx16_cache_full: 8026124051
>>      rx16_cache_empty: 37376
>>      rx16_cache_busy: 8026126465
>>      rx16_cache_waive: 0
>>      rx16_congst_umr: 0
>>      rx16_arfs_err: 0
>>      rx16_xdp_tx_xmit: 0
>>      rx16_xdp_tx_full: 0
>>      rx16_xdp_tx_err: 0
>>      rx16_xdp_tx_cqes: 0
>>      rx17_packets: 16314055017
>>      rx17_bytes: 21589139030173
>>      rx17_csum_complete: 16314055017
>>      rx17_csum_unnecessary: 0
>>      rx17_csum_unnecessary_inner: 0
>>      rx17_csum_none: 0
>>      rx17_xdp_drop: 0
>>      rx17_xdp_redirect: 0
>>      rx17_lro_packets: 0
>>      rx17_lro_bytes: 0
>>      rx17_ecn_mark: 0
>>      rx17_removed_vlan_packets: 16314055017
>>      rx17_wqe_err: 0
>>      rx17_mpwqe_filler_cqes: 0
>>      rx17_mpwqe_filler_strides: 0
>>      rx17_buff_alloc_err: 0
>>      rx17_cqe_compress_blks: 387834541
>>      rx17_cqe_compress_pkts: 871851081
>>      rx17_page_reuse: 0
>>      rx17_cache_reuse: 24021313
>>      rx17_cache_full: 8133003829
>>      rx17_cache_empty: 37376
>>      rx17_cache_busy: 8133006175
>>      rx17_cache_waive: 0
>>      rx17_congst_umr: 0
>>      rx17_arfs_err: 0
>>      rx17_xdp_tx_xmit: 0
>>      rx17_xdp_tx_full: 0
>>      rx17_xdp_tx_err: 0
>>      rx17_xdp_tx_cqes: 0
>>      rx18_packets: 16439016814
>>      rx18_bytes: 21648651917475
>>      rx18_csum_complete: 16439016814
>>      rx18_csum_unnecessary: 0
>>      rx18_csum_unnecessary_inner: 0
>>      rx18_csum_none: 0
>>      rx18_xdp_drop: 0
>>      rx18_xdp_redirect: 0
>>      rx18_lro_packets: 0
>>      rx18_lro_bytes: 0
>>      rx18_ecn_mark: 0
>>      rx18_removed_vlan_packets: 16439016814
>>      rx18_wqe_err: 0
>>      rx18_mpwqe_filler_cqes: 0
>>      rx18_mpwqe_filler_strides: 0
>>      rx18_buff_alloc_err: 0
>>      rx18_cqe_compress_blks: 375066666
>>      rx18_cqe_compress_pkts: 843563974
>>      rx18_page_reuse: 0
>>      rx18_cache_reuse: 8151064266
>>      rx18_cache_full: 68442025
>>      rx18_cache_empty: 37376
>>      rx18_cache_busy: 68444122
>>      rx18_cache_waive: 0
>>      rx18_congst_umr: 0
>>      rx18_arfs_err: 0
>>      rx18_xdp_tx_xmit: 0
>>      rx18_xdp_tx_full: 0
>>      rx18_xdp_tx_err: 0
>>      rx18_xdp_tx_cqes: 0
>>      rx19_packets: 16641223506
>>      rx19_bytes: 21964749940935
>>      rx19_csum_complete: 16641223506
>>      rx19_csum_unnecessary: 0
>>      rx19_csum_unnecessary_inner: 0
>>      rx19_csum_none: 0
>>      rx19_xdp_drop: 0
>>      rx19_xdp_redirect: 0
>>      rx19_lro_packets: 0
>>      rx19_lro_bytes: 0
>>      rx19_ecn_mark: 0
>>      rx19_removed_vlan_packets: 16641223506
>>      rx19_wqe_err: 0
>>      rx19_mpwqe_filler_cqes: 0
>>      rx19_mpwqe_filler_strides: 0
>>      rx19_buff_alloc_err: 0
>>      rx19_cqe_compress_blks: 387825932
>>      rx19_cqe_compress_pkts: 872266355
>>      rx19_page_reuse: 0
>>      rx19_cache_reuse: 116433620
>>      rx19_cache_full: 8204175954
>>      rx19_cache_empty: 37376
>>      rx19_cache_busy: 8204178120
>>      rx19_cache_waive: 0
>>      rx19_congst_umr: 0
>>      rx19_arfs_err: 0
>>      rx19_xdp_tx_xmit: 0
>>      rx19_xdp_tx_full: 0
>>      rx19_xdp_tx_err: 0
>>      rx19_xdp_tx_cqes: 0
>>      rx20_packets: 16206927741
>>      rx20_bytes: 21387447038430
>>      rx20_csum_complete: 16206927741
>>      rx20_csum_unnecessary: 0
>>      rx20_csum_unnecessary_inner: 0
>>      rx20_csum_none: 0
>>      rx20_xdp_drop: 0
>>      rx20_xdp_redirect: 0
>>      rx20_lro_packets: 0
>>      rx20_lro_bytes: 0
>>      rx20_ecn_mark: 0
>>      rx20_removed_vlan_packets: 16206927741
>>      rx20_wqe_err: 0
>>      rx20_mpwqe_filler_cqes: 0
>>      rx20_mpwqe_filler_strides: 0
>>      rx20_buff_alloc_err: 0
>>      rx20_cqe_compress_blks: 370144620
>>      rx20_cqe_compress_pkts: 829122671
>>      rx20_page_reuse: 0
>>      rx20_cache_reuse: 8053733744
>>      rx20_cache_full: 49728026
>>      rx20_cache_empty: 37376
>>      rx20_cache_busy: 49730116
>>      rx20_cache_waive: 0
>>      rx20_congst_umr: 0
>>      rx20_arfs_err: 0
>>      rx20_xdp_tx_xmit: 0
>>      rx20_xdp_tx_full: 0
>>      rx20_xdp_tx_err: 0
>>      rx20_xdp_tx_cqes: 0
>>      rx21_packets: 16562361314
>>      rx21_bytes: 21856653284356
>>      rx21_csum_complete: 16562361314
>>      rx21_csum_unnecessary: 0
>>      rx21_csum_unnecessary_inner: 0
>>      rx21_csum_none: 0
>>      rx21_xdp_drop: 0
>>      rx21_xdp_redirect: 0
>>      rx21_lro_packets: 0
>>      rx21_lro_bytes: 0
>>      rx21_ecn_mark: 0
>>      rx21_removed_vlan_packets: 16562361314
>>      rx21_wqe_err: 0
>>      rx21_mpwqe_filler_cqes: 0
>>      rx21_mpwqe_filler_strides: 0
>>      rx21_buff_alloc_err: 0
>>      rx21_cqe_compress_blks: 350790425
>>      rx21_cqe_compress_pkts: 783850729
>>      rx21_page_reuse: 0
>>      rx21_cache_reuse: 28077493
>>      rx21_cache_full: 8253100706
>>      rx21_cache_empty: 37376
>>      rx21_cache_busy: 8253103147
>>      rx21_cache_waive: 0
>>      rx21_congst_umr: 0
>>      rx21_arfs_err: 0
>>      rx21_xdp_tx_xmit: 0
>>      rx21_xdp_tx_full: 0
>>      rx21_xdp_tx_err: 0
>>      rx21_xdp_tx_cqes: 0
>>      rx22_packets: 16350307571
>>      rx22_bytes: 21408575325592
>>      rx22_csum_complete: 16350307571
>>      rx22_csum_unnecessary: 0
>>      rx22_csum_unnecessary_inner: 0
>>      rx22_csum_none: 0
>>      rx22_xdp_drop: 0
>>      rx22_xdp_redirect: 0
>>      rx22_lro_packets: 0
>>      rx22_lro_bytes: 0
>>      rx22_ecn_mark: 0
>>      rx22_removed_vlan_packets: 16350307571
>>      rx22_wqe_err: 0
>>      rx22_mpwqe_filler_cqes: 0
>>      rx22_mpwqe_filler_strides: 0
>>      rx22_buff_alloc_err: 0
>>      rx22_cqe_compress_blks: 353531065
>>      rx22_cqe_compress_pkts: 790814415
>>      rx22_page_reuse: 0
>>      rx22_cache_reuse: 16934343
>>      rx22_cache_full: 8158216889
>>      rx22_cache_empty: 37376
>>      rx22_cache_busy: 8158219417
>>      rx22_cache_waive: 0
>>      rx22_congst_umr: 0
>>      rx22_arfs_err: 0
>>      rx22_xdp_tx_xmit: 0
>>      rx22_xdp_tx_full: 0
>>      rx22_xdp_tx_err: 0
>>      rx22_xdp_tx_cqes: 0
>>      rx23_packets: 16019811764
>>      rx23_bytes: 21137182570985
>>      rx23_csum_complete: 16019811764
>>      rx23_csum_unnecessary: 0
>>      rx23_csum_unnecessary_inner: 0
>>      rx23_csum_none: 0
>>      rx23_xdp_drop: 0
>>      rx23_xdp_redirect: 0
>>      rx23_lro_packets: 0
>>      rx23_lro_bytes: 0
>>      rx23_ecn_mark: 0
>>      rx23_removed_vlan_packets: 16019811764
>>      rx23_wqe_err: 0
>>      rx23_mpwqe_filler_cqes: 0
>>      rx23_mpwqe_filler_strides: 0
>>      rx23_buff_alloc_err: 0
>>      rx23_cqe_compress_blks: 349733033
>>      rx23_cqe_compress_pkts: 781248862
>>      rx23_page_reuse: 0
>>      rx23_cache_reuse: 33422343
>>      rx23_cache_full: 7976481152
>>      rx23_cache_empty: 37376
>>      rx23_cache_busy: 7976483525
>>      rx23_cache_waive: 0
>>      rx23_congst_umr: 0
>>      rx23_arfs_err: 0
>>      rx23_xdp_tx_xmit: 0
>>      rx23_xdp_tx_full: 0
>>      rx23_xdp_tx_err: 0
>>      rx23_xdp_tx_cqes: 0
>>      rx24_packets: 16212040646
>>      rx24_bytes: 21393399325700
>>      rx24_csum_complete: 16212040646
>>      rx24_csum_unnecessary: 0
>>      rx24_csum_unnecessary_inner: 0
>>      rx24_csum_none: 0
>>      rx24_xdp_drop: 0
>>      rx24_xdp_redirect: 0
>>      rx24_lro_packets: 0
>>      rx24_lro_bytes: 0
>>      rx24_ecn_mark: 0
>>      rx24_removed_vlan_packets: 16212040646
>>      rx24_wqe_err: 0
>>      rx24_mpwqe_filler_cqes: 0
>>      rx24_mpwqe_filler_strides: 0
>>      rx24_buff_alloc_err: 0
>>      rx24_cqe_compress_blks: 379833752
>>      rx24_cqe_compress_pkts: 852020179
>>      rx24_page_reuse: 0
>>      rx24_cache_reuse: 8033552512
>>      rx24_cache_full: 72465843
>>      rx24_cache_empty: 37376
>>      rx24_cache_busy: 72467789
>>      rx24_cache_waive: 0
>>      rx24_congst_umr: 0
>>      rx24_arfs_err: 0
>>      rx24_xdp_tx_xmit: 0
>>      rx24_xdp_tx_full: 0
>>      rx24_xdp_tx_err: 0
>>      rx24_xdp_tx_cqes: 0
>>      rx25_packets: 16412186257
>>      rx25_bytes: 21651198388407
>>      rx25_csum_complete: 16412186257
>>      rx25_csum_unnecessary: 0
>>      rx25_csum_unnecessary_inner: 0
>>      rx25_csum_none: 0
>>      rx25_xdp_drop: 0
>>      rx25_xdp_redirect: 0
>>      rx25_lro_packets: 0
>>      rx25_lro_bytes: 0
>>      rx25_ecn_mark: 0
>>      rx25_removed_vlan_packets: 16412186257
>>      rx25_wqe_err: 0
>>      rx25_mpwqe_filler_cqes: 0
>>      rx25_mpwqe_filler_strides: 0
>>      rx25_buff_alloc_err: 0
>>      rx25_cqe_compress_blks: 383979685
>>      rx25_cqe_compress_pkts: 861985772
>>      rx25_page_reuse: 0
>>      rx25_cache_reuse: 8129807841
>>      rx25_cache_full: 76283342
>>      rx25_cache_empty: 37376
>>      rx25_cache_busy: 76285271
>>      rx25_cache_waive: 0
>>      rx25_congst_umr: 0
>>      rx25_arfs_err: 0
>>      rx25_xdp_tx_xmit: 0
>>      rx25_xdp_tx_full: 0
>>      rx25_xdp_tx_err: 0
>>      rx25_xdp_tx_cqes: 0
>>      rx26_packets: 16304310003
>>      rx26_bytes: 21571217538721
>>      rx26_csum_complete: 16304310003
>>      rx26_csum_unnecessary: 0
>>      rx26_csum_unnecessary_inner: 0
>>      rx26_csum_none: 0
>>      rx26_xdp_drop: 0
>>      rx26_xdp_redirect: 0
>>      rx26_lro_packets: 0
>>      rx26_lro_bytes: 0
>>      rx26_ecn_mark: 0
>>      rx26_removed_vlan_packets: 16304310003
>>      rx26_wqe_err: 0
>>      rx26_mpwqe_filler_cqes: 0
>>      rx26_mpwqe_filler_strides: 0
>>      rx26_buff_alloc_err: 0
>>      rx26_cqe_compress_blks: 353314041
>>      rx26_cqe_compress_pkts: 788838424
>>      rx26_page_reuse: 0
>>      rx26_cache_reuse: 19673790
>>      rx26_cache_full: 8132478659
>>      rx26_cache_empty: 37376
>>      rx26_cache_busy: 8132481198
>>      rx26_cache_waive: 0
>>      rx26_congst_umr: 0
>>      rx26_arfs_err: 0
>>      rx26_xdp_tx_xmit: 0
>>      rx26_xdp_tx_full: 0
>>      rx26_xdp_tx_err: 0
>>      rx26_xdp_tx_cqes: 0
>>      rx27_packets: 16171856079
>>      rx27_bytes: 21376891736540
>>      rx27_csum_complete: 16171856079
>>      rx27_csum_unnecessary: 0
>>      rx27_csum_unnecessary_inner: 0
>>      rx27_csum_none: 0
>>      rx27_xdp_drop: 0
>>      rx27_xdp_redirect: 0
>>      rx27_lro_packets: 0
>>      rx27_lro_bytes: 0
>>      rx27_ecn_mark: 0
>>      rx27_removed_vlan_packets: 16171856079
>>      rx27_wqe_err: 0
>>      rx27_mpwqe_filler_cqes: 0
>>      rx27_mpwqe_filler_strides: 0
>>      rx27_buff_alloc_err: 0
>>      rx27_cqe_compress_blks: 386632845
>>      rx27_cqe_compress_pkts: 869362576
>>      rx27_page_reuse: 0
>>      rx27_cache_reuse: 10070560
>>      rx27_cache_full: 8075854928
>>      rx27_cache_empty: 37376
>>      rx27_cache_busy: 8075857468
>>      rx27_cache_waive: 0
>>      rx27_congst_umr: 0
>>      rx27_arfs_err: 0
>>      rx27_xdp_tx_xmit: 0
>>      rx27_xdp_tx_full: 0
>>      rx27_xdp_tx_err: 0
>>      rx27_xdp_tx_cqes: 0
>>      rx28_packets: 0
>>      rx28_bytes: 0
>>      rx28_csum_complete: 0
>>      rx28_csum_unnecessary: 0
>>      rx28_csum_unnecessary_inner: 0
>>      rx28_csum_none: 0
>>      rx28_xdp_drop: 0
>>      rx28_xdp_redirect: 0
>>      rx28_lro_packets: 0
>>      rx28_lro_bytes: 0
>>      rx28_ecn_mark: 0
>>      rx28_removed_vlan_packets: 0
>>      rx28_wqe_err: 0
>>      rx28_mpwqe_filler_cqes: 0
>>      rx28_mpwqe_filler_strides: 0
>>      rx28_buff_alloc_err: 0
>>      rx28_cqe_compress_blks: 0
>>      rx28_cqe_compress_pkts: 0
>>      rx28_page_reuse: 0
>>      rx28_cache_reuse: 0
>>      rx28_cache_full: 0
>>      rx28_cache_empty: 2560
>>      rx28_cache_busy: 0
>>      rx28_cache_waive: 0
>>      rx28_congst_umr: 0
>>      rx28_arfs_err: 0
>>      rx28_xdp_tx_xmit: 0
>>      rx28_xdp_tx_full: 0
>>      rx28_xdp_tx_err: 0
>>      rx28_xdp_tx_cqes: 0
>>      rx29_packets: 0
>>      rx29_bytes: 0
>>      rx29_csum_complete: 0
>>      rx29_csum_unnecessary: 0
>>      rx29_csum_unnecessary_inner: 0
>>      rx29_csum_none: 0
>>      rx29_xdp_drop: 0
>>      rx29_xdp_redirect: 0
>>      rx29_lro_packets: 0
>>      rx29_lro_bytes: 0
>>      rx29_ecn_mark: 0
>>      rx29_removed_vlan_packets: 0
>>      rx29_wqe_err: 0
>>      rx29_mpwqe_filler_cqes: 0
>>      rx29_mpwqe_filler_strides: 0
>>      rx29_buff_alloc_err: 0
>>      rx29_cqe_compress_blks: 0
>>      rx29_cqe_compress_pkts: 0
>>      rx29_page_reuse: 0
>>      rx29_cache_reuse: 0
>>      rx29_cache_full: 0
>>      rx29_cache_empty: 2560
>>      rx29_cache_busy: 0
>>      rx29_cache_waive: 0
>>      rx29_congst_umr: 0
>>      rx29_arfs_err: 0
>>      rx29_xdp_tx_xmit: 0
>>      rx29_xdp_tx_full: 0
>>      rx29_xdp_tx_err: 0
>>      rx29_xdp_tx_cqes: 0
>>      rx30_packets: 0
>>      rx30_bytes: 0
>>      rx30_csum_complete: 0
>>      rx30_csum_unnecessary: 0
>>      rx30_csum_unnecessary_inner: 0
>>      rx30_csum_none: 0
>>      rx30_xdp_drop: 0
>>      rx30_xdp_redirect: 0
>>      rx30_lro_packets: 0
>>      rx30_lro_bytes: 0
>>      rx30_ecn_mark: 0
>>      rx30_removed_vlan_packets: 0
>>      rx30_wqe_err: 0
>>      rx30_mpwqe_filler_cqes: 0
>>      rx30_mpwqe_filler_strides: 0
>>      rx30_buff_alloc_err: 0
>>      rx30_cqe_compress_blks: 0
>>      rx30_cqe_compress_pkts: 0
>>      rx30_page_reuse: 0
>>      rx30_cache_reuse: 0
>>      rx30_cache_full: 0
>>      rx30_cache_empty: 2560
>>      rx30_cache_busy: 0
>>      rx30_cache_waive: 0
>>      rx30_congst_umr: 0
>>      rx30_arfs_err: 0
>>      rx30_xdp_tx_xmit: 0
>>      rx30_xdp_tx_full: 0
>>      rx30_xdp_tx_err: 0
>>      rx30_xdp_tx_cqes: 0
>>      rx31_packets: 0
>>      rx31_bytes: 0
>>      rx31_csum_complete: 0
>>      rx31_csum_unnecessary: 0
>>      rx31_csum_unnecessary_inner: 0
>>      rx31_csum_none: 0
>>      rx31_xdp_drop: 0
>>      rx31_xdp_redirect: 0
>>      rx31_lro_packets: 0
>>      rx31_lro_bytes: 0
>>      rx31_ecn_mark: 0
>>      rx31_removed_vlan_packets: 0
>>      rx31_wqe_err: 0
>>      rx31_mpwqe_filler_cqes: 0
>>      rx31_mpwqe_filler_strides: 0
>>      rx31_buff_alloc_err: 0
>>      rx31_cqe_compress_blks: 0
>>      rx31_cqe_compress_pkts: 0
>>      rx31_page_reuse: 0
>>      rx31_cache_reuse: 0
>>      rx31_cache_full: 0
>>      rx31_cache_empty: 2560
>>      rx31_cache_busy: 0
>>      rx31_cache_waive: 0
>>      rx31_congst_umr: 0
>>      rx31_arfs_err: 0
>>      rx31_xdp_tx_xmit: 0
>>      rx31_xdp_tx_full: 0
>>      rx31_xdp_tx_err: 0
>>      rx31_xdp_tx_cqes: 0
>>      rx32_packets: 0
>>      rx32_bytes: 0
>>      rx32_csum_complete: 0
>>      rx32_csum_unnecessary: 0
>>      rx32_csum_unnecessary_inner: 0
>>      rx32_csum_none: 0
>>      rx32_xdp_drop: 0
>>      rx32_xdp_redirect: 0
>>      rx32_lro_packets: 0
>>      rx32_lro_bytes: 0
>>      rx32_ecn_mark: 0
>>      rx32_removed_vlan_packets: 0
>>      rx32_wqe_err: 0
>>      rx32_mpwqe_filler_cqes: 0
>>      rx32_mpwqe_filler_strides: 0
>>      rx32_buff_alloc_err: 0
>>      rx32_cqe_compress_blks: 0
>>      rx32_cqe_compress_pkts: 0
>>      rx32_page_reuse: 0
>>      rx32_cache_reuse: 0
>>      rx32_cache_full: 0
>>      rx32_cache_empty: 2560
>>      rx32_cache_busy: 0
>>      rx32_cache_waive: 0
>>      rx32_congst_umr: 0
>>      rx32_arfs_err: 0
>>      rx32_xdp_tx_xmit: 0
>>      rx32_xdp_tx_full: 0
>>      rx32_xdp_tx_err: 0
>>      rx32_xdp_tx_cqes: 0
>>      rx33_packets: 0
>>      rx33_bytes: 0
>>      rx33_csum_complete: 0
>>      rx33_csum_unnecessary: 0
>>      rx33_csum_unnecessary_inner: 0
>>      rx33_csum_none: 0
>>      rx33_xdp_drop: 0
>>      rx33_xdp_redirect: 0
>>      rx33_lro_packets: 0
>>      rx33_lro_bytes: 0
>>      rx33_ecn_mark: 0
>>      rx33_removed_vlan_packets: 0
>>      rx33_wqe_err: 0
>>      rx33_mpwqe_filler_cqes: 0
>>      rx33_mpwqe_filler_strides: 0
>>      rx33_buff_alloc_err: 0
>>      rx33_cqe_compress_blks: 0
>>      rx33_cqe_compress_pkts: 0
>>      rx33_page_reuse: 0
>>      rx33_cache_reuse: 0
>>      rx33_cache_full: 0
>>      rx33_cache_empty: 2560
>>      rx33_cache_busy: 0
>>      rx33_cache_waive: 0
>>      rx33_congst_umr: 0
>>      rx33_arfs_err: 0
>>      rx33_xdp_tx_xmit: 0
>>      rx33_xdp_tx_full: 0
>>      rx33_xdp_tx_err: 0
>>      rx33_xdp_tx_cqes: 0
>>      rx34_packets: 0
>>      rx34_bytes: 0
>>      rx34_csum_complete: 0
>>      rx34_csum_unnecessary: 0
>>      rx34_csum_unnecessary_inner: 0
>>      rx34_csum_none: 0
>>      rx34_xdp_drop: 0
>>      rx34_xdp_redirect: 0
>>      rx34_lro_packets: 0
>>      rx34_lro_bytes: 0
>>      rx34_ecn_mark: 0
>>      rx34_removed_vlan_packets: 0
>>      rx34_wqe_err: 0
>>      rx34_mpwqe_filler_cqes: 0
>>      rx34_mpwqe_filler_strides: 0
>>      rx34_buff_alloc_err: 0
>>      rx34_cqe_compress_blks: 0
>>      rx34_cqe_compress_pkts: 0
>>      rx34_page_reuse: 0
>>      rx34_cache_reuse: 0
>>      rx34_cache_full: 0
>>      rx34_cache_empty: 2560
>>      rx34_cache_busy: 0
>>      rx34_cache_waive: 0
>>      rx34_congst_umr: 0
>>      rx34_arfs_err: 0
>>      rx34_xdp_tx_xmit: 0
>>      rx34_xdp_tx_full: 0
>>      rx34_xdp_tx_err: 0
>>      rx34_xdp_tx_cqes: 0
>>      rx35_packets: 0
>>      rx35_bytes: 0
>>      rx35_csum_complete: 0
>>      rx35_csum_unnecessary: 0
>>      rx35_csum_unnecessary_inner: 0
>>      rx35_csum_none: 0
>>      rx35_xdp_drop: 0
>>      rx35_xdp_redirect: 0
>>      rx35_lro_packets: 0
>>      rx35_lro_bytes: 0
>>      rx35_ecn_mark: 0
>>      rx35_removed_vlan_packets: 0
>>      rx35_wqe_err: 0
>>      rx35_mpwqe_filler_cqes: 0
>>      rx35_mpwqe_filler_strides: 0
>>      rx35_buff_alloc_err: 0
>>      rx35_cqe_compress_blks: 0
>>      rx35_cqe_compress_pkts: 0
>>      rx35_page_reuse: 0
>>      rx35_cache_reuse: 0
>>      rx35_cache_full: 0
>>      rx35_cache_empty: 2560
>>      rx35_cache_busy: 0
>>      rx35_cache_waive: 0
>>      rx35_congst_umr: 0
>>      rx35_arfs_err: 0
>>      rx35_xdp_tx_xmit: 0
>>      rx35_xdp_tx_full: 0
>>      rx35_xdp_tx_err: 0
>>      rx35_xdp_tx_cqes: 0
>>      rx36_packets: 0
>>      rx36_bytes: 0
>>      rx36_csum_complete: 0
>>      rx36_csum_unnecessary: 0
>>      rx36_csum_unnecessary_inner: 0
>>      rx36_csum_none: 0
>>      rx36_xdp_drop: 0
>>      rx36_xdp_redirect: 0
>>      rx36_lro_packets: 0
>>      rx36_lro_bytes: 0
>>      rx36_ecn_mark: 0
>>      rx36_removed_vlan_packets: 0
>>      rx36_wqe_err: 0
>>      rx36_mpwqe_filler_cqes: 0
>>      rx36_mpwqe_filler_strides: 0
>>      rx36_buff_alloc_err: 0
>>      rx36_cqe_compress_blks: 0
>>      rx36_cqe_compress_pkts: 0
>>      rx36_page_reuse: 0
>>      rx36_cache_reuse: 0
>>      rx36_cache_full: 0
>>      rx36_cache_empty: 2560
>>      rx36_cache_busy: 0
>>      rx36_cache_waive: 0
>>      rx36_congst_umr: 0
>>      rx36_arfs_err: 0
>>      rx36_xdp_tx_xmit: 0
>>      rx36_xdp_tx_full: 0
>>      rx36_xdp_tx_err: 0
>>      rx36_xdp_tx_cqes: 0
>>      rx37_packets: 0
>>      rx37_bytes: 0
>>      rx37_csum_complete: 0
>>      rx37_csum_unnecessary: 0
>>      rx37_csum_unnecessary_inner: 0
>>      rx37_csum_none: 0
>>      rx37_xdp_drop: 0
>>      rx37_xdp_redirect: 0
>>      rx37_lro_packets: 0
>>      rx37_lro_bytes: 0
>>      rx37_ecn_mark: 0
>>      rx37_removed_vlan_packets: 0
>>      rx37_wqe_err: 0
>>      rx37_mpwqe_filler_cqes: 0
>>      rx37_mpwqe_filler_strides: 0
>>      rx37_buff_alloc_err: 0
>>      rx37_cqe_compress_blks: 0
>>      rx37_cqe_compress_pkts: 0
>>      rx37_page_reuse: 0
>>      rx37_cache_reuse: 0
>>      rx37_cache_full: 0
>>      rx37_cache_empty: 2560
>>      rx37_cache_busy: 0
>>      rx37_cache_waive: 0
>>      rx37_congst_umr: 0
>>      rx37_arfs_err: 0
>>      rx37_xdp_tx_xmit: 0
>>      rx37_xdp_tx_full: 0
>>      rx37_xdp_tx_err: 0
>>      rx37_xdp_tx_cqes: 0
>>      rx38_packets: 0
>>      rx38_bytes: 0
>>      rx38_csum_complete: 0
>>      rx38_csum_unnecessary: 0
>>      rx38_csum_unnecessary_inner: 0
>>      rx38_csum_none: 0
>>      rx38_xdp_drop: 0
>>      rx38_xdp_redirect: 0
>>      rx38_lro_packets: 0
>>      rx38_lro_bytes: 0
>>      rx38_ecn_mark: 0
>>      rx38_removed_vlan_packets: 0
>>      rx38_wqe_err: 0
>>      rx38_mpwqe_filler_cqes: 0
>>      rx38_mpwqe_filler_strides: 0
>>      rx38_buff_alloc_err: 0
>>      rx38_cqe_compress_blks: 0
>>      rx38_cqe_compress_pkts: 0
>>      rx38_page_reuse: 0
>>      rx38_cache_reuse: 0
>>      rx38_cache_full: 0
>>      rx38_cache_empty: 2560
>>      rx38_cache_busy: 0
>>      rx38_cache_waive: 0
>>      rx38_congst_umr: 0
>>      rx38_arfs_err: 0
>>      rx38_xdp_tx_xmit: 0
>>      rx38_xdp_tx_full: 0
>>      rx38_xdp_tx_err: 0
>>      rx38_xdp_tx_cqes: 0
>>      rx39_packets: 0
>>      rx39_bytes: 0
>>      rx39_csum_complete: 0
>>      rx39_csum_unnecessary: 0
>>      rx39_csum_unnecessary_inner: 0
>>      rx39_csum_none: 0
>>      rx39_xdp_drop: 0
>>      rx39_xdp_redirect: 0
>>      rx39_lro_packets: 0
>>      rx39_lro_bytes: 0
>>      rx39_ecn_mark: 0
>>      rx39_removed_vlan_packets: 0
>>      rx39_wqe_err: 0
>>      rx39_mpwqe_filler_cqes: 0
>>      rx39_mpwqe_filler_strides: 0
>>      rx39_buff_alloc_err: 0
>>      rx39_cqe_compress_blks: 0
>>      rx39_cqe_compress_pkts: 0
>>      rx39_page_reuse: 0
>>      rx39_cache_reuse: 0
>>      rx39_cache_full: 0
>>      rx39_cache_empty: 2560
>>      rx39_cache_busy: 0
>>      rx39_cache_waive: 0
>>      rx39_congst_umr: 0
>>      rx39_arfs_err: 0
>>      rx39_xdp_tx_xmit: 0
>>      rx39_xdp_tx_full: 0
>>      rx39_xdp_tx_err: 0
>>      rx39_xdp_tx_cqes: 0
>>      rx40_packets: 0
>>      rx40_bytes: 0
>>      rx40_csum_complete: 0
>>      rx40_csum_unnecessary: 0
>>      rx40_csum_unnecessary_inner: 0
>>      rx40_csum_none: 0
>>      rx40_xdp_drop: 0
>>      rx40_xdp_redirect: 0
>>      rx40_lro_packets: 0
>>      rx40_lro_bytes: 0
>>      rx40_ecn_mark: 0
>>      rx40_removed_vlan_packets: 0
>>      rx40_wqe_err: 0
>>      rx40_mpwqe_filler_cqes: 0
>>      rx40_mpwqe_filler_strides: 0
>>      rx40_buff_alloc_err: 0
>>      rx40_cqe_compress_blks: 0
>>      rx40_cqe_compress_pkts: 0
>>      rx40_page_reuse: 0
>>      rx40_cache_reuse: 0
>>      rx40_cache_full: 0
>>      rx40_cache_empty: 2560
>>      rx40_cache_busy: 0
>>      rx40_cache_waive: 0
>>      rx40_congst_umr: 0
>>      rx40_arfs_err: 0
>>      rx40_xdp_tx_xmit: 0
>>      rx40_xdp_tx_full: 0
>>      rx40_xdp_tx_err: 0
>>      rx40_xdp_tx_cqes: 0
>>      rx41_packets: 0
>>      rx41_bytes: 0
>>      rx41_csum_complete: 0
>>      rx41_csum_unnecessary: 0
>>      rx41_csum_unnecessary_inner: 0
>>      rx41_csum_none: 0
>>      rx41_xdp_drop: 0
>>      rx41_xdp_redirect: 0
>>      rx41_lro_packets: 0
>>      rx41_lro_bytes: 0
>>      rx41_ecn_mark: 0
>>      rx41_removed_vlan_packets: 0
>>      rx41_wqe_err: 0
>>      rx41_mpwqe_filler_cqes: 0
>>      rx41_mpwqe_filler_strides: 0
>>      rx41_buff_alloc_err: 0
>>      rx41_cqe_compress_blks: 0
>>      rx41_cqe_compress_pkts: 0
>>      rx41_page_reuse: 0
>>      rx41_cache_reuse: 0
>>      rx41_cache_full: 0
>>      rx41_cache_empty: 2560
>>      rx41_cache_busy: 0
>>      rx41_cache_waive: 0
>>      rx41_congst_umr: 0
>>      rx41_arfs_err: 0
>>      rx41_xdp_tx_xmit: 0
>>      rx41_xdp_tx_full: 0
>>      rx41_xdp_tx_err: 0
>>      rx41_xdp_tx_cqes: 0
>>      rx42_packets: 0
>>      rx42_bytes: 0
>>      rx42_csum_complete: 0
>>      rx42_csum_unnecessary: 0
>>      rx42_csum_unnecessary_inner: 0
>>      rx42_csum_none: 0
>>      rx42_xdp_drop: 0
>>      rx42_xdp_redirect: 0
>>      rx42_lro_packets: 0
>>      rx42_lro_bytes: 0
>>      rx42_ecn_mark: 0
>>      rx42_removed_vlan_packets: 0
>>      rx42_wqe_err: 0
>>      rx42_mpwqe_filler_cqes: 0
>>      rx42_mpwqe_filler_strides: 0
>>      rx42_buff_alloc_err: 0
>>      rx42_cqe_compress_blks: 0
>>      rx42_cqe_compress_pkts: 0
>>      rx42_page_reuse: 0
>>      rx42_cache_reuse: 0
>>      rx42_cache_full: 0
>>      rx42_cache_empty: 2560
>>      rx42_cache_busy: 0
>>      rx42_cache_waive: 0
>>      rx42_congst_umr: 0
>>      rx42_arfs_err: 0
>>      rx42_xdp_tx_xmit: 0
>>      rx42_xdp_tx_full: 0
>>      rx42_xdp_tx_err: 0
>>      rx42_xdp_tx_cqes: 0
>>      rx43_packets: 0
>>      rx43_bytes: 0
>>      rx43_csum_complete: 0
>>      rx43_csum_unnecessary: 0
>>      rx43_csum_unnecessary_inner: 0
>>      rx43_csum_none: 0
>>      rx43_xdp_drop: 0
>>      rx43_xdp_redirect: 0
>>      rx43_lro_packets: 0
>>      rx43_lro_bytes: 0
>>      rx43_ecn_mark: 0
>>      rx43_removed_vlan_packets: 0
>>      rx43_wqe_err: 0
>>      rx43_mpwqe_filler_cqes: 0
>>      rx43_mpwqe_filler_strides: 0
>>      rx43_buff_alloc_err: 0
>>      rx43_cqe_compress_blks: 0
>>      rx43_cqe_compress_pkts: 0
>>      rx43_page_reuse: 0
>>      rx43_cache_reuse: 0
>>      rx43_cache_full: 0
>>      rx43_cache_empty: 2560
>>      rx43_cache_busy: 0
>>      rx43_cache_waive: 0
>>      rx43_congst_umr: 0
>>      rx43_arfs_err: 0
>>      rx43_xdp_tx_xmit: 0
>>      rx43_xdp_tx_full: 0
>>      rx43_xdp_tx_err: 0
>>      rx43_xdp_tx_cqes: 0
>>      rx44_packets: 0
>>      rx44_bytes: 0
>>      rx44_csum_complete: 0
>>      rx44_csum_unnecessary: 0
>>      rx44_csum_unnecessary_inner: 0
>>      rx44_csum_none: 0
>>      rx44_xdp_drop: 0
>>      rx44_xdp_redirect: 0
>>      rx44_lro_packets: 0
>>      rx44_lro_bytes: 0
>>      rx44_ecn_mark: 0
>>      rx44_removed_vlan_packets: 0
>>      rx44_wqe_err: 0
>>      rx44_mpwqe_filler_cqes: 0
>>      rx44_mpwqe_filler_strides: 0
>>      rx44_buff_alloc_err: 0
>>      rx44_cqe_compress_blks: 0
>>      rx44_cqe_compress_pkts: 0
>>      rx44_page_reuse: 0
>>      rx44_cache_reuse: 0
>>      rx44_cache_full: 0
>>      rx44_cache_empty: 2560
>>      rx44_cache_busy: 0
>>      rx44_cache_waive: 0
>>      rx44_congst_umr: 0
>>      rx44_arfs_err: 0
>>      rx44_xdp_tx_xmit: 0
>>      rx44_xdp_tx_full: 0
>>      rx44_xdp_tx_err: 0
>>      rx44_xdp_tx_cqes: 0
>>      rx45_packets: 0
>>      rx45_bytes: 0
>>      rx45_csum_complete: 0
>>      rx45_csum_unnecessary: 0
>>      rx45_csum_unnecessary_inner: 0
>>      rx45_csum_none: 0
>>      rx45_xdp_drop: 0
>>      rx45_xdp_redirect: 0
>>      rx45_lro_packets: 0
>>      rx45_lro_bytes: 0
>>      rx45_ecn_mark: 0
>>      rx45_removed_vlan_packets: 0
>>      rx45_wqe_err: 0
>>      rx45_mpwqe_filler_cqes: 0
>>      rx45_mpwqe_filler_strides: 0
>>      rx45_buff_alloc_err: 0
>>      rx45_cqe_compress_blks: 0
>>      rx45_cqe_compress_pkts: 0
>>      rx45_page_reuse: 0
>>      rx45_cache_reuse: 0
>>      rx45_cache_full: 0
>>      rx45_cache_empty: 2560
>>      rx45_cache_busy: 0
>>      rx45_cache_waive: 0
>>      rx45_congst_umr: 0
>>      rx45_arfs_err: 0
>>      rx45_xdp_tx_xmit: 0
>>      rx45_xdp_tx_full: 0
>>      rx45_xdp_tx_err: 0
>>      rx45_xdp_tx_cqes: 0
>>      rx46_packets: 0
>>      rx46_bytes: 0
>>      rx46_csum_complete: 0
>>      rx46_csum_unnecessary: 0
>>      rx46_csum_unnecessary_inner: 0
>>      rx46_csum_none: 0
>>      rx46_xdp_drop: 0
>>      rx46_xdp_redirect: 0
>>      rx46_lro_packets: 0
>>      rx46_lro_bytes: 0
>>      rx46_ecn_mark: 0
>>      rx46_removed_vlan_packets: 0
>>      rx46_wqe_err: 0
>>      rx46_mpwqe_filler_cqes: 0
>>      rx46_mpwqe_filler_strides: 0
>>      rx46_buff_alloc_err: 0
>>      rx46_cqe_compress_blks: 0
>>      rx46_cqe_compress_pkts: 0
>>      rx46_page_reuse: 0
>>      rx46_cache_reuse: 0
>>      rx46_cache_full: 0
>>      rx46_cache_empty: 2560
>>      rx46_cache_busy: 0
>>      rx46_cache_waive: 0
>>      rx46_congst_umr: 0
>>      rx46_arfs_err: 0
>>      rx46_xdp_tx_xmit: 0
>>      rx46_xdp_tx_full: 0
>>      rx46_xdp_tx_err: 0
>>      rx46_xdp_tx_cqes: 0
>>      rx47_packets: 0
>>      rx47_bytes: 0
>>      rx47_csum_complete: 0
>>      rx47_csum_unnecessary: 0
>>      rx47_csum_unnecessary_inner: 0
>>      rx47_csum_none: 0
>>      rx47_xdp_drop: 0
>>      rx47_xdp_redirect: 0
>>      rx47_lro_packets: 0
>>      rx47_lro_bytes: 0
>>      rx47_ecn_mark: 0
>>      rx47_removed_vlan_packets: 0
>>      rx47_wqe_err: 0
>>      rx47_mpwqe_filler_cqes: 0
>>      rx47_mpwqe_filler_strides: 0
>>      rx47_buff_alloc_err: 0
>>      rx47_cqe_compress_blks: 0
>>      rx47_cqe_compress_pkts: 0
>>      rx47_page_reuse: 0
>>      rx47_cache_reuse: 0
>>      rx47_cache_full: 0
>>      rx47_cache_empty: 2560
>>      rx47_cache_busy: 0
>>      rx47_cache_waive: 0
>>      rx47_congst_umr: 0
>>      rx47_arfs_err: 0
>>      rx47_xdp_tx_xmit: 0
>>      rx47_xdp_tx_full: 0
>>      rx47_xdp_tx_err: 0
>>      rx47_xdp_tx_cqes: 0
>>      rx48_packets: 0
>>      rx48_bytes: 0
>>      rx48_csum_complete: 0
>>      rx48_csum_unnecessary: 0
>>      rx48_csum_unnecessary_inner: 0
>>      rx48_csum_none: 0
>>      rx48_xdp_drop: 0
>>      rx48_xdp_redirect: 0
>>      rx48_lro_packets: 0
>>      rx48_lro_bytes: 0
>>      rx48_ecn_mark: 0
>>      rx48_removed_vlan_packets: 0
>>      rx48_wqe_err: 0
>>      rx48_mpwqe_filler_cqes: 0
>>      rx48_mpwqe_filler_strides: 0
>>      rx48_buff_alloc_err: 0
>>      rx48_cqe_compress_blks: 0
>>      rx48_cqe_compress_pkts: 0
>>      rx48_page_reuse: 0
>>      rx48_cache_reuse: 0
>>      rx48_cache_full: 0
>>      rx48_cache_empty: 2560
>>      rx48_cache_busy: 0
>>      rx48_cache_waive: 0
>>      rx48_congst_umr: 0
>>      rx48_arfs_err: 0
>>      rx48_xdp_tx_xmit: 0
>>      rx48_xdp_tx_full: 0
>>      rx48_xdp_tx_err: 0
>>      rx48_xdp_tx_cqes: 0
>>      rx49_packets: 0
>>      rx49_bytes: 0
>>      rx49_csum_complete: 0
>>      rx49_csum_unnecessary: 0
>>      rx49_csum_unnecessary_inner: 0
>>      rx49_csum_none: 0
>>      rx49_xdp_drop: 0
>>      rx49_xdp_redirect: 0
>>      rx49_lro_packets: 0
>>      rx49_lro_bytes: 0
>>      rx49_ecn_mark: 0
>>      rx49_removed_vlan_packets: 0
>>      rx49_wqe_err: 0
>>      rx49_mpwqe_filler_cqes: 0
>>      rx49_mpwqe_filler_strides: 0
>>      rx49_buff_alloc_err: 0
>>      rx49_cqe_compress_blks: 0
>>      rx49_cqe_compress_pkts: 0
>>      rx49_page_reuse: 0
>>      rx49_cache_reuse: 0
>>      rx49_cache_full: 0
>>      rx49_cache_empty: 2560
>>      rx49_cache_busy: 0
>>      rx49_cache_waive: 0
>>      rx49_congst_umr: 0
>>      rx49_arfs_err: 0
>>      rx49_xdp_tx_xmit: 0
>>      rx49_xdp_tx_full: 0
>>      rx49_xdp_tx_err: 0
>>      rx49_xdp_tx_cqes: 0
>>      rx50_packets: 0
>>      rx50_bytes: 0
>>      rx50_csum_complete: 0
>>      rx50_csum_unnecessary: 0
>>      rx50_csum_unnecessary_inner: 0
>>      rx50_csum_none: 0
>>      rx50_xdp_drop: 0
>>      rx50_xdp_redirect: 0
>>      rx50_lro_packets: 0
>>      rx50_lro_bytes: 0
>>      rx50_ecn_mark: 0
>>      rx50_removed_vlan_packets: 0
>>      rx50_wqe_err: 0
>>      rx50_mpwqe_filler_cqes: 0
>>      rx50_mpwqe_filler_strides: 0
>>      rx50_buff_alloc_err: 0
>>      rx50_cqe_compress_blks: 0
>>      rx50_cqe_compress_pkts: 0
>>      rx50_page_reuse: 0
>>      rx50_cache_reuse: 0
>>      rx50_cache_full: 0
>>      rx50_cache_empty: 2560
>>      rx50_cache_busy: 0
>>      rx50_cache_waive: 0
>>      rx50_congst_umr: 0
>>      rx50_arfs_err: 0
>>      rx50_xdp_tx_xmit: 0
>>      rx50_xdp_tx_full: 0
>>      rx50_xdp_tx_err: 0
>>      rx50_xdp_tx_cqes: 0
>>      rx51_packets: 0
>>      rx51_bytes: 0
>>      rx51_csum_complete: 0
>>      rx51_csum_unnecessary: 0
>>      rx51_csum_unnecessary_inner: 0
>>      rx51_csum_none: 0
>>      rx51_xdp_drop: 0
>>      rx51_xdp_redirect: 0
>>      rx51_lro_packets: 0
>>      rx51_lro_bytes: 0
>>      rx51_ecn_mark: 0
>>      rx51_removed_vlan_packets: 0
>>      rx51_wqe_err: 0
>>      rx51_mpwqe_filler_cqes: 0
>>      rx51_mpwqe_filler_strides: 0
>>      rx51_buff_alloc_err: 0
>>      rx51_cqe_compress_blks: 0
>>      rx51_cqe_compress_pkts: 0
>>      rx51_page_reuse: 0
>>      rx51_cache_reuse: 0
>>      rx51_cache_full: 0
>>      rx51_cache_empty: 2560
>>      rx51_cache_busy: 0
>>      rx51_cache_waive: 0
>>      rx51_congst_umr: 0
>>      rx51_arfs_err: 0
>>      rx51_xdp_tx_xmit: 0
>>      rx51_xdp_tx_full: 0
>>      rx51_xdp_tx_err: 0
>>      rx51_xdp_tx_cqes: 0
>>      rx52_packets: 0
>>      rx52_bytes: 0
>>      rx52_csum_complete: 0
>>      rx52_csum_unnecessary: 0
>>      rx52_csum_unnecessary_inner: 0
>>      rx52_csum_none: 0
>>      rx52_xdp_drop: 0
>>      rx52_xdp_redirect: 0
>>      rx52_lro_packets: 0
>>      rx52_lro_bytes: 0
>>      rx52_ecn_mark: 0
>>      rx52_removed_vlan_packets: 0
>>      rx52_wqe_err: 0
>>      rx52_mpwqe_filler_cqes: 0
>>      rx52_mpwqe_filler_strides: 0
>>      rx52_buff_alloc_err: 0
>>      rx52_cqe_compress_blks: 0
>>      rx52_cqe_compress_pkts: 0
>>      rx52_page_reuse: 0
>>      rx52_cache_reuse: 0
>>      rx52_cache_full: 0
>>      rx52_cache_empty: 2560
>>      rx52_cache_busy: 0
>>      rx52_cache_waive: 0
>>      rx52_congst_umr: 0
>>      rx52_arfs_err: 0
>>      rx52_xdp_tx_xmit: 0
>>      rx52_xdp_tx_full: 0
>>      rx52_xdp_tx_err: 0
>>      rx52_xdp_tx_cqes: 0
>>      rx53_packets: 0
>>      rx53_bytes: 0
>>      rx53_csum_complete: 0
>>      rx53_csum_unnecessary: 0
>>      rx53_csum_unnecessary_inner: 0
>>      rx53_csum_none: 0
>>      rx53_xdp_drop: 0
>>      rx53_xdp_redirect: 0
>>      rx53_lro_packets: 0
>>      rx53_lro_bytes: 0
>>      rx53_ecn_mark: 0
>>      rx53_removed_vlan_packets: 0
>>      rx53_wqe_err: 0
>>      rx53_mpwqe_filler_cqes: 0
>>      rx53_mpwqe_filler_strides: 0
>>      rx53_buff_alloc_err: 0
>>      rx53_cqe_compress_blks: 0
>>      rx53_cqe_compress_pkts: 0
>>      rx53_page_reuse: 0
>>      rx53_cache_reuse: 0
>>      rx53_cache_full: 0
>>      rx53_cache_empty: 2560
>>      rx53_cache_busy: 0
>>      rx53_cache_waive: 0
>>      rx53_congst_umr: 0
>>      rx53_arfs_err: 0
>>      rx53_xdp_tx_xmit: 0
>>      rx53_xdp_tx_full: 0
>>      rx53_xdp_tx_err: 0
>>      rx53_xdp_tx_cqes: 0
>>      rx54_packets: 0
>>      rx54_bytes: 0
>>      rx54_csum_complete: 0
>>      rx54_csum_unnecessary: 0
>>      rx54_csum_unnecessary_inner: 0
>>      rx54_csum_none: 0
>>      rx54_xdp_drop: 0
>>      rx54_xdp_redirect: 0
>>      rx54_lro_packets: 0
>>      rx54_lro_bytes: 0
>>      rx54_ecn_mark: 0
>>      rx54_removed_vlan_packets: 0
>>      rx54_wqe_err: 0
>>      rx54_mpwqe_filler_cqes: 0
>>      rx54_mpwqe_filler_strides: 0
>>      rx54_buff_alloc_err: 0
>>      rx54_cqe_compress_blks: 0
>>      rx54_cqe_compress_pkts: 0
>>      rx54_page_reuse: 0
>>      rx54_cache_reuse: 0
>>      rx54_cache_full: 0
>>      rx54_cache_empty: 2560
>>      rx54_cache_busy: 0
>>      rx54_cache_waive: 0
>>      rx54_congst_umr: 0
>>      rx54_arfs_err: 0
>>      rx54_xdp_tx_xmit: 0
>>      rx54_xdp_tx_full: 0
>>      rx54_xdp_tx_err: 0
>>      rx54_xdp_tx_cqes: 0
>>      rx55_packets: 0
>>      rx55_bytes: 0
>>      rx55_csum_complete: 0
>>      rx55_csum_unnecessary: 0
>>      rx55_csum_unnecessary_inner: 0
>>      rx55_csum_none: 0
>>      rx55_xdp_drop: 0
>>      rx55_xdp_redirect: 0
>>      rx55_lro_packets: 0
>>      rx55_lro_bytes: 0
>>      rx55_ecn_mark: 0
>>      rx55_removed_vlan_packets: 0
>>      rx55_wqe_err: 0
>>      rx55_mpwqe_filler_cqes: 0
>>      rx55_mpwqe_filler_strides: 0
>>      rx55_buff_alloc_err: 0
>>      rx55_cqe_compress_blks: 0
>>      rx55_cqe_compress_pkts: 0
>>      rx55_page_reuse: 0
>>      rx55_cache_reuse: 0
>>      rx55_cache_full: 0
>>      rx55_cache_empty: 2560
>>      rx55_cache_busy: 0
>>      rx55_cache_waive: 0
>>      rx55_congst_umr: 0
>>      rx55_arfs_err: 0
>>      rx55_xdp_tx_xmit: 0
>>      rx55_xdp_tx_full: 0
>>      rx55_xdp_tx_err: 0
>>      rx55_xdp_tx_cqes: 0
>>      tx0_packets: 24512439668
>>      tx0_bytes: 15287569052791
>>      tx0_tso_packets: 1536157106
>>      tx0_tso_bytes: 8571753637944
>>      tx0_tso_inner_packets: 0
>>      tx0_tso_inner_bytes: 0
>>      tx0_csum_partial: 2132156117
>>      tx0_csum_partial_inner: 0
>>      tx0_added_vlan_packets: 19906601448
>>      tx0_nop: 308098536
>>      tx0_csum_none: 17774445331
>>      tx0_stopped: 19625
>>      tx0_dropped: 0
>>      tx0_xmit_more: 67864870
>>      tx0_recover: 0
>>      tx0_cqes: 19838744246
>>      tx0_wake: 19624
>>      tx0_cqe_err: 0
>>      tx1_packets: 22598557053
>>      tx1_bytes: 13568850145010
>>      tx1_tso_packets: 1369529475
>>      tx1_tso_bytes: 7661777265382
>>      tx1_tso_inner_packets: 0
>>      tx1_tso_inner_bytes: 0
>>      tx1_csum_partial: 1884639496
>>      tx1_csum_partial_inner: 0
>>      tx1_added_vlan_packets: 18468333696
>>      tx1_nop: 281301783
>>      tx1_csum_none: 16583694200
>>      tx1_stopped: 19457
>>      tx1_dropped: 0
>>      tx1_xmit_more: 55170875
>>      tx1_recover: 0
>>      tx1_cqes: 18413169824
>>      tx1_wake: 19455
>>      tx1_cqe_err: 0
>>      tx2_packets: 22821611433
>>      tx2_bytes: 13752535163683
>>      tx2_tso_packets: 1396978825
>>      tx2_tso_bytes: 7774704508463
>>      tx2_tso_inner_packets: 0
>>      tx2_tso_inner_bytes: 0
>>      tx2_csum_partial: 1897834558
>>      tx2_csum_partial_inner: 0
>>      tx2_added_vlan_packets: 18641958085
>>      tx2_nop: 286934891
>>      tx2_csum_none: 16744123527
>>      tx2_stopped: 13214
>>      tx2_dropped: 0
>>      tx2_xmit_more: 61749446
>>      tx2_recover: 0
>>      tx2_cqes: 18580215654
>>      tx2_wake: 13214
>>      tx2_cqe_err: 0
>>      tx3_packets: 22580809948
>>      tx3_bytes: 13730542936609
>>      tx3_tso_packets: 1370434579
>>      tx3_tso_bytes: 7605636711455
>>      tx3_tso_inner_packets: 0
>>      tx3_tso_inner_bytes: 0
>>      tx3_csum_partial: 1865573748
>>      tx3_csum_partial_inner: 0
>>      tx3_added_vlan_packets: 18491873644
>>      tx3_nop: 281195875
>>      tx3_csum_none: 16626299896
>>      tx3_stopped: 12542
>>      tx3_dropped: 0
>>      tx3_xmit_more: 57681647
>>      tx3_recover: 0
>>      tx3_cqes: 18434198757
>>      tx3_wake: 12540
>>      tx3_cqe_err: 0
>>      tx4_packets: 27801801208
>>      tx4_bytes: 17058453171137
>>      tx4_tso_packets: 1740500105
>>      tx4_tso_bytes: 9474905622036
>>      tx4_tso_inner_packets: 0
>>      tx4_tso_inner_bytes: 0
>>      tx4_csum_partial: 2279225376
>>      tx4_csum_partial_inner: 0
>>      tx4_added_vlan_packets: 22744081633
>>      tx4_nop: 349753979
>>      tx4_csum_none: 20464856257
>>      tx4_stopped: 14816
>>      tx4_dropped: 0
>>      tx4_xmit_more: 65469322
>>      tx4_recover: 0
>>      tx4_cqes: 22678618972
>>      tx4_wake: 14816
>>      tx4_cqe_err: 0
>>      tx5_packets: 25099783024
>>      tx5_bytes: 14917740698381
>>      tx5_tso_packets: 1512988013
>>      tx5_tso_bytes: 8571208921023
>>      tx5_tso_inner_packets: 0
>>      tx5_tso_inner_bytes: 0
>>      tx5_csum_partial: 2078498561
>>      tx5_csum_partial_inner: 0
>>      tx5_added_vlan_packets: 20465533760
>>      tx5_nop: 312614719
>>      tx5_csum_none: 18387035199
>>      tx5_stopped: 4605
>>      tx5_dropped: 0
>>      tx5_xmit_more: 64188936
>>      tx5_recover: 0
>>      tx5_cqes: 20401350718
>>      tx5_wake: 4604
>>      tx5_cqe_err: 0
>>      tx6_packets: 25025504896
>>      tx6_bytes: 14908021946070
>>      tx6_tso_packets: 1515718977
>>      tx6_tso_bytes: 8511442522461
>>      tx6_tso_inner_packets: 0
>>      tx6_tso_inner_bytes: 0
>>      tx6_csum_partial: 2056378610
>>      tx6_csum_partial_inner: 0
>>      tx6_added_vlan_packets: 20434066400
>>      tx6_nop: 310594020
>>      tx6_csum_none: 18377687790
>>      tx6_stopped: 15234
>>      tx6_dropped: 0
>>      tx6_xmit_more: 61130422
>>      tx6_recover: 0
>>      tx6_cqes: 20372943611
>>      tx6_wake: 15234
>>      tx6_cqe_err: 0
>>      tx7_packets: 25457096169
>>      tx7_bytes: 15456289446172
>>      tx7_tso_packets: 1553342799
>>      tx7_tso_bytes: 8764550988105
>>      tx7_tso_inner_packets: 0
>>      tx7_tso_inner_bytes: 0
>>      tx7_csum_partial: 2105765233
>>      tx7_csum_partial_inner: 0
>>      tx7_added_vlan_packets: 20720382377
>>      tx7_nop: 319044853
>>      tx7_csum_none: 18614617145
>>      tx7_stopped: 18745
>>      tx7_dropped: 0
>>      tx7_xmit_more: 57050107
>>      tx7_recover: 0
>>      tx7_cqes: 20663340775
>>      tx7_wake: 18746
>>      tx7_cqe_err: 0
>>      tx8_packets: 25389771649
>>      tx8_bytes: 15225503883962
>>      tx8_tso_packets: 1563367648
>>      tx8_tso_bytes: 8710384514258
>>      tx8_tso_inner_packets: 0
>>      tx8_tso_inner_bytes: 0
>>      tx8_csum_partial: 2106586634
>>      tx8_csum_partial_inner: 0
>>      tx8_added_vlan_packets: 20704676274
>>      tx8_nop: 318149261
>>      tx8_csum_none: 18598089640
>>      tx8_stopped: 4733
>>      tx8_dropped: 0
>>      tx8_xmit_more: 61014317
>>      tx8_recover: 0
>>      tx8_cqes: 20643667301
>>      tx8_wake: 4735
>>      tx8_cqe_err: 0
>>      tx9_packets: 25521500166
>>      tx9_bytes: 15302293145755
>>      tx9_tso_packets: 1546316697
>>      tx9_tso_bytes: 8770688145926
>>      tx9_tso_inner_packets: 0
>>      tx9_tso_inner_bytes: 0
>>      tx9_csum_partial: 2097652880
>>      tx9_csum_partial_inner: 0
>>      tx9_added_vlan_packets: 20778408432
>>      tx9_nop: 318538543
>>      tx9_csum_none: 18680755556
>>      tx9_stopped: 16118
>>      tx9_dropped: 0
>>      tx9_xmit_more: 68509728
>>      tx9_recover: 0
>>      tx9_cqes: 20709906498
>>      tx9_wake: 16118
>>      tx9_cqe_err: 0
>>      tx10_packets: 25451605829
>>      tx10_bytes: 15386896170792
>>      tx10_tso_packets: 1576473520
>>      tx10_tso_bytes: 8880888676383
>>      tx10_tso_inner_packets: 0
>>      tx10_tso_inner_bytes: 0
>>      tx10_csum_partial: 2129796141
>>      tx10_csum_partial_inner: 0
>>      tx10_added_vlan_packets: 20659622590
>>      tx10_nop: 319117433
>>      tx10_csum_none: 18529826450
>>      tx10_stopped: 20187
>>      tx10_dropped: 0
>>      tx10_xmit_more: 58892184
>>      tx10_recover: 0
>>      tx10_cqes: 20600737739
>>      tx10_wake: 20188
>>      tx10_cqe_err: 0
>>      tx11_packets: 27008919793
>>      tx11_bytes: 16587719213058
>>      tx11_tso_packets: 1734884654
>>      tx11_tso_bytes: 9475681471870
>>      tx11_tso_inner_packets: 0
>>      tx11_tso_inner_bytes: 0
>>      tx11_csum_partial: 2296162292
>>      tx11_csum_partial_inner: 0
>>      tx11_added_vlan_packets: 21943096263
>>      tx11_nop: 344188182
>>      tx11_csum_none: 19646933971
>>      tx11_stopped: 9703
>>      tx11_dropped: 0
>>      tx11_xmit_more: 66530718
>>      tx11_recover: 0
>>      tx11_cqes: 21876571667
>>      tx11_wake: 9704
>>      tx11_cqe_err: 0
>>      tx12_packets: 25969493269
>>      tx12_bytes: 15980767963416
>>      tx12_tso_packets: 1671396456
>>      tx12_tso_bytes: 9268973672821
>>      tx12_tso_inner_packets: 0
>>      tx12_tso_inner_bytes: 0
>>      tx12_csum_partial: 2243809182
>>      tx12_csum_partial_inner: 0
>>      tx12_added_vlan_packets: 20980642456
>>      tx12_nop: 330241007
>>      tx12_csum_none: 18736833276
>>      tx12_stopped: 10341
>>      tx12_dropped: 0
>>      tx12_xmit_more: 57834100
>>      tx12_recover: 0
>>      tx12_cqes: 20922815079
>>      tx12_wake: 10342
>>      tx12_cqe_err: 0
>>      tx13_packets: 25332762261
>>      tx13_bytes: 15353213283280
>>      tx13_tso_packets: 1577433599
>>      tx13_tso_bytes: 8785240284281
>>      tx13_tso_inner_packets: 0
>>      tx13_tso_inner_bytes: 0
>>      tx13_csum_partial: 2110640515
>>      tx13_csum_partial_inner: 0
>>      tx13_added_vlan_packets: 20605670910
>>      tx13_nop: 319805741
>>      tx13_csum_none: 18495030395
>>      tx13_stopped: 7006
>>      tx13_dropped: 0
>>      tx13_xmit_more: 58314402
>>      tx13_recover: 0
>>      tx13_cqes: 20547362770
>>      tx13_wake: 7008
>>      tx13_cqe_err: 0
>>      tx14_packets: 26333743548
>>      tx14_bytes: 16070719060573
>>      tx14_tso_packets: 1677922970
>>      tx14_tso_bytes: 9240299765487
>>      tx14_tso_inner_packets: 0
>>      tx14_tso_inner_bytes: 0
>>      tx14_csum_partial: 2215668906
>>      tx14_csum_partial_inner: 0
>>      tx14_added_vlan_packets: 21384410786
>>      tx14_nop: 332734939
>>      tx14_csum_none: 19168741880
>>      tx14_stopped: 13160
>>      tx14_dropped: 0
>>      tx14_xmit_more: 57650391
>>      tx14_recover: 0
>>      tx14_cqes: 21326767783
>>      tx14_wake: 13161
>>      tx14_cqe_err: 0
>>      tx15_packets: 26824968971
>>      tx15_bytes: 16687994233452
>>      tx15_tso_packets: 1755745052
>>      tx15_tso_bytes: 9533814012441
>>      tx15_tso_inner_packets: 0
>>      tx15_tso_inner_bytes: 0
>>      tx15_csum_partial: 2304778064
>>      tx15_csum_partial_inner: 0
>>      tx15_added_vlan_packets: 21740906107
>>      tx15_nop: 344143287
>>      tx15_csum_none: 19436128058
>>      tx15_stopped: 75
>>      tx15_dropped: 0
>>      tx15_xmit_more: 63325832
>>      tx15_recover: 0
>>      tx15_cqes: 21677585345
>>      tx15_wake: 74
>>      tx15_cqe_err: 0
>>      tx16_packets: 24488158946
>>      tx16_bytes: 15027415004570
>>      tx16_tso_packets: 1559127391
>>      tx16_tso_bytes: 8658691917845
>>      tx16_tso_inner_packets: 0
>>      tx16_tso_inner_bytes: 0
>>      tx16_csum_partial: 2075856395
>>      tx16_csum_partial_inner: 0
>>      tx16_added_vlan_packets: 19835695731
>>      tx16_nop: 308464189
>>      tx16_csum_none: 17759839340
>>      tx16_stopped: 4567
>>      tx16_dropped: 0
>>      tx16_xmit_more: 62631422
>>      tx16_recover: 0
>>      tx16_cqes: 19773070012
>>      tx16_wake: 4568
>>      tx16_cqe_err: 0
>>      tx17_packets: 24700413784
>>      tx17_bytes: 15216529713715
>>      tx17_tso_packets: 1597555108
>>      tx17_tso_bytes: 8773728661243
>>      tx17_tso_inner_packets: 0
>>      tx17_tso_inner_bytes: 0
>>      tx17_csum_partial: 2127177297
>>      tx17_csum_partial_inner: 0
>>      tx17_added_vlan_packets: 20003144561
>>      tx17_nop: 313356918
>>      tx17_csum_none: 17875967264
>>      tx17_stopped: 12572
>>      tx17_dropped: 0
>>      tx17_xmit_more: 62742980
>>      tx17_recover: 0
>>      tx17_cqes: 19940407615
>>      tx17_wake: 12573
>>      tx17_cqe_err: 0
>>      tx18_packets: 24887710046
>>      tx18_bytes: 15245034034664
>>      tx18_tso_packets: 1582550520
>>      tx18_tso_bytes: 8782692335483
>>      tx18_tso_inner_packets: 0
>>      tx18_tso_inner_bytes: 0
>>      tx18_csum_partial: 2084514331
>>      tx18_csum_partial_inner: 0
>>      tx18_added_vlan_packets: 20173879181
>>      tx18_nop: 314818702
>>      tx18_csum_none: 18089364850
>>      tx18_stopped: 21366
>>      tx18_dropped: 0
>>      tx18_xmit_more: 62485819
>>      tx18_recover: 0
>>      tx18_cqes: 20111400935
>>      tx18_wake: 21366
>>      tx18_cqe_err: 0
>>      tx19_packets: 24831057648
>>      tx19_bytes: 15164663890576
>>      tx19_tso_packets: 1599135489
>>      tx19_tso_bytes: 8756045449746
>>      tx19_tso_inner_packets: 0
>>      tx19_tso_inner_bytes: 0
>>      tx19_csum_partial: 2119746608
>>      tx19_csum_partial_inner: 0
>>      tx19_added_vlan_packets: 20143573903
>>      tx19_nop: 316966450
>>      tx19_csum_none: 18023827295
>>      tx19_stopped: 11431
>>      tx19_dropped: 0
>>      tx19_xmit_more: 57535904
>>      tx19_recover: 0
>>      tx19_cqes: 20086045325
>>      tx19_wake: 11431
>>      tx19_cqe_err: 0
>>      tx20_packets: 21943735263
>>      tx20_bytes: 13528749492187
>>      tx20_tso_packets: 1390048103
>>      tx20_tso_bytes: 7629058809637
>>      tx20_tso_inner_packets: 0
>>      tx20_tso_inner_bytes: 0
>>      tx20_csum_partial: 1848533941
>>      tx20_csum_partial_inner: 0
>>      tx20_added_vlan_packets: 17861417651
>>      tx20_nop: 276840365
>>      tx20_csum_none: 16012883710
>>      tx20_stopped: 38457
>>      tx20_dropped: 0
>>      tx20_xmit_more: 57042753
>>      tx20_recover: 0
>>      tx20_cqes: 17804384839
>>      tx20_wake: 38457
>>      tx20_cqe_err: 0
>>      tx21_packets: 21476926958
>>      tx21_bytes: 13096410597896
>>      tx21_tso_packets: 1367724090
>>      tx21_tso_bytes: 7568364585127
>>      tx21_tso_inner_packets: 0
>>      tx21_tso_inner_bytes: 0
>>      tx21_csum_partial: 1830570727
>>      tx21_csum_partial_inner: 0
>>      tx21_added_vlan_packets: 17421087814
>>      tx21_nop: 270611519
>>      tx21_csum_none: 15590517087
>>      tx21_stopped: 31213
>>      tx21_dropped: 0
>>      tx21_xmit_more: 60305389
>>      tx21_recover: 0
>>      tx21_cqes: 17360791205
>>      tx21_wake: 31213
>>      tx21_cqe_err: 0
>>      tx22_packets: 21819106444
>>      tx22_bytes: 13492871887100
>>      tx22_tso_packets: 1387002018
>>      tx22_tso_bytes: 7617705727669
>>      tx22_tso_inner_packets: 0
>>      tx22_tso_inner_bytes: 0
>>      tx22_csum_partial: 1853632107
>>      tx22_csum_partial_inner: 0
>>      tx22_added_vlan_packets: 17743255447
>>      tx22_nop: 274820992
>>      tx22_csum_none: 15889623340
>>      tx22_stopped: 24814
>>      tx22_dropped: 0
>>      tx22_xmit_more: 60811304
>>      tx22_recover: 0
>>      tx22_cqes: 17682451111
>>      tx22_wake: 24815
>>      tx22_cqe_err: 0
>>      tx23_packets: 21830455800
>>      tx23_bytes: 13427551902532
>>      tx23_tso_packets: 1388556038
>>      tx23_tso_bytes: 7604040587125
>>      tx23_tso_inner_packets: 0
>>      tx23_tso_inner_bytes: 0
>>      tx23_csum_partial: 1850819694
>>      tx23_csum_partial_inner: 0
>>      tx23_added_vlan_packets: 17761271122
>>      tx23_nop: 275142775
>>      tx23_csum_none: 15910451428
>>      tx23_stopped: 29899
>>      tx23_dropped: 0
>>      tx23_xmit_more: 58924909
>>      tx23_recover: 0
>>      tx23_cqes: 17702355187
>>      tx23_wake: 29898
>>      tx23_cqe_err: 0
>>      tx24_packets: 21961484213
>>      tx24_bytes: 13531373062497
>>      tx24_tso_packets: 1394697504
>>      tx24_tso_bytes: 7663866609308
>>      tx24_tso_inner_packets: 0
>>      tx24_tso_inner_bytes: 0
>>      tx24_csum_partial: 1857072074
>>      tx24_csum_partial_inner: 0
>>      tx24_added_vlan_packets: 17856887568
>>      tx24_nop: 276352855
>>      tx24_csum_none: 15999815494
>>      tx24_stopped: 33924
>>      tx24_dropped: 0
>>      tx24_xmit_more: 63992426
>>      tx24_recover: 0
>>      tx24_cqes: 17792905243
>>      tx24_wake: 33923
>>      tx24_cqe_err: 0
>>      tx25_packets: 21853593838
>>      tx25_bytes: 13357487830519
>>      tx25_tso_packets: 1398822411
>>      tx25_tso_bytes: 7691191518838
>>      tx25_tso_inner_packets: 0
>>      tx25_tso_inner_bytes: 0
>>      tx25_csum_partial: 1869483109
>>      tx25_csum_partial_inner: 0
>>      tx25_added_vlan_packets: 17734634614
>>      tx25_nop: 276327643
>>      tx25_csum_none: 15865151505
>>      tx25_stopped: 38651
>>      tx25_dropped: 0
>>      tx25_xmit_more: 56410535
>>      tx25_recover: 0
>>      tx25_cqes: 17678234537
>>      tx25_wake: 38650
>>      tx25_cqe_err: 0
>>      tx26_packets: 21480261205
>>      tx26_bytes: 13148973015935
>>      tx26_tso_packets: 1348132284
>>      tx26_tso_bytes: 7523489481775
>>      tx26_tso_inner_packets: 0
>>      tx26_tso_inner_bytes: 0
>>      tx26_csum_partial: 1839740745
>>      tx26_csum_partial_inner: 0
>>      tx26_added_vlan_packets: 17430592911
>>      tx26_nop: 270367836
>>      tx26_csum_none: 15590852166
>>      tx26_stopped: 34044
>>      tx26_dropped: 0
>>      tx26_xmit_more: 59870114
>>      tx26_recover: 0
>>      tx26_cqes: 17370736612
>>      tx26_wake: 34043
>>      tx26_cqe_err: 0
>>      tx27_packets: 22694273108
>>      tx27_bytes: 14135473431004
>>      tx27_tso_packets: 1418371875
>>      tx27_tso_bytes: 7784842263038
>>      tx27_tso_inner_packets: 0
>>      tx27_tso_inner_bytes: 0
>>      tx27_csum_partial: 1919170584
>>      tx27_csum_partial_inner: 0
>>      tx27_added_vlan_packets: 18520826023
>>      tx27_nop: 286296272
>>      tx27_csum_none: 16601655439
>>      tx27_stopped: 38125
>>      tx27_dropped: 0
>>      tx27_xmit_more: 72749775
>>      tx27_recover: 0
>>      tx27_cqes: 18448090270
>>      tx27_wake: 38127
>>      tx27_cqe_err: 0
>>      tx28_packets: 0
>>      tx28_bytes: 0
>>      tx28_tso_packets: 0
>>      tx28_tso_bytes: 0
>>      tx28_tso_inner_packets: 0
>>      tx28_tso_inner_bytes: 0
>>      tx28_csum_partial: 0
>>      tx28_csum_partial_inner: 0
>>      tx28_added_vlan_packets: 0
>>      tx28_nop: 0
>>      tx28_csum_none: 0
>>      tx28_stopped: 0
>>      tx28_dropped: 0
>>      tx28_xmit_more: 0
>>      tx28_recover: 0
>>      tx28_cqes: 0
>>      tx28_wake: 0
>>      tx28_cqe_err: 0
>>      tx29_packets: 3
>>      tx29_bytes: 266
>>      tx29_tso_packets: 0
>>      tx29_tso_bytes: 0
>>      tx29_tso_inner_packets: 0
>>      tx29_tso_inner_bytes: 0
>>      tx29_csum_partial: 0
>>      tx29_csum_partial_inner: 0
>>      tx29_added_vlan_packets: 0
>>      tx29_nop: 0
>>      tx29_csum_none: 3
>>      tx29_stopped: 0
>>      tx29_dropped: 0
>>      tx29_xmit_more: 1
>>      tx29_recover: 0
>>      tx29_cqes: 2
>>      tx29_wake: 0
>>      tx29_cqe_err: 0
>>      tx30_packets: 0
>>      tx30_bytes: 0
>>      tx30_tso_packets: 0
>>      tx30_tso_bytes: 0
>>      tx30_tso_inner_packets: 0
>>      tx30_tso_inner_bytes: 0
>>      tx30_csum_partial: 0
>>      tx30_csum_partial_inner: 0
>>      tx30_added_vlan_packets: 0
>>      tx30_nop: 0
>>      tx30_csum_none: 0
>>      tx30_stopped: 0
>>      tx30_dropped: 0
>>      tx30_xmit_more: 0
>>      tx30_recover: 0
>>      tx30_cqes: 0
>>      tx30_wake: 0
>>      tx30_cqe_err: 0
>>      tx31_packets: 0
>>      tx31_bytes: 0
>>      tx31_tso_packets: 0
>>      tx31_tso_bytes: 0
>>      tx31_tso_inner_packets: 0
>>      tx31_tso_inner_bytes: 0
>>      tx31_csum_partial: 0
>>      tx31_csum_partial_inner: 0
>>      tx31_added_vlan_packets: 0
>>      tx31_nop: 0
>>      tx31_csum_none: 0
>>      tx31_stopped: 0
>>      tx31_dropped: 0
>>      tx31_xmit_more: 0
>>      tx31_recover: 0
>>      tx31_cqes: 0
>>      tx31_wake: 0
>>      tx31_cqe_err: 0
>>      tx32_packets: 0
>>      tx32_bytes: 0
>>      tx32_tso_packets: 0
>>      tx32_tso_bytes: 0
>>      tx32_tso_inner_packets: 0
>>      tx32_tso_inner_bytes: 0
>>      tx32_csum_partial: 0
>>      tx32_csum_partial_inner: 0
>>      tx32_added_vlan_packets: 0
>>      tx32_nop: 0
>>      tx32_csum_none: 0
>>      tx32_stopped: 0
>>      tx32_dropped: 0
>>      tx32_xmit_more: 0
>>      tx32_recover: 0
>>      tx32_cqes: 0
>>      tx32_wake: 0
>>      tx32_cqe_err: 0
>>      tx33_packets: 0
>>      tx33_bytes: 0
>>      tx33_tso_packets: 0
>>      tx33_tso_bytes: 0
>>      tx33_tso_inner_packets: 0
>>      tx33_tso_inner_bytes: 0
>>      tx33_csum_partial: 0
>>      tx33_csum_partial_inner: 0
>>      tx33_added_vlan_packets: 0
>>      tx33_nop: 0
>>      tx33_csum_none: 0
>>      tx33_stopped: 0
>>      tx33_dropped: 0
>>      tx33_xmit_more: 0
>>      tx33_recover: 0
>>      tx33_cqes: 0
>>      tx33_wake: 0
>>      tx33_cqe_err: 0
>>      tx34_packets: 0
>>      tx34_bytes: 0
>>      tx34_tso_packets: 0
>>      tx34_tso_bytes: 0
>>      tx34_tso_inner_packets: 0
>>      tx34_tso_inner_bytes: 0
>>      tx34_csum_partial: 0
>>      tx34_csum_partial_inner: 0
>>      tx34_added_vlan_packets: 0
>>      tx34_nop: 0
>>      tx34_csum_none: 0
>>      tx34_stopped: 0
>>      tx34_dropped: 0
>>      tx34_xmit_more: 0
>>      tx34_recover: 0
>>      tx34_cqes: 0
>>      tx34_wake: 0
>>      tx34_cqe_err: 0
>>      tx35_packets: 0
>>      tx35_bytes: 0
>>      tx35_tso_packets: 0
>>      tx35_tso_bytes: 0
>>      tx35_tso_inner_packets: 0
>>      tx35_tso_inner_bytes: 0
>>      tx35_csum_partial: 0
>>      tx35_csum_partial_inner: 0
>>      tx35_added_vlan_packets: 0
>>      tx35_nop: 0
>>      tx35_csum_none: 0
>>      tx35_stopped: 0
>>      tx35_dropped: 0
>>      tx35_xmit_more: 0
>>      tx35_recover: 0
>>      tx35_cqes: 0
>>      tx35_wake: 0
>>      tx35_cqe_err: 0
>>      tx36_packets: 0
>>      tx36_bytes: 0
>>      tx36_tso_packets: 0
>>      tx36_tso_bytes: 0
>>      tx36_tso_inner_packets: 0
>>      tx36_tso_inner_bytes: 0
>>      tx36_csum_partial: 0
>>      tx36_csum_partial_inner: 0
>>      tx36_added_vlan_packets: 0
>>      tx36_nop: 0
>>      tx36_csum_none: 0
>>      tx36_stopped: 0
>>      tx36_dropped: 0
>>      tx36_xmit_more: 0
>>      tx36_recover: 0
>>      tx36_cqes: 0
>>      tx36_wake: 0
>>      tx36_cqe_err: 0
>>      tx37_packets: 0
>>      tx37_bytes: 0
>>      tx37_tso_packets: 0
>>      tx37_tso_bytes: 0
>>      tx37_tso_inner_packets: 0
>>      tx37_tso_inner_bytes: 0
>>      tx37_csum_partial: 0
>>      tx37_csum_partial_inner: 0
>>      tx37_added_vlan_packets: 0
>>      tx37_nop: 0
>>      tx37_csum_none: 0
>>      tx37_stopped: 0
>>      tx37_dropped: 0
>>      tx37_xmit_more: 0
>>      tx37_recover: 0
>>      tx37_cqes: 0
>>      tx37_wake: 0
>>      tx37_cqe_err: 0
>>      tx38_packets: 0
>>      tx38_bytes: 0
>>      tx38_tso_packets: 0
>>      tx38_tso_bytes: 0
>>      tx38_tso_inner_packets: 0
>>      tx38_tso_inner_bytes: 0
>>      tx38_csum_partial: 0
>>      tx38_csum_partial_inner: 0
>>      tx38_added_vlan_packets: 0
>>      tx38_nop: 0
>>      tx38_csum_none: 0
>>      tx38_stopped: 0
>>      tx38_dropped: 0
>>      tx38_xmit_more: 0
>>      tx38_recover: 0
>>      tx38_cqes: 0
>>      tx38_wake: 0
>>      tx38_cqe_err: 0
>>      tx39_packets: 0
>>      tx39_bytes: 0
>>      tx39_tso_packets: 0
>>      tx39_tso_bytes: 0
>>      tx39_tso_inner_packets: 0
>>      tx39_tso_inner_bytes: 0
>>      tx39_csum_partial: 0
>>      tx39_csum_partial_inner: 0
>>      tx39_added_vlan_packets: 0
>>      tx39_nop: 0
>>      tx39_csum_none: 0
>>      tx39_stopped: 0
>>      tx39_dropped: 0
>>      tx39_xmit_more: 0
>>      tx39_recover: 0
>>      tx39_cqes: 0
>>      tx39_wake: 0
>>      tx39_cqe_err: 0
>>      tx40_packets: 0
>>      tx40_bytes: 0
>>      tx40_tso_packets: 0
>>      tx40_tso_bytes: 0
>>      tx40_tso_inner_packets: 0
>>      tx40_tso_inner_bytes: 0
>>      tx40_csum_partial: 0
>>      tx40_csum_partial_inner: 0
>>      tx40_added_vlan_packets: 0
>>      tx40_nop: 0
>>      tx40_csum_none: 0
>>      tx40_stopped: 0
>>      tx40_dropped: 0
>>      tx40_xmit_more: 0
>>      tx40_recover: 0
>>      tx40_cqes: 0
>>      tx40_wake: 0
>>      tx40_cqe_err: 0
>>      tx41_packets: 0
>>      tx41_bytes: 0
>>      tx41_tso_packets: 0
>>      tx41_tso_bytes: 0
>>      tx41_tso_inner_packets: 0
>>      tx41_tso_inner_bytes: 0
>>      tx41_csum_partial: 0
>>      tx41_csum_partial_inner: 0
>>      tx41_added_vlan_packets: 0
>>      tx41_nop: 0
>>      tx41_csum_none: 0
>>      tx41_stopped: 0
>>      tx41_dropped: 0
>>      tx41_xmit_more: 0
>>      tx41_recover: 0
>>      tx41_cqes: 0
>>      tx41_wake: 0
>>      tx41_cqe_err: 0
>>      tx42_packets: 0
>>      tx42_bytes: 0
>>      tx42_tso_packets: 0
>>      tx42_tso_bytes: 0
>>      tx42_tso_inner_packets: 0
>>      tx42_tso_inner_bytes: 0
>>      tx42_csum_partial: 0
>>      tx42_csum_partial_inner: 0
>>      tx42_added_vlan_packets: 0
>>      tx42_nop: 0
>>      tx42_csum_none: 0
>>      tx42_stopped: 0
>>      tx42_dropped: 0
>>      tx42_xmit_more: 0
>>      tx42_recover: 0
>>      tx42_cqes: 0
>>      tx42_wake: 0
>>      tx42_cqe_err: 0
>>      tx43_packets: 0
>>      tx43_bytes: 0
>>      tx43_tso_packets: 0
>>      tx43_tso_bytes: 0
>>      tx43_tso_inner_packets: 0
>>      tx43_tso_inner_bytes: 0
>>      tx43_csum_partial: 0
>>      tx43_csum_partial_inner: 0
>>      tx43_added_vlan_packets: 0
>>      tx43_nop: 0
>>      tx43_csum_none: 0
>>      tx43_stopped: 0
>>      tx43_dropped: 0
>>      tx43_xmit_more: 0
>>      tx43_recover: 0
>>      tx43_cqes: 0
>>      tx43_wake: 0
>>      tx43_cqe_err: 0
>>      tx44_packets: 0
>>      tx44_bytes: 0
>>      tx44_tso_packets: 0
>>      tx44_tso_bytes: 0
>>      tx44_tso_inner_packets: 0
>>      tx44_tso_inner_bytes: 0
>>      tx44_csum_partial: 0
>>      tx44_csum_partial_inner: 0
>>      tx44_added_vlan_packets: 0
>>      tx44_nop: 0
>>      tx44_csum_none: 0
>>      tx44_stopped: 0
>>      tx44_dropped: 0
>>      tx44_xmit_more: 0
>>      tx44_recover: 0
>>      tx44_cqes: 0
>>      tx44_wake: 0
>>      tx44_cqe_err: 0
>>      tx45_packets: 0
>>      tx45_bytes: 0
>>      tx45_tso_packets: 0
>>      tx45_tso_bytes: 0
>>      tx45_tso_inner_packets: 0
>>      tx45_tso_inner_bytes: 0
>>      tx45_csum_partial: 0
>>      tx45_csum_partial_inner: 0
>>      tx45_added_vlan_packets: 0
>>      tx45_nop: 0
>>      tx45_csum_none: 0
>>      tx45_stopped: 0
>>      tx45_dropped: 0
>>      tx45_xmit_more: 0
>>      tx45_recover: 0
>>      tx45_cqes: 0
>>      tx45_wake: 0
>>      tx45_cqe_err: 0
>>      tx46_packets: 0
>>      tx46_bytes: 0
>>      tx46_tso_packets: 0
>>      tx46_tso_bytes: 0
>>      tx46_tso_inner_packets: 0
>>      tx46_tso_inner_bytes: 0
>>      tx46_csum_partial: 0
>>      tx46_csum_partial_inner: 0
>>      tx46_added_vlan_packets: 0
>>      tx46_nop: 0
>>      tx46_csum_none: 0
>>      tx46_stopped: 0
>>      tx46_dropped: 0
>>      tx46_xmit_more: 0
>>      tx46_recover: 0
>>      tx46_cqes: 0
>>      tx46_wake: 0
>>      tx46_cqe_err: 0
>>      tx47_packets: 0
>>      tx47_bytes: 0
>>      tx47_tso_packets: 0
>>      tx47_tso_bytes: 0
>>      tx47_tso_inner_packets: 0
>>      tx47_tso_inner_bytes: 0
>>      tx47_csum_partial: 0
>>      tx47_csum_partial_inner: 0
>>      tx47_added_vlan_packets: 0
>>      tx47_nop: 0
>>      tx47_csum_none: 0
>>      tx47_stopped: 0
>>      tx47_dropped: 0
>>      tx47_xmit_more: 0
>>      tx47_recover: 0
>>      tx47_cqes: 0
>>      tx47_wake: 0
>>      tx47_cqe_err: 0
>>      tx48_packets: 0
>>      tx48_bytes: 0
>>      tx48_tso_packets: 0
>>      tx48_tso_bytes: 0
>>      tx48_tso_inner_packets: 0
>>      tx48_tso_inner_bytes: 0
>>      tx48_csum_partial: 0
>>      tx48_csum_partial_inner: 0
>>      tx48_added_vlan_packets: 0
>>      tx48_nop: 0
>>      tx48_csum_none: 0
>>      tx48_stopped: 0
>>      tx48_dropped: 0
>>      tx48_xmit_more: 0
>>      tx48_recover: 0
>>      tx48_cqes: 0
>>      tx48_wake: 0
>>      tx48_cqe_err: 0
>>      tx49_packets: 0
>>      tx49_bytes: 0
>>      tx49_tso_packets: 0
>>      tx49_tso_bytes: 0
>>      tx49_tso_inner_packets: 0
>>      tx49_tso_inner_bytes: 0
>>      tx49_csum_partial: 0
>>      tx49_csum_partial_inner: 0
>>      tx49_added_vlan_packets: 0
>>      tx49_nop: 0
>>      tx49_csum_none: 0
>>      tx49_stopped: 0
>>      tx49_dropped: 0
>>      tx49_xmit_more: 0
>>      tx49_recover: 0
>>      tx49_cqes: 0
>>      tx49_wake: 0
>>      tx49_cqe_err: 0
>>      tx50_packets: 0
>>      tx50_bytes: 0
>>      tx50_tso_packets: 0
>>      tx50_tso_bytes: 0
>>      tx50_tso_inner_packets: 0
>>      tx50_tso_inner_bytes: 0
>>      tx50_csum_partial: 0
>>      tx50_csum_partial_inner: 0
>>      tx50_added_vlan_packets: 0
>>      tx50_nop: 0
>>      tx50_csum_none: 0
>>      tx50_stopped: 0
>>      tx50_dropped: 0
>>      tx50_xmit_more: 0
>>      tx50_recover: 0
>>      tx50_cqes: 0
>>      tx50_wake: 0
>>      tx50_cqe_err: 0
>>      tx51_packets: 0
>>      tx51_bytes: 0
>>      tx51_tso_packets: 0
>>      tx51_tso_bytes: 0
>>      tx51_tso_inner_packets: 0
>>      tx51_tso_inner_bytes: 0
>>      tx51_csum_partial: 0
>>      tx51_csum_partial_inner: 0
>>      tx51_added_vlan_packets: 0
>>      tx51_nop: 0
>>      tx51_csum_none: 0
>>      tx51_stopped: 0
>>      tx51_dropped: 0
>>      tx51_xmit_more: 0
>>      tx51_recover: 0
>>      tx51_cqes: 0
>>      tx51_wake: 0
>>      tx51_cqe_err: 0
>>      tx52_packets: 0
>>      tx52_bytes: 0
>>      tx52_tso_packets: 0
>>      tx52_tso_bytes: 0
>>      tx52_tso_inner_packets: 0
>>      tx52_tso_inner_bytes: 0
>>      tx52_csum_partial: 0
>>      tx52_csum_partial_inner: 0
>>      tx52_added_vlan_packets: 0
>>      tx52_nop: 0
>>      tx52_csum_none: 0
>>      tx52_stopped: 0
>>      tx52_dropped: 0
>>      tx52_xmit_more: 0
>>      tx52_recover: 0
>>      tx52_cqes: 0
>>      tx52_wake: 0
>>      tx52_cqe_err: 0
>>      tx53_packets: 0
>>      tx53_bytes: 0
>>      tx53_tso_packets: 0
>>      tx53_tso_bytes: 0
>>      tx53_tso_inner_packets: 0
>>      tx53_tso_inner_bytes: 0
>>      tx53_csum_partial: 0
>>      tx53_csum_partial_inner: 0
>>      tx53_added_vlan_packets: 0
>>      tx53_nop: 0
>>      tx53_csum_none: 0
>>      tx53_stopped: 0
>>      tx53_dropped: 0
>>      tx53_xmit_more: 0
>>      tx53_recover: 0
>>      tx53_cqes: 0
>>      tx53_wake: 0
>>      tx53_cqe_err: 0
>>      tx54_packets: 0
>>      tx54_bytes: 0
>>      tx54_tso_packets: 0
>>      tx54_tso_bytes: 0
>>      tx54_tso_inner_packets: 0
>>      tx54_tso_inner_bytes: 0
>>      tx54_csum_partial: 0
>>      tx54_csum_partial_inner: 0
>>      tx54_added_vlan_packets: 0
>>      tx54_nop: 0
>>      tx54_csum_none: 0
>>      tx54_stopped: 0
>>      tx54_dropped: 0
>>      tx54_xmit_more: 0
>>      tx54_recover: 0
>>      tx54_cqes: 0
>>      tx54_wake: 0
>>      tx54_cqe_err: 0
>>      tx55_packets: 0
>>      tx55_bytes: 0
>>      tx55_tso_packets: 0
>>      tx55_tso_bytes: 0
>>      tx55_tso_inner_packets: 0
>>      tx55_tso_inner_bytes: 0
>>      tx55_csum_partial: 0
>>      tx55_csum_partial_inner: 0
>>      tx55_added_vlan_packets: 0
>>      tx55_nop: 0
>>      tx55_csum_none: 0
>>      tx55_stopped: 0
>>      tx55_dropped: 0
>>      tx55_xmit_more: 0
>>      tx55_recover: 0
>>      tx55_cqes: 0
>>      tx55_wake: 0
>>      tx55_cqe_err: 0
>>      tx0_xdp_xmit: 0
>>      tx0_xdp_full: 0
>>      tx0_xdp_err: 0
>>      tx0_xdp_cqes: 0
>>      tx1_xdp_xmit: 0
>>      tx1_xdp_full: 0
>>      tx1_xdp_err: 0
>>      tx1_xdp_cqes: 0
>>      tx2_xdp_xmit: 0
>>      tx2_xdp_full: 0
>>      tx2_xdp_err: 0
>>      tx2_xdp_cqes: 0
>>      tx3_xdp_xmit: 0
>>      tx3_xdp_full: 0
>>      tx3_xdp_err: 0
>>      tx3_xdp_cqes: 0
>>      tx4_xdp_xmit: 0
>>      tx4_xdp_full: 0
>>      tx4_xdp_err: 0
>>      tx4_xdp_cqes: 0
>>      tx5_xdp_xmit: 0
>>      tx5_xdp_full: 0
>>      tx5_xdp_err: 0
>>      tx5_xdp_cqes: 0
>>      tx6_xdp_xmit: 0
>>      tx6_xdp_full: 0
>>      tx6_xdp_err: 0
>>      tx6_xdp_cqes: 0
>>      tx7_xdp_xmit: 0
>>      tx7_xdp_full: 0
>>      tx7_xdp_err: 0
>>      tx7_xdp_cqes: 0
>>      tx8_xdp_xmit: 0
>>      tx8_xdp_full: 0
>>      tx8_xdp_err: 0
>>      tx8_xdp_cqes: 0
>>      tx9_xdp_xmit: 0
>>      tx9_xdp_full: 0
>>      tx9_xdp_err: 0
>>      tx9_xdp_cqes: 0
>>      tx10_xdp_xmit: 0
>>      tx10_xdp_full: 0
>>      tx10_xdp_err: 0
>>      tx10_xdp_cqes: 0
>>      tx11_xdp_xmit: 0
>>      tx11_xdp_full: 0
>>      tx11_xdp_err: 0
>>      tx11_xdp_cqes: 0
>>      tx12_xdp_xmit: 0
>>      tx12_xdp_full: 0
>>      tx12_xdp_err: 0
>>      tx12_xdp_cqes: 0
>>      tx13_xdp_xmit: 0
>>      tx13_xdp_full: 0
>>      tx13_xdp_err: 0
>>      tx13_xdp_cqes: 0
>>      tx14_xdp_xmit: 0
>>      tx14_xdp_full: 0
>>      tx14_xdp_err: 0
>>      tx14_xdp_cqes: 0
>>      tx15_xdp_xmit: 0
>>      tx15_xdp_full: 0
>>      tx15_xdp_err: 0
>>      tx15_xdp_cqes: 0
>>      tx16_xdp_xmit: 0
>>      tx16_xdp_full: 0
>>      tx16_xdp_err: 0
>>      tx16_xdp_cqes: 0
>>      tx17_xdp_xmit: 0
>>      tx17_xdp_full: 0
>>      tx17_xdp_err: 0
>>      tx17_xdp_cqes: 0
>>      tx18_xdp_xmit: 0
>>      tx18_xdp_full: 0
>>      tx18_xdp_err: 0
>>      tx18_xdp_cqes: 0
>>      tx19_xdp_xmit: 0
>>      tx19_xdp_full: 0
>>      tx19_xdp_err: 0
>>      tx19_xdp_cqes: 0
>>      tx20_xdp_xmit: 0
>>      tx20_xdp_full: 0
>>      tx20_xdp_err: 0
>>      tx20_xdp_cqes: 0
>>      tx21_xdp_xmit: 0
>>      tx21_xdp_full: 0
>>      tx21_xdp_err: 0
>>      tx21_xdp_cqes: 0
>>      tx22_xdp_xmit: 0
>>      tx22_xdp_full: 0
>>      tx22_xdp_err: 0
>>      tx22_xdp_cqes: 0
>>      tx23_xdp_xmit: 0
>>      tx23_xdp_full: 0
>>      tx23_xdp_err: 0
>>      tx23_xdp_cqes: 0
>>      tx24_xdp_xmit: 0
>>      tx24_xdp_full: 0
>>      tx24_xdp_err: 0
>>      tx24_xdp_cqes: 0
>>      tx25_xdp_xmit: 0
>>      tx25_xdp_full: 0
>>      tx25_xdp_err: 0
>>      tx25_xdp_cqes: 0
>>      tx26_xdp_xmit: 0
>>      tx26_xdp_full: 0
>>      tx26_xdp_err: 0
>>      tx26_xdp_cqes: 0
>>      tx27_xdp_xmit: 0
>>      tx27_xdp_full: 0
>>      tx27_xdp_err: 0
>>      tx27_xdp_cqes: 0
>>      tx28_xdp_xmit: 0
>>      tx28_xdp_full: 0
>>      tx28_xdp_err: 0
>>      tx28_xdp_cqes: 0
>>      tx29_xdp_xmit: 0
>>      tx29_xdp_full: 0
>>      tx29_xdp_err: 0
>>      tx29_xdp_cqes: 0
>>      tx30_xdp_xmit: 0
>>      tx30_xdp_full: 0
>>      tx30_xdp_err: 0
>>      tx30_xdp_cqes: 0
>>      tx31_xdp_xmit: 0
>>      tx31_xdp_full: 0
>>      tx31_xdp_err: 0
>>      tx31_xdp_cqes: 0
>>      tx32_xdp_xmit: 0
>>      tx32_xdp_full: 0
>>      tx32_xdp_err: 0
>>      tx32_xdp_cqes: 0
>>      tx33_xdp_xmit: 0
>>      tx33_xdp_full: 0
>>      tx33_xdp_err: 0
>>      tx33_xdp_cqes: 0
>>      tx34_xdp_xmit: 0
>>      tx34_xdp_full: 0
>>      tx34_xdp_err: 0
>>      tx34_xdp_cqes: 0
>>      tx35_xdp_xmit: 0
>>      tx35_xdp_full: 0
>>      tx35_xdp_err: 0
>>      tx35_xdp_cqes: 0
>>      tx36_xdp_xmit: 0
>>      tx36_xdp_full: 0
>>      tx36_xdp_err: 0
>>      tx36_xdp_cqes: 0
>>      tx37_xdp_xmit: 0
>>      tx37_xdp_full: 0
>>      tx37_xdp_err: 0
>>      tx37_xdp_cqes: 0
>>      tx38_xdp_xmit: 0
>>      tx38_xdp_full: 0
>>      tx38_xdp_err: 0
>>      tx38_xdp_cqes: 0
>>      tx39_xdp_xmit: 0
>>      tx39_xdp_full: 0
>>      tx39_xdp_err: 0
>>      tx39_xdp_cqes: 0
>>      tx40_xdp_xmit: 0
>>      tx40_xdp_full: 0
>>      tx40_xdp_err: 0
>>      tx40_xdp_cqes: 0
>>      tx41_xdp_xmit: 0
>>      tx41_xdp_full: 0
>>      tx41_xdp_err: 0
>>      tx41_xdp_cqes: 0
>>      tx42_xdp_xmit: 0
>>      tx42_xdp_full: 0
>>      tx42_xdp_err: 0
>>      tx42_xdp_cqes: 0
>>      tx43_xdp_xmit: 0
>>      tx43_xdp_full: 0
>>      tx43_xdp_err: 0
>>      tx43_xdp_cqes: 0
>>      tx44_xdp_xmit: 0
>>      tx44_xdp_full: 0
>>      tx44_xdp_err: 0
>>      tx44_xdp_cqes: 0
>>      tx45_xdp_xmit: 0
>>      tx45_xdp_full: 0
>>      tx45_xdp_err: 0
>>      tx45_xdp_cqes: 0
>>      tx46_xdp_xmit: 0
>>      tx46_xdp_full: 0
>>      tx46_xdp_err: 0
>>      tx46_xdp_cqes: 0
>>      tx47_xdp_xmit: 0
>>      tx47_xdp_full: 0
>>      tx47_xdp_err: 0
>>      tx47_xdp_cqes: 0
>>      tx48_xdp_xmit: 0
>>      tx48_xdp_full: 0
>>      tx48_xdp_err: 0
>>      tx48_xdp_cqes: 0
>>      tx49_xdp_xmit: 0
>>      tx49_xdp_full: 0
>>      tx49_xdp_err: 0
>>      tx49_xdp_cqes: 0
>>      tx50_xdp_xmit: 0
>>      tx50_xdp_full: 0
>>      tx50_xdp_err: 0
>>      tx50_xdp_cqes: 0
>>      tx51_xdp_xmit: 0
>>      tx51_xdp_full: 0
>>      tx51_xdp_err: 0
>>      tx51_xdp_cqes: 0
>>      tx52_xdp_xmit: 0
>>      tx52_xdp_full: 0
>>      tx52_xdp_err: 0
>>      tx52_xdp_cqes: 0
>>      tx53_xdp_xmit: 0
>>      tx53_xdp_full: 0
>>      tx53_xdp_err: 0
>>      tx53_xdp_cqes: 0
>>      tx54_xdp_xmit: 0
>>      tx54_xdp_full: 0
>>      tx54_xdp_err: 0
>>      tx54_xdp_cqes: 0
>>      tx55_xdp_xmit: 0
>>      tx55_xdp_full: 0
>>      tx55_xdp_err: 0
>>      tx55_xdp_cqes: 0
>>
>>
>>> [...]
>>>
>>>>>> ethtool -S enp175s0f0
>>>>>> NIC statistics:
>>>>>>         rx_packets: 141574897253
>>>>>>         rx_bytes: 184445040406258
>>>>>>         tx_packets: 172569543894
>>>>>>         tx_bytes: 99486882076365
>>>>>>         tx_tso_packets: 9367664195
>>>>>>         tx_tso_bytes: 56435233992948
>>>>>>         tx_tso_inner_packets: 0
>>>>>>         tx_tso_inner_bytes: 0
>>>>>>         tx_added_vlan_packets: 141297671626
>>>>>>         tx_nop: 2102916272
>>>>>>         rx_lro_packets: 0
>>>>>>         rx_lro_bytes: 0
>>>>>>         rx_ecn_mark: 0
>>>>>>         rx_removed_vlan_packets: 141574897252
>>>>>>         rx_csum_unnecessary: 0
>>>>>>         rx_csum_none: 23135854
>>>>>>         rx_csum_complete: 141551761398
>>>>>>         rx_csum_unnecessary_inner: 0
>>>>>>         rx_xdp_drop: 0
>>>>>>         rx_xdp_redirect: 0
>>>>>>         rx_xdp_tx_xmit: 0
>>>>>>         rx_xdp_tx_full: 0
>>>>>>         rx_xdp_tx_err: 0
>>>>>>         rx_xdp_tx_cqe: 0
>>>>>>         tx_csum_none: 127934791664
>>>>> It is a good idea to look into this, tx is not requesting hw tx
>>>>> csumming for a lot of packets, maybe you are wasting a lot of cpu
>>>>> on
>>>>> calculating csum, or maybe this is just the rx csum complete..
>>>>>
>>>>>>         tx_csum_partial: 13362879974
>>>>>>         tx_csum_partial_inner: 0
>>>>>>         tx_queue_stopped: 232561
>>>>> TX queues are stalling, could be an indentation for the pcie
>>>>> bottelneck.
>>>>>
>>>>>>         tx_queue_dropped: 0
>>>>>>         tx_xmit_more: 1266021946
>>>>>>         tx_recover: 0
>>>>>>         tx_cqes: 140031716469
>>>>>>         tx_queue_wake: 232561
>>>>>>         tx_udp_seg_rem: 0
>>>>>>         tx_cqe_err: 0
>>>>>>         tx_xdp_xmit: 0
>>>>>>         tx_xdp_full: 0
>>>>>>         tx_xdp_err: 0
>>>>>>         tx_xdp_cqes: 0
>>>>>>         rx_wqe_err: 0
>>>>>>         rx_mpwqe_filler_cqes: 0
>>>>>>         rx_mpwqe_filler_strides: 0
>>>>>>         rx_buff_alloc_err: 0
>>>>>>         rx_cqe_compress_blks: 0
>>>>>>         rx_cqe_compress_pkts: 0
>>>>>>         rx_page_reuse: 0
>>>>>>         rx_cache_reuse: 16625975793
>>>>>>         rx_cache_full: 54161465914
>>>>>>         rx_cache_empty: 258048
>>>>>>         rx_cache_busy: 54161472735
>>>>>>         rx_cache_waive: 0
>>>>>>         rx_congst_umr: 0
>>>>>>         rx_arfs_err: 0
>>>>>>         ch_events: 40572621887
>>>>>>         ch_poll: 40885650979
>>>>>>         ch_arm: 40429276692
>>>>>>         ch_aff_change: 0
>>>>>>         ch_eq_rearm: 0
>>>>>>         rx_out_of_buffer: 2791690
>>>>>>         rx_if_down_packets: 74
>>>>>>         rx_vport_unicast_packets: 141843476308
>>>>>>         rx_vport_unicast_bytes: 185421265403318
>>>>>>         tx_vport_unicast_packets: 172569484005
>>>>>>         tx_vport_unicast_bytes: 100019940094298
>>>>>>         rx_vport_multicast_packets: 85122935
>>>>>>         rx_vport_multicast_bytes: 5761316431
>>>>>>         tx_vport_multicast_packets: 6452
>>>>>>         tx_vport_multicast_bytes: 643540
>>>>>>         rx_vport_broadcast_packets: 22423624
>>>>>>         rx_vport_broadcast_bytes: 1390127090
>>>>>>         tx_vport_broadcast_packets: 22024
>>>>>>         tx_vport_broadcast_bytes: 1321440
>>>>>>         rx_vport_rdma_unicast_packets: 0
>>>>>>         rx_vport_rdma_unicast_bytes: 0
>>>>>>         tx_vport_rdma_unicast_packets: 0
>>>>>>         tx_vport_rdma_unicast_bytes: 0
>>>>>>         rx_vport_rdma_multicast_packets: 0
>>>>>>         rx_vport_rdma_multicast_bytes: 0
>>>>>>         tx_vport_rdma_multicast_packets: 0
>>>>>>         tx_vport_rdma_multicast_bytes: 0
>>>>>>         tx_packets_phy: 172569501577
>>>>>>         rx_packets_phy: 142871314588
>>>>>>         rx_crc_errors_phy: 0
>>>>>>         tx_bytes_phy: 100710212814151
>>>>>>         rx_bytes_phy: 187209224289564
>>>>>>         tx_multicast_phy: 6452
>>>>>>         tx_broadcast_phy: 22024
>>>>>>         rx_multicast_phy: 85122933
>>>>>>         rx_broadcast_phy: 22423623
>>>>>>         rx_in_range_len_errors_phy: 2
>>>>>>         rx_out_of_range_len_phy: 0
>>>>>>         rx_oversize_pkts_phy: 0
>>>>>>         rx_symbol_err_phy: 0
>>>>>>         tx_mac_control_phy: 0
>>>>>>         rx_mac_control_phy: 0
>>>>>>         rx_unsupported_op_phy: 0
>>>>>>         rx_pause_ctrl_phy: 0
>>>>>>         tx_pause_ctrl_phy: 0
>>>>>>         rx_discards_phy: 920161423
>>>>> Ok, this port seem to be suffering more, RX is congested, maybe due
>>>>> to
>>>>> the pcie bottleneck.
>>>> Yes this side is receiving more traffic - second port is +10G more tx
>>>>
>>> [...]
>>>
>>>
>>>>>> Average:      17    0.00    0.00 16.60    0.00    0.00 52.10
>>>>>> 0.00    0.00    0.00   31.30
>>>>>> Average:      18    0.00    0.00   13.90    0.00    0.00 61.20
>>>>>> 0.00    0.00    0.00   24.90
>>>>>> Average:      19    0.00    0.00    9.99    0.00    0.00 70.33
>>>>>> 0.00    0.00    0.00   19.68
>>>>>> Average:      20    0.00    0.00    9.00    0.00    0.00 73.00
>>>>>> 0.00    0.00    0.00   18.00
>>>>>> Average:      21    0.00    0.00    8.70    0.00    0.00 73.90
>>>>>> 0.00    0.00    0.00   17.40
>>>>>> Average:      22    0.00    0.00   15.42    0.00    0.00 58.56
>>>>>> 0.00    0.00    0.00   26.03
>>>>>> Average:      23    0.00    0.00   10.81    0.00    0.00 71.67
>>>>>> 0.00    0.00    0.00   17.52
>>>>>> Average:      24    0.00    0.00   10.00    0.00    0.00 71.80
>>>>>> 0.00    0.00    0.00   18.20
>>>>>> Average:      25    0.00    0.00   11.19    0.00    0.00 71.13
>>>>>> 0.00    0.00    0.00   17.68
>>>>>> Average:      26    0.00    0.00   11.00    0.00    0.00 70.80
>>>>>> 0.00    0.00    0.00   18.20
>>>>>> Average:      27    0.00    0.00   10.01    0.00    0.00 69.57
>>>>>> 0.00    0.00    0.00   20.42
>>>>> The numa cores are not at 100% util, you have around 20% of idle on
>>>>> each one.
>>>> Yes - no 100% cpu - but the difference between 80% and 100% is like
>>>> push
>>>> aditional 1-2Gbit/s
>>>>
>>> yes but, it doens't look like the bottleneck is the cpu, although it is
>>> close to be :)..
>>>
>>>>>> Average:      28    0.00    0.00 0.00    0.00    0.00 0.00
>>>>>> 0.00
>>>>>> 0.00    0.00  100.00
>>>>>> Average:      29    0.00    0.00    0.00    0.00    0.00 0.00
>>>>>> 0.00
>>>>>> 0.00    0.00  100.00
>>>>>> Average:      30    0.00    0.00    0.00    0.00    0.00 0.00
>>>>>> 0.00
>>>>>> 0.00    0.00  100.00
>>>>>> Average:      31    0.00    0.00    0.00    0.00    0.00 0.00
>>>>>> 0.00
>>>>>> 0.00    0.00  100.00
>>>>>> Average:      32    0.00    0.00    0.00    0.00    0.00 0.00
>>>>>> 0.00
>>>>>> 0.00    0.00  100.00
>>>>>> Average:      33    0.00    0.00    3.90    0.00    0.00 0.00
>>>>>> 0.00
>>>>>> 0.00    0.00   96.10
>>>>>> Average:      34    0.00    0.00    0.00    0.00    0.00 0.00
>>>>>> 0.00
>>>>>> 0.00    0.00  100.00
>>>>>> Average:      35    0.00    0.00    0.00    0.00    0.00 0.00
>>>>>> 0.00
>>>>>> 0.00    0.00  100.00
>>>>>> Average:      36    0.10    0.00    0.20    0.00    0.00 0.00
>>>>>> 0.00
>>>>>> 0.00    0.00   99.70
>>>>>> Average:      37    0.20    0.00    0.30    0.00    0.00 0.00
>>>>>> 0.00
>>>>>> 0.00    0.00   99.50
>>>>>> Average:      38    0.00    0.00    0.00    0.00    0.00 0.00
>>>>>> 0.00
>>>>>> 0.00    0.00  100.00
>>>>>> Average:      39    0.00    0.00    2.60    0.00    0.00 0.00
>>>>>> 0.00
>>>>>> 0.00    0.00   97.40
>>>>>> Average:      40    0.00    0.00    0.90    0.00    0.00 0.00
>>>>>> 0.00
>>>>>> 0.00    0.00   99.10
>>>>>> Average:      41    0.10    0.00    0.50    0.00    0.00 0.00
>>>>>> 0.00
>>>>>> 0.00    0.00   99.40
>>>>>> Average:      42    0.00    0.00    9.91    0.00    0.00 70.67
>>>>>> 0.00    0.00    0.00   19.42
>>>>>> Average:      43    0.00    0.00   15.90    0.00    0.00 57.50
>>>>>> 0.00    0.00    0.00   26.60
>>>>>> Average:      44    0.00    0.00   12.20    0.00    0.00 66.20
>>>>>> 0.00    0.00    0.00   21.60
>>>>>> Average:      45    0.00    0.00   12.00    0.00    0.00 67.50
>>>>>> 0.00    0.00    0.00   20.50
>>>>>> Average:      46    0.00    0.00   12.90    0.00    0.00 65.50
>>>>>> 0.00    0.00    0.00   21.60
>>>>>> Average:      47    0.00    0.00   14.59    0.00    0.00 60.84
>>>>>> 0.00    0.00    0.00   24.58
>>>>>> Average:      48    0.00    0.00   13.59    0.00    0.00 61.74
>>>>>> 0.00    0.00    0.00   24.68
>>>>>> Average:      49    0.00    0.00   18.36    0.00    0.00 53.29
>>>>>> 0.00    0.00    0.00   28.34
>>>>>> Average:      50    0.00    0.00   15.32    0.00    0.00 58.86
>>>>>> 0.00    0.00    0.00   25.83
>>>>>> Average:      51    0.00    0.00   17.60    0.00    0.00 55.20
>>>>>> 0.00    0.00    0.00   27.20
>>>>>> Average:      52    0.00    0.00   15.92    0.00    0.00 56.06
>>>>>> 0.00    0.00    0.00   28.03
>>>>>> Average:      53    0.00    0.00   13.00    0.00    0.00 62.30
>>>>>> 0.00    0.00    0.00   24.70
>>>>>> Average:      54    0.00    0.00   13.20    0.00    0.00 61.50
>>>>>> 0.00    0.00    0.00   25.30
>>>>>> Average:      55    0.00    0.00   14.59    0.00    0.00 58.64
>>>>>> 0.00    0.00    0.00   26.77
>>>>>>
>>>>>>
>>>>>> ethtool -k enp175s0f0
>>>>>> Features for enp175s0f0:
>>>>>> rx-checksumming: on
>>>>>> tx-checksumming: on
>>>>>>            tx-checksum-ipv4: on
>>>>>>            tx-checksum-ip-generic: off [fixed]
>>>>>>            tx-checksum-ipv6: on
>>>>>>            tx-checksum-fcoe-crc: off [fixed]
>>>>>>            tx-checksum-sctp: off [fixed]
>>>>>> scatter-gather: on
>>>>>>            tx-scatter-gather: on
>>>>>>            tx-scatter-gather-fraglist: off [fixed]
>>>>>> tcp-segmentation-offload: on
>>>>>>            tx-tcp-segmentation: on
>>>>>>            tx-tcp-ecn-segmentation: off [fixed]
>>>>>>            tx-tcp-mangleid-segmentation: off
>>>>>>            tx-tcp6-segmentation: on
>>>>>> udp-fragmentation-offload: off
>>>>>> generic-segmentation-offload: on
>>>>>> generic-receive-offload: on
>>>>>> large-receive-offload: off [fixed]
>>>>>> rx-vlan-offload: on
>>>>>> tx-vlan-offload: on
>>>>>> ntuple-filters: off
>>>>>> receive-hashing: on
>>>>>> highdma: on [fixed]
>>>>>> rx-vlan-filter: on
>>>>>> vlan-challenged: off [fixed]
>>>>>> tx-lockless: off [fixed]
>>>>>> netns-local: off [fixed]
>>>>>> tx-gso-robust: off [fixed]
>>>>>> tx-fcoe-segmentation: off [fixed]
>>>>>> tx-gre-segmentation: on
>>>>>> tx-gre-csum-segmentation: on
>>>>>> tx-ipxip4-segmentation: off [fixed]
>>>>>> tx-ipxip6-segmentation: off [fixed]
>>>>>> tx-udp_tnl-segmentation: on
>>>>>> tx-udp_tnl-csum-segmentation: on
>>>>>> tx-gso-partial: on
>>>>>> tx-sctp-segmentation: off [fixed]
>>>>>> tx-esp-segmentation: off [fixed]
>>>>>> tx-udp-segmentation: on
>>>>>> fcoe-mtu: off [fixed]
>>>>>> tx-nocache-copy: off
>>>>>> loopback: off [fixed]
>>>>>> rx-fcs: off
>>>>>> rx-all: off
>>>>>> tx-vlan-stag-hw-insert: on
>>>>>> rx-vlan-stag-hw-parse: off [fixed]
>>>>>> rx-vlan-stag-filter: on [fixed]
>>>>>> l2-fwd-offload: off [fixed]
>>>>>> hw-tc-offload: off
>>>>>> esp-hw-offload: off [fixed]
>>>>>> esp-tx-csum-hw-offload: off [fixed]
>>>>>> rx-udp_tunnel-port-offload: on
>>>>>> tls-hw-tx-offload: off [fixed]
>>>>>> tls-hw-rx-offload: off [fixed]
>>>>>> rx-gro-hw: off [fixed]
>>>>>> tls-hw-record: off [fixed]
>>>>>>
>>>>>> ethtool -c enp175s0f0
>>>>>> Coalesce parameters for enp175s0f0:
>>>>>> Adaptive RX: off  TX: on
>>>>>> stats-block-usecs: 0
>>>>>> sample-interval: 0
>>>>>> pkt-rate-low: 0
>>>>>> pkt-rate-high: 0
>>>>>> dmac: 32703
>>>>>>
>>>>>> rx-usecs: 256
>>>>>> rx-frames: 128
>>>>>> rx-usecs-irq: 0
>>>>>> rx-frames-irq: 0
>>>>>>
>>>>>> tx-usecs: 8
>>>>>> tx-frames: 128
>>>>>> tx-usecs-irq: 0
>>>>>> tx-frames-irq: 0
>>>>>>
>>>>>> rx-usecs-low: 0
>>>>>> rx-frame-low: 0
>>>>>> tx-usecs-low: 0
>>>>>> tx-frame-low: 0
>>>>>>
>>>>>> rx-usecs-high: 0
>>>>>> rx-frame-high: 0
>>>>>> tx-usecs-high: 0
>>>>>> tx-frame-high: 0
>>>>>>
>>>>>> ethtool -g enp175s0f0
>>>>>> Ring parameters for enp175s0f0:
>>>>>> Pre-set maximums:
>>>>>> RX:             8192
>>>>>> RX Mini:        0
>>>>>> RX Jumbo:       0
>>>>>> TX:             8192
>>>>>> Current hardware settings:
>>>>>> RX:             4096
>>>>>> RX Mini:        0
>>>>>> RX Jumbo:       0
>>>>>> TX:             4096
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>> Also changed a little coalesce params - and best for this config are:
>>>> ethtool -c enp175s0f0
>>>> Coalesce parameters for enp175s0f0:
>>>> Adaptive RX: off  TX: off
>>>> stats-block-usecs: 0
>>>> sample-interval: 0
>>>> pkt-rate-low: 0
>>>> pkt-rate-high: 0
>>>> dmac: 32573
>>>>
>>>> rx-usecs: 40
>>>> rx-frames: 128
>>>> rx-usecs-irq: 0
>>>> rx-frames-irq: 0
>>>>
>>>> tx-usecs: 8
>>>> tx-frames: 8
>>>> tx-usecs-irq: 0
>>>> tx-frames-irq: 0
>>>>
>>>> rx-usecs-low: 0
>>>> rx-frame-low: 0
>>>> tx-usecs-low: 0
>>>> tx-frame-low: 0
>>>>
>>>> rx-usecs-high: 0
>>>> rx-frame-high: 0
>>>> tx-usecs-high: 0
>>>> tx-frame-high: 0
>>>>
>>>>
>>>> Less drops on RX side - and more pps in overall forwarded.
>>>>
>>> how much improvement ? maybe we can improve our adaptive rx coal to be
>>> efficient for this work load.
>>>
>>>
>> I can prepare more stats with ethtool maybee to compare - but 
>> normally tested with simple icmp forwarded from interface to interface
>> - before change  coalescence params:
>> adaptive-rx off rx-usecs 384 rx-frames 128
>> 3% loss for icmp
>> - after change to:
>> adaptive-rx off rx-usecs 40 rx-frames 128 adaptive-tx off tx-usecs 8 
>> tx-frames 8
>> 2% loss for icmp
>>
>> But yes - to know better will need to compare rx/tx counters from 
>> ethtool + /proc/net/dev
>>
>>
>> Was trying to turn on adaptative-tx+rx - but 100% saturation at 
>> 43Gbit/s RX / 43Gbit/s TX
>>
>>
>>
>>
>>
>>
>>
>
>