| lists.openwall.net | lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC | |
|
Open Source and information security mailing list archives
| ||
|
Message-ID: <5c55fc87-4508-bb67-8f89-f95708e33e0f@itcare.pl>
Date: Thu, 1 Nov 2018 22:18:32 +0100
From: Paweł Staszewski <pstaszewski@...are.pl>
To: Saeed Mahameed <saeedm@...lanox.com>,
"netdev@...r.kernel.org" <netdev@...r.kernel.org>
Subject: Re: Kernel 4.19 network performance - forwarding/routing normal users
traffic
W dniu 01.11.2018 o 21:37, Saeed Mahameed pisze:
> On Thu, 2018-11-01 at 12:09 +0100, Paweł Staszewski wrote:
>> W dniu 01.11.2018 o 10:50, Saeed Mahameed pisze:
>>> On Wed, 2018-10-31 at 22:57 +0100, Paweł Staszewski wrote:
>>>> Hi
>>>>
>>>> So maybee someone will be interested how linux kernel handles
>>>> normal
>>>> traffic (not pktgen :) )
>>>>
>>>>
>>>> Server HW configuration:
>>>>
>>>> CPU : Intel(R) Xeon(R) Gold 6132 CPU @ 2.60GHz
>>>>
>>>> NIC's: 2x 100G Mellanox ConnectX-4 (connected to x16 pcie 8GT)
>>>>
>>>>
>>>> Server software:
>>>>
>>>> FRR - as routing daemon
>>>>
>>>> enp175s0f0 (100G) - 16 vlans from upstreams (28 RSS binded to
>>>> local
>>>> numa
>>>> node)
>>>>
>>>> enp175s0f1 (100G) - 343 vlans to clients (28 RSS binded to local
>>>> numa
>>>> node)
>>>>
>>>>
>>>> Maximum traffic that server can handle:
>>>>
>>>> Bandwidth
>>>>
>>>> bwm-ng v0.6.1 (probing every 1.000s), press 'h' for help
>>>> input: /proc/net/dev type: rate
>>>> \ iface Rx Tx Total
>>>> =================================================================
>>>> ====
>>>> =========
>>>> enp175s0f1: 28.51 Gb/s 37.24
>>>> Gb/s
>>>> 65.74 Gb/s
>>>> enp175s0f0: 38.07 Gb/s 28.44
>>>> Gb/s
>>>> 66.51 Gb/s
>>>> ---------------------------------------------------------------
>>>> ----
>>>> -----------
>>>> total: 66.58 Gb/s 65.67
>>>> Gb/s
>>>> 132.25 Gb/s
>>>>
>>>>
>>>> Packets per second:
>>>>
>>>> bwm-ng v0.6.1 (probing every 1.000s), press 'h' for help
>>>> input: /proc/net/dev type: rate
>>>> - iface Rx Tx Total
>>>> =================================================================
>>>> ====
>>>> =========
>>>> enp175s0f1: 5248589.00 P/s 3486617.75 P/s
>>>> 8735207.00 P/s
>>>> enp175s0f0: 3557944.25 P/s 5232516.00 P/s
>>>> 8790460.00 P/s
>>>> ---------------------------------------------------------------
>>>> ----
>>>> -----------
>>>> total: 8806533.00 P/s 8719134.00 P/s
>>>> 17525668.00 P/s
>>>>
>>>>
>>>> After reaching that limits nics on the upstream side (more RX
>>>> traffic)
>>>> start to drop packets
>>>>
>>>>
>>>> I just dont understand that server can't handle more bandwidth
>>>> (~40Gbit/s is limit where all cpu's are 100% util) - where pps on
>>>> RX
>>>> side are increasing.
>>>>
>>> Where do you see 40 Gb/s ? you showed that both ports on the same
>>> NIC (
>>> same pcie link) are doing 66.58 Gb/s (RX) + 65.67 Gb/s (TX) =
>>> 132.25
>>> Gb/s which aligns with your pcie link limit, what am i missing ?
>> hmm yes that was my concern also - cause cant find anywhere
>> informations
>> about that bandwidth is uni or bidirectional - so if 126Gbit for x16
>> 8GT
>> is unidir - then bidir will be 126/2 ~68Gbit - which will fit total
>> bw
>> on both ports
> i think it is bidir
>
>> This can explain maybee also why cpuload is rising rapidly from
>> 120Gbit/s in total to 132Gbit (counters of bwmng are from /proc/net -
>> so
>> there can be some error in reading them when offloading (gro/gso/tso)
>> on
>> nic's is enabled that is why
>>
>>>> Was thinking that maybee reached some pcie x16 limit - but x16
>>>> 8GT
>>>> is
>>>> 126Gbit - and also when testing with pktgen i can reach more bw
>>>> and
>>>> pps
>>>> (like 4x more comparing to normal internet traffic)
>>>>
>>> Are you forwarding when using pktgen as well or you just testing
>>> the RX
>>> side pps ?
>> Yes pktgen was tested on single port RX
>> Can check also forwarding to eliminate pciex limits
>>
> So this explains why you have more RX pps, since tx is idle and pcie
> will be free to do only rx.
>
> [...]
>
>
>>>> ethtool -S enp175s0f1
>>>> NIC statistics:
>>>> rx_packets: 173730800927
>>>> rx_bytes: 99827422751332
>>>> tx_packets: 142532009512
>>>> tx_bytes: 184633045911222
>>>> tx_tso_packets: 25989113891
>>>> tx_tso_bytes: 132933363384458
>>>> tx_tso_inner_packets: 0
>>>> tx_tso_inner_bytes: 0
>>>> tx_added_vlan_packets: 74630239613
>>>> tx_nop: 2029817748
>>>> rx_lro_packets: 0
>>>> rx_lro_bytes: 0
>>>> rx_ecn_mark: 0
>>>> rx_removed_vlan_packets: 173730800927
>>>> rx_csum_unnecessary: 0
>>>> rx_csum_none: 434357
>>>> rx_csum_complete: 173730366570
>>>> rx_csum_unnecessary_inner: 0
>>>> rx_xdp_drop: 0
>>>> rx_xdp_redirect: 0
>>>> rx_xdp_tx_xmit: 0
>>>> rx_xdp_tx_full: 0
>>>> rx_xdp_tx_err: 0
>>>> rx_xdp_tx_cqe: 0
>>>> tx_csum_none: 38260960853
>>>> tx_csum_partial: 36369278774
>>>> tx_csum_partial_inner: 0
>>>> tx_queue_stopped: 1
>>>> tx_queue_dropped: 0
>>>> tx_xmit_more: 748638099
>>>> tx_recover: 0
>>>> tx_cqes: 73881645031
>>>> tx_queue_wake: 1
>>>> tx_udp_seg_rem: 0
>>>> tx_cqe_err: 0
>>>> tx_xdp_xmit: 0
>>>> tx_xdp_full: 0
>>>> tx_xdp_err: 0
>>>> tx_xdp_cqes: 0
>>>> rx_wqe_err: 0
>>>> rx_mpwqe_filler_cqes: 0
>>>> rx_mpwqe_filler_strides: 0
>>>> rx_buff_alloc_err: 0
>>>> rx_cqe_compress_blks: 0
>>>> rx_cqe_compress_pkts: 0
>>> If this is a pcie bottleneck it might be useful to enable CQE
>>> compression (to reduce PCIe completion descriptors transactions)
>>> you should see the above rx_cqe_compress_pkts increasing when
>>> enabled.
>>>
>>> $ ethtool --set-priv-flags enp175s0f1 rx_cqe_compress on
>>> $ ethtool --show-priv-flags enp175s0f1
>>> Private flags for p6p1:
>>> rx_cqe_moder : on
>>> cqe_moder : off
>>> rx_cqe_compress : on
>>> ...
>>>
>>> try this on both interfaces.
>> Done
>> ethtool --show-priv-flags enp175s0f1
>> Private flags for enp175s0f1:
>> rx_cqe_moder : on
>> tx_cqe_moder : off
>> rx_cqe_compress : on
>> rx_striding_rq : off
>> rx_no_csum_complete: off
>>
>> ethtool --show-priv-flags enp175s0f0
>> Private flags for enp175s0f0:
>> rx_cqe_moder : on
>> tx_cqe_moder : off
>> rx_cqe_compress : on
>> rx_striding_rq : off
>> rx_no_csum_complete: off
>>
> did it help reduce the load on the pcie ? do you see more pps ?
> what is the ratio between rx_cqe_compress_pkts and over all rx packets
> ?
So - a little more pps
Before change top - graph / after bottom -> image with graph stats from
proc/net/dev
cqe_compress enabled at 11:55
Sorry - but for real life traffic it is hard to do any counter
differences - cause traffic just rising alone from minute to minute :)
But for that time the change is visible on graph - cause was almost same
for past 20minutes before change.
full ethtool below:
NIC statistics:
rx_packets: 516522465438
rx_bytes: 680052911258729
tx_packets: 677697545586
tx_bytes: 413647643141709
tx_tso_packets: 42530913279
tx_tso_bytes: 235655668554142
tx_tso_inner_packets: 0
tx_tso_inner_bytes: 0
tx_added_vlan_packets: 551156530885
tx_nop: 8536823558
rx_lro_packets: 0
rx_lro_bytes: 0
rx_ecn_mark: 0
rx_removed_vlan_packets: 516522465438
rx_csum_unnecessary: 0
rx_csum_none: 50382868
rx_csum_complete: 516472082570
rx_csum_unnecessary_inner: 0
rx_xdp_drop: 0
rx_xdp_redirect: 0
rx_xdp_tx_xmit: 0
rx_xdp_tx_full: 0
rx_xdp_tx_err: 0
rx_xdp_tx_cqe: 0
tx_csum_none: 494075047017
tx_csum_partial: 57081483898
tx_csum_partial_inner: 0
tx_queue_stopped: 518624
tx_queue_dropped: 0
tx_xmit_more: 1717880628
tx_recover: 0
tx_cqes: 549438869029
tx_queue_wake: 518627
tx_udp_seg_rem: 0
tx_cqe_err: 0
tx_xdp_xmit: 0
tx_xdp_full: 0
tx_xdp_err: 0
tx_xdp_cqes: 0
rx_wqe_err: 0
rx_mpwqe_filler_cqes: 0
rx_mpwqe_filler_strides: 0
rx_buff_alloc_err: 0
rx_cqe_compress_blks: 11483228712
rx_cqe_compress_pkts: 25794213324
rx_page_reuse: 0
rx_cache_reuse: 63610249810
rx_cache_full: 194650916511
rx_cache_empty: 1118208
rx_cache_busy: 194650982430
rx_cache_waive: 0
rx_congst_umr: 0
rx_arfs_err: 0
ch_events: 119556002196
ch_poll: 121107424977
ch_arm: 115856746008
ch_aff_change: 31
ch_eq_rearm: 0
rx_out_of_buffer: 6880325
rx_if_down_packets: 2062529
rx_vport_unicast_packets: 517433716795
rx_vport_unicast_bytes: 683464347301443
tx_vport_unicast_packets: 677697453738
tx_vport_unicast_bytes: 415788589663315
rx_vport_multicast_packets: 208258309
rx_vport_multicast_bytes: 14224046052
tx_vport_multicast_packets: 21689
tx_vport_multicast_bytes: 2158334
rx_vport_broadcast_packets: 75838646
rx_vport_broadcast_bytes: 4697944695
tx_vport_broadcast_packets: 68730
tx_vport_broadcast_bytes: 4123800
rx_vport_rdma_unicast_packets: 0
rx_vport_rdma_unicast_bytes: 0
tx_vport_rdma_unicast_packets: 0
tx_vport_rdma_unicast_bytes: 0
rx_vport_rdma_multicast_packets: 0
rx_vport_rdma_multicast_bytes: 0
tx_vport_rdma_multicast_packets: 0
tx_vport_rdma_multicast_bytes: 0
tx_packets_phy: 677697543252
rx_packets_phy: 521319491878
rx_crc_errors_phy: 0
tx_bytes_phy: 418499385791411
rx_bytes_phy: 690322537017274
tx_multicast_phy: 21689
tx_broadcast_phy: 68730
rx_multicast_phy: 208258305
rx_broadcast_phy: 75838646
rx_in_range_len_errors_phy: 4
rx_out_of_range_len_phy: 0
rx_oversize_pkts_phy: 0
rx_symbol_err_phy: 0
tx_mac_control_phy: 0
rx_mac_control_phy: 0
rx_unsupported_op_phy: 0
rx_pause_ctrl_phy: 0
tx_pause_ctrl_phy: 0
rx_discards_phy: 3601449265
tx_discards_phy: 0
tx_errors_phy: 0
rx_undersize_pkts_phy: 0
rx_fragments_phy: 0
rx_jabbers_phy: 0
rx_64_bytes_phy: 1416456771
rx_65_to_127_bytes_phy: 40750434737
rx_128_to_255_bytes_phy: 11518110310
rx_256_to_511_bytes_phy: 7055850637
rx_512_to_1023_bytes_phy: 7811550424
rx_1024_to_1518_bytes_phy: 265547564845
rx_1519_to_2047_bytes_phy: 187219522899
rx_2048_to_4095_bytes_phy: 0
rx_4096_to_8191_bytes_phy: 0
rx_8192_to_10239_bytes_phy: 0
link_down_events_phy: 0
rx_pcs_symbol_err_phy: 0
rx_corrected_bits_phy: 0
rx_pci_signal_integrity: 0
tx_pci_signal_integrity: 48
rx_prio0_bytes: 688807632117485
rx_prio0_packets: 516310309931
tx_prio0_bytes: 418499382756025
tx_prio0_packets: 677697534982
rx_prio1_bytes: 1497701612877
rx_prio1_packets: 1206768094
tx_prio1_bytes: 0
tx_prio1_packets: 0
rx_prio2_bytes: 112271227
rx_prio2_packets: 337295
tx_prio2_bytes: 0
tx_prio2_packets: 0
rx_prio3_bytes: 1165455555
rx_prio3_packets: 1544310
tx_prio3_bytes: 0
tx_prio3_packets: 0
rx_prio4_bytes: 161857240
rx_prio4_packets: 341392
tx_prio4_bytes: 0
tx_prio4_packets: 0
rx_prio5_bytes: 455031612
rx_prio5_packets: 2861469
tx_prio5_bytes: 0
tx_prio5_packets: 0
rx_prio6_bytes: 1873928697
rx_prio6_packets: 5146981
tx_prio6_bytes: 0
tx_prio6_packets: 0
rx_prio7_bytes: 13423452430
rx_prio7_packets: 190724796
tx_prio7_bytes: 0
tx_prio7_packets: 0
module_unplug: 0
module_bus_stuck: 0
module_high_temp: 0
module_bad_shorted: 0
ch0_events: 4252266777
ch0_poll: 4330804273
ch0_arm: 4120233182
ch0_aff_change: 2
ch0_eq_rearm: 0
ch1_events: 3938415938
ch1_poll: 4012621322
ch1_arm: 3810131188
ch1_aff_change: 2
ch1_eq_rearm: 0
ch2_events: 3897428860
ch2_poll: 3973886848
ch2_arm: 3773019397
ch2_aff_change: 1
ch2_eq_rearm: 0
ch3_events: 4108000541
ch3_poll: 4180139872
ch3_arm: 3982093366
ch3_aff_change: 1
ch3_eq_rearm: 0
ch4_events: 4652570079
ch4_poll: 4720541090
ch4_arm: 4524475054
ch4_aff_change: 2
ch4_eq_rearm: 0
ch5_events: 3899177385
ch5_poll: 3974274186
ch5_arm: 3772299186
ch5_aff_change: 2
ch5_eq_rearm: 0
ch6_events: 3915161350
ch6_poll: 3992338199
ch6_arm: 3794710989
ch6_aff_change: 0
ch6_eq_rearm: 0
ch7_events: 4008175631
ch7_poll: 4081321248
ch7_arm: 3882263723
ch7_aff_change: 0
ch7_eq_rearm: 0
ch8_events: 4207422352
ch8_poll: 4276465449
ch8_arm: 4077650366
ch8_aff_change: 0
ch8_eq_rearm: 0
ch9_events: 4036491879
ch9_poll: 4108975987
ch9_arm: 3914493694
ch9_aff_change: 0
ch9_eq_rearm: 0
ch10_events: 4066261595
ch10_poll: 4134419606
ch10_arm: 3936637711
ch10_aff_change: 1
ch10_eq_rearm: 0
ch11_events: 4440494043
ch11_poll: 4507578730
ch11_arm: 4318629438
ch11_aff_change: 0
ch11_eq_rearm: 0
ch12_events: 4066958252
ch12_poll: 4130191506
ch12_arm: 3934337782
ch12_aff_change: 0
ch12_eq_rearm: 0
ch13_events: 4051309159
ch13_poll: 4118864120
ch13_arm: 3921011919
ch13_aff_change: 0
ch13_eq_rearm: 0
ch14_events: 4321664800
ch14_poll: 4382433680
ch14_arm: 4186130552
ch14_aff_change: 0
ch14_eq_rearm: 0
ch15_events: 4701102075
ch15_poll: 4760373932
ch15_arm: 4570151468
ch15_aff_change: 0
ch15_eq_rearm: 0
ch16_events: 4311052687
ch16_poll: 4345937129
ch16_arm: 4170883819
ch16_aff_change: 0
ch16_eq_rearm: 0
ch17_events: 4647570931
ch17_poll: 4680218533
ch17_arm: 4509426288
ch17_aff_change: 0
ch17_eq_rearm: 0
ch18_events: 4598195702
ch18_poll: 4631314898
ch18_arm: 4457267084
ch18_aff_change: 0
ch18_eq_rearm: 0
ch19_events: 4808094560
ch19_poll: 4841368340
ch19_arm: 4670604358
ch19_aff_change: 0
ch19_eq_rearm: 0
ch20_events: 4240910605
ch20_poll: 4276531502
ch20_arm: 4101767278
ch20_aff_change: 1
ch20_eq_rearm: 0
ch21_events: 4389371472
ch21_poll: 4426870311
ch21_arm: 4249339045
ch21_aff_change: 2
ch21_eq_rearm: 0
ch22_events: 4282958754
ch22_poll: 4319228073
ch22_arm: 4145102991
ch22_aff_change: 2
ch22_eq_rearm: 0
ch23_events: 4440196528
ch23_poll: 4474090188
ch23_arm: 4300837147
ch23_aff_change: 2
ch23_eq_rearm: 0
ch24_events: 4326875785
ch24_poll: 4364971263
ch24_arm: 4186404526
ch24_aff_change: 2
ch24_eq_rearm: 0
ch25_events: 4286528453
ch25_poll: 4324089445
ch25_arm: 4147222616
ch25_aff_change: 3
ch25_eq_rearm: 0
ch26_events: 4098043104
ch26_poll: 4138133745
ch26_arm: 3967438971
ch26_aff_change: 4
ch26_eq_rearm: 0
ch27_events: 4563302840
ch27_poll: 4599441446
ch27_arm: 4432182806
ch27_aff_change: 4
ch27_eq_rearm: 0
ch28_events: 4
ch28_poll: 4
ch28_arm: 4
ch28_aff_change: 0
ch28_eq_rearm: 0
ch29_events: 6
ch29_poll: 6
ch29_arm: 6
ch29_aff_change: 0
ch29_eq_rearm: 0
ch30_events: 4
ch30_poll: 4
ch30_arm: 4
ch30_aff_change: 0
ch30_eq_rearm: 0
ch31_events: 4
ch31_poll: 4
ch31_arm: 4
ch31_aff_change: 0
ch31_eq_rearm: 0
ch32_events: 4
ch32_poll: 4
ch32_arm: 4
ch32_aff_change: 0
ch32_eq_rearm: 0
ch33_events: 4
ch33_poll: 4
ch33_arm: 4
ch33_aff_change: 0
ch33_eq_rearm: 0
ch34_events: 4
ch34_poll: 4
ch34_arm: 4
ch34_aff_change: 0
ch34_eq_rearm: 0
ch35_events: 4
ch35_poll: 4
ch35_arm: 4
ch35_aff_change: 0
ch35_eq_rearm: 0
ch36_events: 4
ch36_poll: 4
ch36_arm: 4
ch36_aff_change: 0
ch36_eq_rearm: 0
ch37_events: 4
ch37_poll: 4
ch37_arm: 4
ch37_aff_change: 0
ch37_eq_rearm: 0
ch38_events: 4
ch38_poll: 4
ch38_arm: 4
ch38_aff_change: 0
ch38_eq_rearm: 0
ch39_events: 4
ch39_poll: 4
ch39_arm: 4
ch39_aff_change: 0
ch39_eq_rearm: 0
ch40_events: 4
ch40_poll: 4
ch40_arm: 4
ch40_aff_change: 0
ch40_eq_rearm: 0
ch41_events: 4
ch41_poll: 4
ch41_arm: 4
ch41_aff_change: 0
ch41_eq_rearm: 0
ch42_events: 4
ch42_poll: 4
ch42_arm: 4
ch42_aff_change: 0
ch42_eq_rearm: 0
ch43_events: 4
ch43_poll: 4
ch43_arm: 4
ch43_aff_change: 0
ch43_eq_rearm: 0
ch44_events: 4
ch44_poll: 4
ch44_arm: 4
ch44_aff_change: 0
ch44_eq_rearm: 0
ch45_events: 4
ch45_poll: 4
ch45_arm: 4
ch45_aff_change: 0
ch45_eq_rearm: 0
ch46_events: 4
ch46_poll: 4
ch46_arm: 4
ch46_aff_change: 0
ch46_eq_rearm: 0
ch47_events: 4
ch47_poll: 4
ch47_arm: 4
ch47_aff_change: 0
ch47_eq_rearm: 0
ch48_events: 4
ch48_poll: 4
ch48_arm: 4
ch48_aff_change: 0
ch48_eq_rearm: 0
ch49_events: 4
ch49_poll: 4
ch49_arm: 4
ch49_aff_change: 0
ch49_eq_rearm: 0
ch50_events: 4
ch50_poll: 4
ch50_arm: 4
ch50_aff_change: 0
ch50_eq_rearm: 0
ch51_events: 4
ch51_poll: 4
ch51_arm: 4
ch51_aff_change: 0
ch51_eq_rearm: 0
ch52_events: 4
ch52_poll: 4
ch52_arm: 4
ch52_aff_change: 0
ch52_eq_rearm: 0
ch53_events: 4
ch53_poll: 4
ch53_arm: 4
ch53_aff_change: 0
ch53_eq_rearm: 0
ch54_events: 4
ch54_poll: 4
ch54_arm: 4
ch54_aff_change: 0
ch54_eq_rearm: 0
ch55_events: 4
ch55_poll: 4
ch55_arm: 4
ch55_aff_change: 0
ch55_eq_rearm: 0
rx0_packets: 21390033774
rx0_bytes: 27326856299122
rx0_csum_complete: 21339650906
rx0_csum_unnecessary: 0
rx0_csum_unnecessary_inner: 0
rx0_csum_none: 50382868
rx0_xdp_drop: 0
rx0_xdp_redirect: 0
rx0_lro_packets: 0
rx0_lro_bytes: 0
rx0_ecn_mark: 0
rx0_removed_vlan_packets: 21390033774
rx0_wqe_err: 0
rx0_mpwqe_filler_cqes: 0
rx0_mpwqe_filler_strides: 0
rx0_buff_alloc_err: 0
rx0_cqe_compress_blks: 481077641
rx0_cqe_compress_pkts: 1085647489
rx0_page_reuse: 0
rx0_cache_reuse: 19050049
rx0_cache_full: 10675964285
rx0_cache_empty: 37376
rx0_cache_busy: 10675966819
rx0_cache_waive: 0
rx0_congst_umr: 0
rx0_arfs_err: 0
rx0_xdp_tx_xmit: 0
rx0_xdp_tx_full: 0
rx0_xdp_tx_err: 0
rx0_xdp_tx_cqes: 0
rx1_packets: 19868919527
rx1_bytes: 26149716991561
rx1_csum_complete: 19868919527
rx1_csum_unnecessary: 0
rx1_csum_unnecessary_inner: 0
rx1_csum_none: 0
rx1_xdp_drop: 0
rx1_xdp_redirect: 0
rx1_lro_packets: 0
rx1_lro_bytes: 0
rx1_ecn_mark: 0
rx1_removed_vlan_packets: 19868919527
rx1_wqe_err: 0
rx1_mpwqe_filler_cqes: 0
rx1_mpwqe_filler_strides: 0
rx1_buff_alloc_err: 0
rx1_cqe_compress_blks: 420210560
rx1_cqe_compress_pkts: 941233388
rx1_page_reuse: 0
rx1_cache_reuse: 46200002
rx1_cache_full: 9888257242
rx1_cache_empty: 37376
rx1_cache_busy: 9888259746
rx1_cache_waive: 0
rx1_congst_umr: 0
rx1_arfs_err: 0
rx1_xdp_tx_xmit: 0
rx1_xdp_tx_full: 0
rx1_xdp_tx_err: 0
rx1_xdp_tx_cqes: 0
rx2_packets: 19575013662
rx2_bytes: 25759818417945
rx2_csum_complete: 19575013662
rx2_csum_unnecessary: 0
rx2_csum_unnecessary_inner: 0
rx2_csum_none: 0
rx2_xdp_drop: 0
rx2_xdp_redirect: 0
rx2_lro_packets: 0
rx2_lro_bytes: 0
rx2_ecn_mark: 0
rx2_removed_vlan_packets: 19575013662
rx2_wqe_err: 0
rx2_mpwqe_filler_cqes: 0
rx2_mpwqe_filler_strides: 0
rx2_buff_alloc_err: 0
rx2_cqe_compress_blks: 412345511
rx2_cqe_compress_pkts: 923376167
rx2_page_reuse: 0
rx2_cache_reuse: 38837731
rx2_cache_full: 9748666548
rx2_cache_empty: 37376
rx2_cache_busy: 9748669093
rx2_cache_waive: 0
rx2_congst_umr: 0
rx2_arfs_err: 0
rx2_xdp_tx_xmit: 0
rx2_xdp_tx_full: 0
rx2_xdp_tx_err: 0
rx2_xdp_tx_cqes: 0
rx3_packets: 19795911749
rx3_bytes: 25969475566905
rx3_csum_complete: 19795911749
rx3_csum_unnecessary: 0
rx3_csum_unnecessary_inner: 0
rx3_csum_none: 0
rx3_xdp_drop: 0
rx3_xdp_redirect: 0
rx3_lro_packets: 0
rx3_lro_bytes: 0
rx3_ecn_mark: 0
rx3_removed_vlan_packets: 19795911749
rx3_wqe_err: 0
rx3_mpwqe_filler_cqes: 0
rx3_mpwqe_filler_strides: 0
rx3_buff_alloc_err: 0
rx3_cqe_compress_blks: 416658765
rx3_cqe_compress_pkts: 934986266
rx3_page_reuse: 0
rx3_cache_reuse: 34542124
rx3_cache_full: 9863411232
rx3_cache_empty: 37376
rx3_cache_busy: 9863413732
rx3_cache_waive: 0
rx3_congst_umr: 0
rx3_arfs_err: 0
rx3_xdp_tx_xmit: 0
rx3_xdp_tx_full: 0
rx3_xdp_tx_err: 0
rx3_xdp_tx_cqes: 0
rx4_packets: 20445652378
rx4_bytes: 26949065110265
rx4_csum_complete: 20445652378
rx4_csum_unnecessary: 0
rx4_csum_unnecessary_inner: 0
rx4_csum_none: 0
rx4_xdp_drop: 0
rx4_xdp_redirect: 0
rx4_lro_packets: 0
rx4_lro_bytes: 0
rx4_ecn_mark: 0
rx4_removed_vlan_packets: 20445652378
rx4_wqe_err: 0
rx4_mpwqe_filler_cqes: 0
rx4_mpwqe_filler_strides: 0
rx4_buff_alloc_err: 0
rx4_cqe_compress_blks: 506085858
rx4_cqe_compress_pkts: 1147860328
rx4_page_reuse: 0
rx4_cache_reuse: 10122542864
rx4_cache_full: 100281206
rx4_cache_empty: 37376
rx4_cache_busy: 100283304
rx4_cache_waive: 0
rx4_congst_umr: 0
rx4_arfs_err: 0
rx4_xdp_tx_xmit: 0
rx4_xdp_tx_full: 0
rx4_xdp_tx_err: 0
rx4_xdp_tx_cqes: 0
rx5_packets: 19622362246
rx5_bytes: 25843450982982
rx5_csum_complete: 19622362246
rx5_csum_unnecessary: 0
rx5_csum_unnecessary_inner: 0
rx5_csum_none: 0
rx5_xdp_drop: 0
rx5_xdp_redirect: 0
rx5_lro_packets: 0
rx5_lro_bytes: 0
rx5_ecn_mark: 0
rx5_removed_vlan_packets: 19622362246
rx5_wqe_err: 0
rx5_mpwqe_filler_cqes: 0
rx5_mpwqe_filler_strides: 0
rx5_buff_alloc_err: 0
rx5_cqe_compress_blks: 422840924
rx5_cqe_compress_pkts: 948005878
rx5_page_reuse: 0
rx5_cache_reuse: 31285453
rx5_cache_full: 9779893117
rx5_cache_empty: 37376
rx5_cache_busy: 9779895647
rx5_cache_waive: 0
rx5_congst_umr: 0
rx5_arfs_err: 0
rx5_xdp_tx_xmit: 0
rx5_xdp_tx_full: 0
rx5_xdp_tx_err: 0
rx5_xdp_tx_cqes: 0
rx6_packets: 19788231278
rx6_bytes: 25985783006486
rx6_csum_complete: 19788231278
rx6_csum_unnecessary: 0
rx6_csum_unnecessary_inner: 0
rx6_csum_none: 0
rx6_xdp_drop: 0
rx6_xdp_redirect: 0
rx6_lro_packets: 0
rx6_lro_bytes: 0
rx6_ecn_mark: 0
rx6_removed_vlan_packets: 19788231278
rx6_wqe_err: 0
rx6_mpwqe_filler_cqes: 0
rx6_mpwqe_filler_strides: 0
rx6_buff_alloc_err: 0
rx6_cqe_compress_blks: 418799056
rx6_cqe_compress_pkts: 938282685
rx6_page_reuse: 0
rx6_cache_reuse: 18114793
rx6_cache_full: 9875998295
rx6_cache_empty: 37376
rx6_cache_busy: 9876000831
rx6_cache_waive: 0
rx6_congst_umr: 0
rx6_arfs_err: 0
rx6_xdp_tx_xmit: 0
rx6_xdp_tx_full: 0
rx6_xdp_tx_err: 0
rx6_xdp_tx_cqes: 0
rx7_packets: 19795759168
rx7_bytes: 26085056586860
rx7_csum_complete: 19795759168
rx7_csum_unnecessary: 0
rx7_csum_unnecessary_inner: 0
rx7_csum_none: 0
rx7_xdp_drop: 0
rx7_xdp_redirect: 0
rx7_lro_packets: 0
rx7_lro_bytes: 0
rx7_ecn_mark: 0
rx7_removed_vlan_packets: 19795759168
rx7_wqe_err: 0
rx7_mpwqe_filler_cqes: 0
rx7_mpwqe_filler_strides: 0
rx7_buff_alloc_err: 0
rx7_cqe_compress_blks: 413959224
rx7_cqe_compress_pkts: 927675936
rx7_page_reuse: 0
rx7_cache_reuse: 23902990
rx7_cache_full: 9873974042
rx7_cache_empty: 37376
rx7_cache_busy: 9873976574
rx7_cache_waive: 0
rx7_congst_umr: 0
rx7_arfs_err: 0
rx7_xdp_tx_xmit: 0
rx7_xdp_tx_full: 0
rx7_xdp_tx_err: 0
rx7_xdp_tx_cqes: 0
rx8_packets: 19963477439
rx8_bytes: 26384640501789
rx8_csum_complete: 19963477439
rx8_csum_unnecessary: 0
rx8_csum_unnecessary_inner: 0
rx8_csum_none: 0
rx8_xdp_drop: 0
rx8_xdp_redirect: 0
rx8_lro_packets: 0
rx8_lro_bytes: 0
rx8_ecn_mark: 0
rx8_removed_vlan_packets: 19963477439
rx8_wqe_err: 0
rx8_mpwqe_filler_cqes: 0
rx8_mpwqe_filler_strides: 0
rx8_buff_alloc_err: 0
rx8_cqe_compress_blks: 420422857
rx8_cqe_compress_pkts: 942720292
rx8_page_reuse: 0
rx8_cache_reuse: 88181713
rx8_cache_full: 9893554525
rx8_cache_empty: 37376
rx8_cache_busy: 9893556983
rx8_cache_waive: 0
rx8_congst_umr: 0
rx8_arfs_err: 0
rx8_xdp_tx_xmit: 0
rx8_xdp_tx_full: 0
rx8_xdp_tx_err: 0
rx8_xdp_tx_cqes: 0
rx9_packets: 19726642138
rx9_bytes: 26063924286499
rx9_csum_complete: 19726642138
rx9_csum_unnecessary: 0
rx9_csum_unnecessary_inner: 0
rx9_csum_none: 0
rx9_xdp_drop: 0
rx9_xdp_redirect: 0
rx9_lro_packets: 0
rx9_lro_bytes: 0
rx9_ecn_mark: 0
rx9_removed_vlan_packets: 19726642138
rx9_wqe_err: 0
rx9_mpwqe_filler_cqes: 0
rx9_mpwqe_filler_strides: 0
rx9_buff_alloc_err: 0
rx9_cqe_compress_blks: 424227411
rx9_cqe_compress_pkts: 951534873
rx9_page_reuse: 0
rx9_cache_reuse: 482901440
rx9_cache_full: 9380417487
rx9_cache_empty: 37376
rx9_cache_busy: 9380419608
rx9_cache_waive: 0
rx9_congst_umr: 0
rx9_arfs_err: 0
rx9_xdp_tx_xmit: 0
rx9_xdp_tx_full: 0
rx9_xdp_tx_err: 0
rx9_xdp_tx_cqes: 0
rx10_packets: 19901229170
rx10_bytes: 26300854495044
rx10_csum_complete: 19901229170
rx10_csum_unnecessary: 0
rx10_csum_unnecessary_inner: 0
rx10_csum_none: 0
rx10_xdp_drop: 0
rx10_xdp_redirect: 0
rx10_lro_packets: 0
rx10_lro_bytes: 0
rx10_ecn_mark: 0
rx10_removed_vlan_packets: 19901229170
rx10_wqe_err: 0
rx10_mpwqe_filler_cqes: 0
rx10_mpwqe_filler_strides: 0
rx10_buff_alloc_err: 0
rx10_cqe_compress_blks: 419082938
rx10_cqe_compress_pkts: 940791347
rx10_page_reuse: 0
rx10_cache_reuse: 14896055
rx10_cache_full: 9935715977
rx10_cache_empty: 37376
rx10_cache_busy: 9935718513
rx10_cache_waive: 0
rx10_congst_umr: 0
rx10_arfs_err: 0
rx10_xdp_tx_xmit: 0
rx10_xdp_tx_full: 0
rx10_xdp_tx_err: 0
rx10_xdp_tx_cqes: 0
rx11_packets: 20352190494
rx11_bytes: 26851034425372
rx11_csum_complete: 20352190494
rx11_csum_unnecessary: 0
rx11_csum_unnecessary_inner: 0
rx11_csum_none: 0
rx11_xdp_drop: 0
rx11_xdp_redirect: 0
rx11_lro_packets: 0
rx11_lro_bytes: 0
rx11_ecn_mark: 0
rx11_removed_vlan_packets: 20352190494
rx11_wqe_err: 0
rx11_mpwqe_filler_cqes: 0
rx11_mpwqe_filler_strides: 0
rx11_buff_alloc_err: 0
rx11_cqe_compress_blks: 501992147
rx11_cqe_compress_pkts: 1140398610
rx11_page_reuse: 0
rx11_cache_reuse: 10071721531
rx11_cache_full: 104371621
rx11_cache_empty: 37376
rx11_cache_busy: 104373697
rx11_cache_waive: 0
rx11_congst_umr: 0
rx11_arfs_err: 0
rx11_xdp_tx_xmit: 0
rx11_xdp_tx_full: 0
rx11_xdp_tx_err: 0
rx11_xdp_tx_cqes: 0
rx12_packets: 19934747149
rx12_bytes: 26296478787829
rx12_csum_complete: 19934747149
rx12_csum_unnecessary: 0
rx12_csum_unnecessary_inner: 0
rx12_csum_none: 0
rx12_xdp_drop: 0
rx12_xdp_redirect: 0
rx12_lro_packets: 0
rx12_lro_bytes: 0
rx12_ecn_mark: 0
rx12_removed_vlan_packets: 19934747149
rx12_wqe_err: 0
rx12_mpwqe_filler_cqes: 0
rx12_mpwqe_filler_strides: 0
rx12_buff_alloc_err: 0
rx12_cqe_compress_blks: 443350570
rx12_cqe_compress_pkts: 995997220
rx12_page_reuse: 0
rx12_cache_reuse: 9864934174
rx12_cache_full: 102437428
rx12_cache_empty: 37376
rx12_cache_busy: 102439382
rx12_cache_waive: 0
rx12_congst_umr: 0
rx12_arfs_err: 0
rx12_xdp_tx_xmit: 0
rx12_xdp_tx_full: 0
rx12_xdp_tx_err: 0
rx12_xdp_tx_cqes: 0
rx13_packets: 19866908096
rx13_bytes: 26160931936186
rx13_csum_complete: 19866908096
rx13_csum_unnecessary: 0
rx13_csum_unnecessary_inner: 0
rx13_csum_none: 0
rx13_xdp_drop: 0
rx13_xdp_redirect: 0
rx13_lro_packets: 0
rx13_lro_bytes: 0
rx13_ecn_mark: 0
rx13_removed_vlan_packets: 19866908096
rx13_wqe_err: 0
rx13_mpwqe_filler_cqes: 0
rx13_mpwqe_filler_strides: 0
rx13_buff_alloc_err: 0
rx13_cqe_compress_blks: 413640141
rx13_cqe_compress_pkts: 926175066
rx13_page_reuse: 0
rx13_cache_reuse: 36358610
rx13_cache_full: 9897092921
rx13_cache_empty: 37376
rx13_cache_busy: 9897095422
rx13_cache_waive: 0
rx13_congst_umr: 0
rx13_arfs_err: 0
rx13_xdp_tx_xmit: 0
rx13_xdp_tx_full: 0
rx13_xdp_tx_err: 0
rx13_xdp_tx_cqes: 0
rx14_packets: 20229035746
rx14_bytes: 26655092809172
rx14_csum_complete: 20229035746
rx14_csum_unnecessary: 0
rx14_csum_unnecessary_inner: 0
rx14_csum_none: 0
rx14_xdp_drop: 0
rx14_xdp_redirect: 0
rx14_lro_packets: 0
rx14_lro_bytes: 0
rx14_ecn_mark: 0
rx14_removed_vlan_packets: 20229035746
rx14_wqe_err: 0
rx14_mpwqe_filler_cqes: 0
rx14_mpwqe_filler_strides: 0
rx14_buff_alloc_err: 0
rx14_cqe_compress_blks: 460990337
rx14_cqe_compress_pkts: 1041287948
rx14_page_reuse: 0
rx14_cache_reuse: 25649275
rx14_cache_full: 10088866045
rx14_cache_empty: 37376
rx14_cache_busy: 10088868574
rx14_cache_waive: 0
rx14_congst_umr: 0
rx14_arfs_err: 0
rx14_xdp_tx_xmit: 0
rx14_xdp_tx_full: 0
rx14_xdp_tx_err: 0
rx14_xdp_tx_cqes: 0
rx15_packets: 20528177154
rx15_bytes: 27029263893264
rx15_csum_complete: 20528177154
rx15_csum_unnecessary: 0
rx15_csum_unnecessary_inner: 0
rx15_csum_none: 0
rx15_xdp_drop: 0
rx15_xdp_redirect: 0
rx15_lro_packets: 0
rx15_lro_bytes: 0
rx15_ecn_mark: 0
rx15_removed_vlan_packets: 20528177154
rx15_wqe_err: 0
rx15_mpwqe_filler_cqes: 0
rx15_mpwqe_filler_strides: 0
rx15_buff_alloc_err: 0
rx15_cqe_compress_blks: 476776176
rx15_cqe_compress_pkts: 1076153263
rx15_page_reuse: 0
rx15_cache_reuse: 48426735
rx15_cache_full: 10215659289
rx15_cache_empty: 37376
rx15_cache_busy: 10215661817
rx15_cache_waive: 0
rx15_congst_umr: 0
rx15_arfs_err: 0
rx15_xdp_tx_xmit: 0
rx15_xdp_tx_full: 0
rx15_xdp_tx_err: 0
rx15_xdp_tx_cqes: 0
rx16_packets: 16104078098
rx16_bytes: 21256361789679
rx16_csum_complete: 16104078098
rx16_csum_unnecessary: 0
rx16_csum_unnecessary_inner: 0
rx16_csum_none: 0
rx16_xdp_drop: 0
rx16_xdp_redirect: 0
rx16_lro_packets: 0
rx16_lro_bytes: 0
rx16_ecn_mark: 0
rx16_removed_vlan_packets: 16104078098
rx16_wqe_err: 0
rx16_mpwqe_filler_cqes: 0
rx16_mpwqe_filler_strides: 0
rx16_buff_alloc_err: 0
rx16_cqe_compress_blks: 352082054
rx16_cqe_compress_pkts: 787161670
rx16_page_reuse: 0
rx16_cache_reuse: 25912567
rx16_cache_full: 8026124051
rx16_cache_empty: 37376
rx16_cache_busy: 8026126465
rx16_cache_waive: 0
rx16_congst_umr: 0
rx16_arfs_err: 0
rx16_xdp_tx_xmit: 0
rx16_xdp_tx_full: 0
rx16_xdp_tx_err: 0
rx16_xdp_tx_cqes: 0
rx17_packets: 16314055017
rx17_bytes: 21589139030173
rx17_csum_complete: 16314055017
rx17_csum_unnecessary: 0
rx17_csum_unnecessary_inner: 0
rx17_csum_none: 0
rx17_xdp_drop: 0
rx17_xdp_redirect: 0
rx17_lro_packets: 0
rx17_lro_bytes: 0
rx17_ecn_mark: 0
rx17_removed_vlan_packets: 16314055017
rx17_wqe_err: 0
rx17_mpwqe_filler_cqes: 0
rx17_mpwqe_filler_strides: 0
rx17_buff_alloc_err: 0
rx17_cqe_compress_blks: 387834541
rx17_cqe_compress_pkts: 871851081
rx17_page_reuse: 0
rx17_cache_reuse: 24021313
rx17_cache_full: 8133003829
rx17_cache_empty: 37376
rx17_cache_busy: 8133006175
rx17_cache_waive: 0
rx17_congst_umr: 0
rx17_arfs_err: 0
rx17_xdp_tx_xmit: 0
rx17_xdp_tx_full: 0
rx17_xdp_tx_err: 0
rx17_xdp_tx_cqes: 0
rx18_packets: 16439016814
rx18_bytes: 21648651917475
rx18_csum_complete: 16439016814
rx18_csum_unnecessary: 0
rx18_csum_unnecessary_inner: 0
rx18_csum_none: 0
rx18_xdp_drop: 0
rx18_xdp_redirect: 0
rx18_lro_packets: 0
rx18_lro_bytes: 0
rx18_ecn_mark: 0
rx18_removed_vlan_packets: 16439016814
rx18_wqe_err: 0
rx18_mpwqe_filler_cqes: 0
rx18_mpwqe_filler_strides: 0
rx18_buff_alloc_err: 0
rx18_cqe_compress_blks: 375066666
rx18_cqe_compress_pkts: 843563974
rx18_page_reuse: 0
rx18_cache_reuse: 8151064266
rx18_cache_full: 68442025
rx18_cache_empty: 37376
rx18_cache_busy: 68444122
rx18_cache_waive: 0
rx18_congst_umr: 0
rx18_arfs_err: 0
rx18_xdp_tx_xmit: 0
rx18_xdp_tx_full: 0
rx18_xdp_tx_err: 0
rx18_xdp_tx_cqes: 0
rx19_packets: 16641223506
rx19_bytes: 21964749940935
rx19_csum_complete: 16641223506
rx19_csum_unnecessary: 0
rx19_csum_unnecessary_inner: 0
rx19_csum_none: 0
rx19_xdp_drop: 0
rx19_xdp_redirect: 0
rx19_lro_packets: 0
rx19_lro_bytes: 0
rx19_ecn_mark: 0
rx19_removed_vlan_packets: 16641223506
rx19_wqe_err: 0
rx19_mpwqe_filler_cqes: 0
rx19_mpwqe_filler_strides: 0
rx19_buff_alloc_err: 0
rx19_cqe_compress_blks: 387825932
rx19_cqe_compress_pkts: 872266355
rx19_page_reuse: 0
rx19_cache_reuse: 116433620
rx19_cache_full: 8204175954
rx19_cache_empty: 37376
rx19_cache_busy: 8204178120
rx19_cache_waive: 0
rx19_congst_umr: 0
rx19_arfs_err: 0
rx19_xdp_tx_xmit: 0
rx19_xdp_tx_full: 0
rx19_xdp_tx_err: 0
rx19_xdp_tx_cqes: 0
rx20_packets: 16206927741
rx20_bytes: 21387447038430
rx20_csum_complete: 16206927741
rx20_csum_unnecessary: 0
rx20_csum_unnecessary_inner: 0
rx20_csum_none: 0
rx20_xdp_drop: 0
rx20_xdp_redirect: 0
rx20_lro_packets: 0
rx20_lro_bytes: 0
rx20_ecn_mark: 0
rx20_removed_vlan_packets: 16206927741
rx20_wqe_err: 0
rx20_mpwqe_filler_cqes: 0
rx20_mpwqe_filler_strides: 0
rx20_buff_alloc_err: 0
rx20_cqe_compress_blks: 370144620
rx20_cqe_compress_pkts: 829122671
rx20_page_reuse: 0
rx20_cache_reuse: 8053733744
rx20_cache_full: 49728026
rx20_cache_empty: 37376
rx20_cache_busy: 49730116
rx20_cache_waive: 0
rx20_congst_umr: 0
rx20_arfs_err: 0
rx20_xdp_tx_xmit: 0
rx20_xdp_tx_full: 0
rx20_xdp_tx_err: 0
rx20_xdp_tx_cqes: 0
rx21_packets: 16562361314
rx21_bytes: 21856653284356
rx21_csum_complete: 16562361314
rx21_csum_unnecessary: 0
rx21_csum_unnecessary_inner: 0
rx21_csum_none: 0
rx21_xdp_drop: 0
rx21_xdp_redirect: 0
rx21_lro_packets: 0
rx21_lro_bytes: 0
rx21_ecn_mark: 0
rx21_removed_vlan_packets: 16562361314
rx21_wqe_err: 0
rx21_mpwqe_filler_cqes: 0
rx21_mpwqe_filler_strides: 0
rx21_buff_alloc_err: 0
rx21_cqe_compress_blks: 350790425
rx21_cqe_compress_pkts: 783850729
rx21_page_reuse: 0
rx21_cache_reuse: 28077493
rx21_cache_full: 8253100706
rx21_cache_empty: 37376
rx21_cache_busy: 8253103147
rx21_cache_waive: 0
rx21_congst_umr: 0
rx21_arfs_err: 0
rx21_xdp_tx_xmit: 0
rx21_xdp_tx_full: 0
rx21_xdp_tx_err: 0
rx21_xdp_tx_cqes: 0
rx22_packets: 16350307571
rx22_bytes: 21408575325592
rx22_csum_complete: 16350307571
rx22_csum_unnecessary: 0
rx22_csum_unnecessary_inner: 0
rx22_csum_none: 0
rx22_xdp_drop: 0
rx22_xdp_redirect: 0
rx22_lro_packets: 0
rx22_lro_bytes: 0
rx22_ecn_mark: 0
rx22_removed_vlan_packets: 16350307571
rx22_wqe_err: 0
rx22_mpwqe_filler_cqes: 0
rx22_mpwqe_filler_strides: 0
rx22_buff_alloc_err: 0
rx22_cqe_compress_blks: 353531065
rx22_cqe_compress_pkts: 790814415
rx22_page_reuse: 0
rx22_cache_reuse: 16934343
rx22_cache_full: 8158216889
rx22_cache_empty: 37376
rx22_cache_busy: 8158219417
rx22_cache_waive: 0
rx22_congst_umr: 0
rx22_arfs_err: 0
rx22_xdp_tx_xmit: 0
rx22_xdp_tx_full: 0
rx22_xdp_tx_err: 0
rx22_xdp_tx_cqes: 0
rx23_packets: 16019811764
rx23_bytes: 21137182570985
rx23_csum_complete: 16019811764
rx23_csum_unnecessary: 0
rx23_csum_unnecessary_inner: 0
rx23_csum_none: 0
rx23_xdp_drop: 0
rx23_xdp_redirect: 0
rx23_lro_packets: 0
rx23_lro_bytes: 0
rx23_ecn_mark: 0
rx23_removed_vlan_packets: 16019811764
rx23_wqe_err: 0
rx23_mpwqe_filler_cqes: 0
rx23_mpwqe_filler_strides: 0
rx23_buff_alloc_err: 0
rx23_cqe_compress_blks: 349733033
rx23_cqe_compress_pkts: 781248862
rx23_page_reuse: 0
rx23_cache_reuse: 33422343
rx23_cache_full: 7976481152
rx23_cache_empty: 37376
rx23_cache_busy: 7976483525
rx23_cache_waive: 0
rx23_congst_umr: 0
rx23_arfs_err: 0
rx23_xdp_tx_xmit: 0
rx23_xdp_tx_full: 0
rx23_xdp_tx_err: 0
rx23_xdp_tx_cqes: 0
rx24_packets: 16212040646
rx24_bytes: 21393399325700
rx24_csum_complete: 16212040646
rx24_csum_unnecessary: 0
rx24_csum_unnecessary_inner: 0
rx24_csum_none: 0
rx24_xdp_drop: 0
rx24_xdp_redirect: 0
rx24_lro_packets: 0
rx24_lro_bytes: 0
rx24_ecn_mark: 0
rx24_removed_vlan_packets: 16212040646
rx24_wqe_err: 0
rx24_mpwqe_filler_cqes: 0
rx24_mpwqe_filler_strides: 0
rx24_buff_alloc_err: 0
rx24_cqe_compress_blks: 379833752
rx24_cqe_compress_pkts: 852020179
rx24_page_reuse: 0
rx24_cache_reuse: 8033552512
rx24_cache_full: 72465843
rx24_cache_empty: 37376
rx24_cache_busy: 72467789
rx24_cache_waive: 0
rx24_congst_umr: 0
rx24_arfs_err: 0
rx24_xdp_tx_xmit: 0
rx24_xdp_tx_full: 0
rx24_xdp_tx_err: 0
rx24_xdp_tx_cqes: 0
rx25_packets: 16412186257
rx25_bytes: 21651198388407
rx25_csum_complete: 16412186257
rx25_csum_unnecessary: 0
rx25_csum_unnecessary_inner: 0
rx25_csum_none: 0
rx25_xdp_drop: 0
rx25_xdp_redirect: 0
rx25_lro_packets: 0
rx25_lro_bytes: 0
rx25_ecn_mark: 0
rx25_removed_vlan_packets: 16412186257
rx25_wqe_err: 0
rx25_mpwqe_filler_cqes: 0
rx25_mpwqe_filler_strides: 0
rx25_buff_alloc_err: 0
rx25_cqe_compress_blks: 383979685
rx25_cqe_compress_pkts: 861985772
rx25_page_reuse: 0
rx25_cache_reuse: 8129807841
rx25_cache_full: 76283342
rx25_cache_empty: 37376
rx25_cache_busy: 76285271
rx25_cache_waive: 0
rx25_congst_umr: 0
rx25_arfs_err: 0
rx25_xdp_tx_xmit: 0
rx25_xdp_tx_full: 0
rx25_xdp_tx_err: 0
rx25_xdp_tx_cqes: 0
rx26_packets: 16304310003
rx26_bytes: 21571217538721
rx26_csum_complete: 16304310003
rx26_csum_unnecessary: 0
rx26_csum_unnecessary_inner: 0
rx26_csum_none: 0
rx26_xdp_drop: 0
rx26_xdp_redirect: 0
rx26_lro_packets: 0
rx26_lro_bytes: 0
rx26_ecn_mark: 0
rx26_removed_vlan_packets: 16304310003
rx26_wqe_err: 0
rx26_mpwqe_filler_cqes: 0
rx26_mpwqe_filler_strides: 0
rx26_buff_alloc_err: 0
rx26_cqe_compress_blks: 353314041
rx26_cqe_compress_pkts: 788838424
rx26_page_reuse: 0
rx26_cache_reuse: 19673790
rx26_cache_full: 8132478659
rx26_cache_empty: 37376
rx26_cache_busy: 8132481198
rx26_cache_waive: 0
rx26_congst_umr: 0
rx26_arfs_err: 0
rx26_xdp_tx_xmit: 0
rx26_xdp_tx_full: 0
rx26_xdp_tx_err: 0
rx26_xdp_tx_cqes: 0
rx27_packets: 16171856079
rx27_bytes: 21376891736540
rx27_csum_complete: 16171856079
rx27_csum_unnecessary: 0
rx27_csum_unnecessary_inner: 0
rx27_csum_none: 0
rx27_xdp_drop: 0
rx27_xdp_redirect: 0
rx27_lro_packets: 0
rx27_lro_bytes: 0
rx27_ecn_mark: 0
rx27_removed_vlan_packets: 16171856079
rx27_wqe_err: 0
rx27_mpwqe_filler_cqes: 0
rx27_mpwqe_filler_strides: 0
rx27_buff_alloc_err: 0
rx27_cqe_compress_blks: 386632845
rx27_cqe_compress_pkts: 869362576
rx27_page_reuse: 0
rx27_cache_reuse: 10070560
rx27_cache_full: 8075854928
rx27_cache_empty: 37376
rx27_cache_busy: 8075857468
rx27_cache_waive: 0
rx27_congst_umr: 0
rx27_arfs_err: 0
rx27_xdp_tx_xmit: 0
rx27_xdp_tx_full: 0
rx27_xdp_tx_err: 0
rx27_xdp_tx_cqes: 0
rx28_packets: 0
rx28_bytes: 0
rx28_csum_complete: 0
rx28_csum_unnecessary: 0
rx28_csum_unnecessary_inner: 0
rx28_csum_none: 0
rx28_xdp_drop: 0
rx28_xdp_redirect: 0
rx28_lro_packets: 0
rx28_lro_bytes: 0
rx28_ecn_mark: 0
rx28_removed_vlan_packets: 0
rx28_wqe_err: 0
rx28_mpwqe_filler_cqes: 0
rx28_mpwqe_filler_strides: 0
rx28_buff_alloc_err: 0
rx28_cqe_compress_blks: 0
rx28_cqe_compress_pkts: 0
rx28_page_reuse: 0
rx28_cache_reuse: 0
rx28_cache_full: 0
rx28_cache_empty: 2560
rx28_cache_busy: 0
rx28_cache_waive: 0
rx28_congst_umr: 0
rx28_arfs_err: 0
rx28_xdp_tx_xmit: 0
rx28_xdp_tx_full: 0
rx28_xdp_tx_err: 0
rx28_xdp_tx_cqes: 0
rx29_packets: 0
rx29_bytes: 0
rx29_csum_complete: 0
rx29_csum_unnecessary: 0
rx29_csum_unnecessary_inner: 0
rx29_csum_none: 0
rx29_xdp_drop: 0
rx29_xdp_redirect: 0
rx29_lro_packets: 0
rx29_lro_bytes: 0
rx29_ecn_mark: 0
rx29_removed_vlan_packets: 0
rx29_wqe_err: 0
rx29_mpwqe_filler_cqes: 0
rx29_mpwqe_filler_strides: 0
rx29_buff_alloc_err: 0
rx29_cqe_compress_blks: 0
rx29_cqe_compress_pkts: 0
rx29_page_reuse: 0
rx29_cache_reuse: 0
rx29_cache_full: 0
rx29_cache_empty: 2560
rx29_cache_busy: 0
rx29_cache_waive: 0
rx29_congst_umr: 0
rx29_arfs_err: 0
rx29_xdp_tx_xmit: 0
rx29_xdp_tx_full: 0
rx29_xdp_tx_err: 0
rx29_xdp_tx_cqes: 0
rx30_packets: 0
rx30_bytes: 0
rx30_csum_complete: 0
rx30_csum_unnecessary: 0
rx30_csum_unnecessary_inner: 0
rx30_csum_none: 0
rx30_xdp_drop: 0
rx30_xdp_redirect: 0
rx30_lro_packets: 0
rx30_lro_bytes: 0
rx30_ecn_mark: 0
rx30_removed_vlan_packets: 0
rx30_wqe_err: 0
rx30_mpwqe_filler_cqes: 0
rx30_mpwqe_filler_strides: 0
rx30_buff_alloc_err: 0
rx30_cqe_compress_blks: 0
rx30_cqe_compress_pkts: 0
rx30_page_reuse: 0
rx30_cache_reuse: 0
rx30_cache_full: 0
rx30_cache_empty: 2560
rx30_cache_busy: 0
rx30_cache_waive: 0
rx30_congst_umr: 0
rx30_arfs_err: 0
rx30_xdp_tx_xmit: 0
rx30_xdp_tx_full: 0
rx30_xdp_tx_err: 0
rx30_xdp_tx_cqes: 0
rx31_packets: 0
rx31_bytes: 0
rx31_csum_complete: 0
rx31_csum_unnecessary: 0
rx31_csum_unnecessary_inner: 0
rx31_csum_none: 0
rx31_xdp_drop: 0
rx31_xdp_redirect: 0
rx31_lro_packets: 0
rx31_lro_bytes: 0
rx31_ecn_mark: 0
rx31_removed_vlan_packets: 0
rx31_wqe_err: 0
rx31_mpwqe_filler_cqes: 0
rx31_mpwqe_filler_strides: 0
rx31_buff_alloc_err: 0
rx31_cqe_compress_blks: 0
rx31_cqe_compress_pkts: 0
rx31_page_reuse: 0
rx31_cache_reuse: 0
rx31_cache_full: 0
rx31_cache_empty: 2560
rx31_cache_busy: 0
rx31_cache_waive: 0
rx31_congst_umr: 0
rx31_arfs_err: 0
rx31_xdp_tx_xmit: 0
rx31_xdp_tx_full: 0
rx31_xdp_tx_err: 0
rx31_xdp_tx_cqes: 0
rx32_packets: 0
rx32_bytes: 0
rx32_csum_complete: 0
rx32_csum_unnecessary: 0
rx32_csum_unnecessary_inner: 0
rx32_csum_none: 0
rx32_xdp_drop: 0
rx32_xdp_redirect: 0
rx32_lro_packets: 0
rx32_lro_bytes: 0
rx32_ecn_mark: 0
rx32_removed_vlan_packets: 0
rx32_wqe_err: 0
rx32_mpwqe_filler_cqes: 0
rx32_mpwqe_filler_strides: 0
rx32_buff_alloc_err: 0
rx32_cqe_compress_blks: 0
rx32_cqe_compress_pkts: 0
rx32_page_reuse: 0
rx32_cache_reuse: 0
rx32_cache_full: 0
rx32_cache_empty: 2560
rx32_cache_busy: 0
rx32_cache_waive: 0
rx32_congst_umr: 0
rx32_arfs_err: 0
rx32_xdp_tx_xmit: 0
rx32_xdp_tx_full: 0
rx32_xdp_tx_err: 0
rx32_xdp_tx_cqes: 0
rx33_packets: 0
rx33_bytes: 0
rx33_csum_complete: 0
rx33_csum_unnecessary: 0
rx33_csum_unnecessary_inner: 0
rx33_csum_none: 0
rx33_xdp_drop: 0
rx33_xdp_redirect: 0
rx33_lro_packets: 0
rx33_lro_bytes: 0
rx33_ecn_mark: 0
rx33_removed_vlan_packets: 0
rx33_wqe_err: 0
rx33_mpwqe_filler_cqes: 0
rx33_mpwqe_filler_strides: 0
rx33_buff_alloc_err: 0
rx33_cqe_compress_blks: 0
rx33_cqe_compress_pkts: 0
rx33_page_reuse: 0
rx33_cache_reuse: 0
rx33_cache_full: 0
rx33_cache_empty: 2560
rx33_cache_busy: 0
rx33_cache_waive: 0
rx33_congst_umr: 0
rx33_arfs_err: 0
rx33_xdp_tx_xmit: 0
rx33_xdp_tx_full: 0
rx33_xdp_tx_err: 0
rx33_xdp_tx_cqes: 0
rx34_packets: 0
rx34_bytes: 0
rx34_csum_complete: 0
rx34_csum_unnecessary: 0
rx34_csum_unnecessary_inner: 0
rx34_csum_none: 0
rx34_xdp_drop: 0
rx34_xdp_redirect: 0
rx34_lro_packets: 0
rx34_lro_bytes: 0
rx34_ecn_mark: 0
rx34_removed_vlan_packets: 0
rx34_wqe_err: 0
rx34_mpwqe_filler_cqes: 0
rx34_mpwqe_filler_strides: 0
rx34_buff_alloc_err: 0
rx34_cqe_compress_blks: 0
rx34_cqe_compress_pkts: 0
rx34_page_reuse: 0
rx34_cache_reuse: 0
rx34_cache_full: 0
rx34_cache_empty: 2560
rx34_cache_busy: 0
rx34_cache_waive: 0
rx34_congst_umr: 0
rx34_arfs_err: 0
rx34_xdp_tx_xmit: 0
rx34_xdp_tx_full: 0
rx34_xdp_tx_err: 0
rx34_xdp_tx_cqes: 0
rx35_packets: 0
rx35_bytes: 0
rx35_csum_complete: 0
rx35_csum_unnecessary: 0
rx35_csum_unnecessary_inner: 0
rx35_csum_none: 0
rx35_xdp_drop: 0
rx35_xdp_redirect: 0
rx35_lro_packets: 0
rx35_lro_bytes: 0
rx35_ecn_mark: 0
rx35_removed_vlan_packets: 0
rx35_wqe_err: 0
rx35_mpwqe_filler_cqes: 0
rx35_mpwqe_filler_strides: 0
rx35_buff_alloc_err: 0
rx35_cqe_compress_blks: 0
rx35_cqe_compress_pkts: 0
rx35_page_reuse: 0
rx35_cache_reuse: 0
rx35_cache_full: 0
rx35_cache_empty: 2560
rx35_cache_busy: 0
rx35_cache_waive: 0
rx35_congst_umr: 0
rx35_arfs_err: 0
rx35_xdp_tx_xmit: 0
rx35_xdp_tx_full: 0
rx35_xdp_tx_err: 0
rx35_xdp_tx_cqes: 0
rx36_packets: 0
rx36_bytes: 0
rx36_csum_complete: 0
rx36_csum_unnecessary: 0
rx36_csum_unnecessary_inner: 0
rx36_csum_none: 0
rx36_xdp_drop: 0
rx36_xdp_redirect: 0
rx36_lro_packets: 0
rx36_lro_bytes: 0
rx36_ecn_mark: 0
rx36_removed_vlan_packets: 0
rx36_wqe_err: 0
rx36_mpwqe_filler_cqes: 0
rx36_mpwqe_filler_strides: 0
rx36_buff_alloc_err: 0
rx36_cqe_compress_blks: 0
rx36_cqe_compress_pkts: 0
rx36_page_reuse: 0
rx36_cache_reuse: 0
rx36_cache_full: 0
rx36_cache_empty: 2560
rx36_cache_busy: 0
rx36_cache_waive: 0
rx36_congst_umr: 0
rx36_arfs_err: 0
rx36_xdp_tx_xmit: 0
rx36_xdp_tx_full: 0
rx36_xdp_tx_err: 0
rx36_xdp_tx_cqes: 0
rx37_packets: 0
rx37_bytes: 0
rx37_csum_complete: 0
rx37_csum_unnecessary: 0
rx37_csum_unnecessary_inner: 0
rx37_csum_none: 0
rx37_xdp_drop: 0
rx37_xdp_redirect: 0
rx37_lro_packets: 0
rx37_lro_bytes: 0
rx37_ecn_mark: 0
rx37_removed_vlan_packets: 0
rx37_wqe_err: 0
rx37_mpwqe_filler_cqes: 0
rx37_mpwqe_filler_strides: 0
rx37_buff_alloc_err: 0
rx37_cqe_compress_blks: 0
rx37_cqe_compress_pkts: 0
rx37_page_reuse: 0
rx37_cache_reuse: 0
rx37_cache_full: 0
rx37_cache_empty: 2560
rx37_cache_busy: 0
rx37_cache_waive: 0
rx37_congst_umr: 0
rx37_arfs_err: 0
rx37_xdp_tx_xmit: 0
rx37_xdp_tx_full: 0
rx37_xdp_tx_err: 0
rx37_xdp_tx_cqes: 0
rx38_packets: 0
rx38_bytes: 0
rx38_csum_complete: 0
rx38_csum_unnecessary: 0
rx38_csum_unnecessary_inner: 0
rx38_csum_none: 0
rx38_xdp_drop: 0
rx38_xdp_redirect: 0
rx38_lro_packets: 0
rx38_lro_bytes: 0
rx38_ecn_mark: 0
rx38_removed_vlan_packets: 0
rx38_wqe_err: 0
rx38_mpwqe_filler_cqes: 0
rx38_mpwqe_filler_strides: 0
rx38_buff_alloc_err: 0
rx38_cqe_compress_blks: 0
rx38_cqe_compress_pkts: 0
rx38_page_reuse: 0
rx38_cache_reuse: 0
rx38_cache_full: 0
rx38_cache_empty: 2560
rx38_cache_busy: 0
rx38_cache_waive: 0
rx38_congst_umr: 0
rx38_arfs_err: 0
rx38_xdp_tx_xmit: 0
rx38_xdp_tx_full: 0
rx38_xdp_tx_err: 0
rx38_xdp_tx_cqes: 0
rx39_packets: 0
rx39_bytes: 0
rx39_csum_complete: 0
rx39_csum_unnecessary: 0
rx39_csum_unnecessary_inner: 0
rx39_csum_none: 0
rx39_xdp_drop: 0
rx39_xdp_redirect: 0
rx39_lro_packets: 0
rx39_lro_bytes: 0
rx39_ecn_mark: 0
rx39_removed_vlan_packets: 0
rx39_wqe_err: 0
rx39_mpwqe_filler_cqes: 0
rx39_mpwqe_filler_strides: 0
rx39_buff_alloc_err: 0
rx39_cqe_compress_blks: 0
rx39_cqe_compress_pkts: 0
rx39_page_reuse: 0
rx39_cache_reuse: 0
rx39_cache_full: 0
rx39_cache_empty: 2560
rx39_cache_busy: 0
rx39_cache_waive: 0
rx39_congst_umr: 0
rx39_arfs_err: 0
rx39_xdp_tx_xmit: 0
rx39_xdp_tx_full: 0
rx39_xdp_tx_err: 0
rx39_xdp_tx_cqes: 0
rx40_packets: 0
rx40_bytes: 0
rx40_csum_complete: 0
rx40_csum_unnecessary: 0
rx40_csum_unnecessary_inner: 0
rx40_csum_none: 0
rx40_xdp_drop: 0
rx40_xdp_redirect: 0
rx40_lro_packets: 0
rx40_lro_bytes: 0
rx40_ecn_mark: 0
rx40_removed_vlan_packets: 0
rx40_wqe_err: 0
rx40_mpwqe_filler_cqes: 0
rx40_mpwqe_filler_strides: 0
rx40_buff_alloc_err: 0
rx40_cqe_compress_blks: 0
rx40_cqe_compress_pkts: 0
rx40_page_reuse: 0
rx40_cache_reuse: 0
rx40_cache_full: 0
rx40_cache_empty: 2560
rx40_cache_busy: 0
rx40_cache_waive: 0
rx40_congst_umr: 0
rx40_arfs_err: 0
rx40_xdp_tx_xmit: 0
rx40_xdp_tx_full: 0
rx40_xdp_tx_err: 0
rx40_xdp_tx_cqes: 0
rx41_packets: 0
rx41_bytes: 0
rx41_csum_complete: 0
rx41_csum_unnecessary: 0
rx41_csum_unnecessary_inner: 0
rx41_csum_none: 0
rx41_xdp_drop: 0
rx41_xdp_redirect: 0
rx41_lro_packets: 0
rx41_lro_bytes: 0
rx41_ecn_mark: 0
rx41_removed_vlan_packets: 0
rx41_wqe_err: 0
rx41_mpwqe_filler_cqes: 0
rx41_mpwqe_filler_strides: 0
rx41_buff_alloc_err: 0
rx41_cqe_compress_blks: 0
rx41_cqe_compress_pkts: 0
rx41_page_reuse: 0
rx41_cache_reuse: 0
rx41_cache_full: 0
rx41_cache_empty: 2560
rx41_cache_busy: 0
rx41_cache_waive: 0
rx41_congst_umr: 0
rx41_arfs_err: 0
rx41_xdp_tx_xmit: 0
rx41_xdp_tx_full: 0
rx41_xdp_tx_err: 0
rx41_xdp_tx_cqes: 0
rx42_packets: 0
rx42_bytes: 0
rx42_csum_complete: 0
rx42_csum_unnecessary: 0
rx42_csum_unnecessary_inner: 0
rx42_csum_none: 0
rx42_xdp_drop: 0
rx42_xdp_redirect: 0
rx42_lro_packets: 0
rx42_lro_bytes: 0
rx42_ecn_mark: 0
rx42_removed_vlan_packets: 0
rx42_wqe_err: 0
rx42_mpwqe_filler_cqes: 0
rx42_mpwqe_filler_strides: 0
rx42_buff_alloc_err: 0
rx42_cqe_compress_blks: 0
rx42_cqe_compress_pkts: 0
rx42_page_reuse: 0
rx42_cache_reuse: 0
rx42_cache_full: 0
rx42_cache_empty: 2560
rx42_cache_busy: 0
rx42_cache_waive: 0
rx42_congst_umr: 0
rx42_arfs_err: 0
rx42_xdp_tx_xmit: 0
rx42_xdp_tx_full: 0
rx42_xdp_tx_err: 0
rx42_xdp_tx_cqes: 0
rx43_packets: 0
rx43_bytes: 0
rx43_csum_complete: 0
rx43_csum_unnecessary: 0
rx43_csum_unnecessary_inner: 0
rx43_csum_none: 0
rx43_xdp_drop: 0
rx43_xdp_redirect: 0
rx43_lro_packets: 0
rx43_lro_bytes: 0
rx43_ecn_mark: 0
rx43_removed_vlan_packets: 0
rx43_wqe_err: 0
rx43_mpwqe_filler_cqes: 0
rx43_mpwqe_filler_strides: 0
rx43_buff_alloc_err: 0
rx43_cqe_compress_blks: 0
rx43_cqe_compress_pkts: 0
rx43_page_reuse: 0
rx43_cache_reuse: 0
rx43_cache_full: 0
rx43_cache_empty: 2560
rx43_cache_busy: 0
rx43_cache_waive: 0
rx43_congst_umr: 0
rx43_arfs_err: 0
rx43_xdp_tx_xmit: 0
rx43_xdp_tx_full: 0
rx43_xdp_tx_err: 0
rx43_xdp_tx_cqes: 0
rx44_packets: 0
rx44_bytes: 0
rx44_csum_complete: 0
rx44_csum_unnecessary: 0
rx44_csum_unnecessary_inner: 0
rx44_csum_none: 0
rx44_xdp_drop: 0
rx44_xdp_redirect: 0
rx44_lro_packets: 0
rx44_lro_bytes: 0
rx44_ecn_mark: 0
rx44_removed_vlan_packets: 0
rx44_wqe_err: 0
rx44_mpwqe_filler_cqes: 0
rx44_mpwqe_filler_strides: 0
rx44_buff_alloc_err: 0
rx44_cqe_compress_blks: 0
rx44_cqe_compress_pkts: 0
rx44_page_reuse: 0
rx44_cache_reuse: 0
rx44_cache_full: 0
rx44_cache_empty: 2560
rx44_cache_busy: 0
rx44_cache_waive: 0
rx44_congst_umr: 0
rx44_arfs_err: 0
rx44_xdp_tx_xmit: 0
rx44_xdp_tx_full: 0
rx44_xdp_tx_err: 0
rx44_xdp_tx_cqes: 0
rx45_packets: 0
rx45_bytes: 0
rx45_csum_complete: 0
rx45_csum_unnecessary: 0
rx45_csum_unnecessary_inner: 0
rx45_csum_none: 0
rx45_xdp_drop: 0
rx45_xdp_redirect: 0
rx45_lro_packets: 0
rx45_lro_bytes: 0
rx45_ecn_mark: 0
rx45_removed_vlan_packets: 0
rx45_wqe_err: 0
rx45_mpwqe_filler_cqes: 0
rx45_mpwqe_filler_strides: 0
rx45_buff_alloc_err: 0
rx45_cqe_compress_blks: 0
rx45_cqe_compress_pkts: 0
rx45_page_reuse: 0
rx45_cache_reuse: 0
rx45_cache_full: 0
rx45_cache_empty: 2560
rx45_cache_busy: 0
rx45_cache_waive: 0
rx45_congst_umr: 0
rx45_arfs_err: 0
rx45_xdp_tx_xmit: 0
rx45_xdp_tx_full: 0
rx45_xdp_tx_err: 0
rx45_xdp_tx_cqes: 0
rx46_packets: 0
rx46_bytes: 0
rx46_csum_complete: 0
rx46_csum_unnecessary: 0
rx46_csum_unnecessary_inner: 0
rx46_csum_none: 0
rx46_xdp_drop: 0
rx46_xdp_redirect: 0
rx46_lro_packets: 0
rx46_lro_bytes: 0
rx46_ecn_mark: 0
rx46_removed_vlan_packets: 0
rx46_wqe_err: 0
rx46_mpwqe_filler_cqes: 0
rx46_mpwqe_filler_strides: 0
rx46_buff_alloc_err: 0
rx46_cqe_compress_blks: 0
rx46_cqe_compress_pkts: 0
rx46_page_reuse: 0
rx46_cache_reuse: 0
rx46_cache_full: 0
rx46_cache_empty: 2560
rx46_cache_busy: 0
rx46_cache_waive: 0
rx46_congst_umr: 0
rx46_arfs_err: 0
rx46_xdp_tx_xmit: 0
rx46_xdp_tx_full: 0
rx46_xdp_tx_err: 0
rx46_xdp_tx_cqes: 0
rx47_packets: 0
rx47_bytes: 0
rx47_csum_complete: 0
rx47_csum_unnecessary: 0
rx47_csum_unnecessary_inner: 0
rx47_csum_none: 0
rx47_xdp_drop: 0
rx47_xdp_redirect: 0
rx47_lro_packets: 0
rx47_lro_bytes: 0
rx47_ecn_mark: 0
rx47_removed_vlan_packets: 0
rx47_wqe_err: 0
rx47_mpwqe_filler_cqes: 0
rx47_mpwqe_filler_strides: 0
rx47_buff_alloc_err: 0
rx47_cqe_compress_blks: 0
rx47_cqe_compress_pkts: 0
rx47_page_reuse: 0
rx47_cache_reuse: 0
rx47_cache_full: 0
rx47_cache_empty: 2560
rx47_cache_busy: 0
rx47_cache_waive: 0
rx47_congst_umr: 0
rx47_arfs_err: 0
rx47_xdp_tx_xmit: 0
rx47_xdp_tx_full: 0
rx47_xdp_tx_err: 0
rx47_xdp_tx_cqes: 0
rx48_packets: 0
rx48_bytes: 0
rx48_csum_complete: 0
rx48_csum_unnecessary: 0
rx48_csum_unnecessary_inner: 0
rx48_csum_none: 0
rx48_xdp_drop: 0
rx48_xdp_redirect: 0
rx48_lro_packets: 0
rx48_lro_bytes: 0
rx48_ecn_mark: 0
rx48_removed_vlan_packets: 0
rx48_wqe_err: 0
rx48_mpwqe_filler_cqes: 0
rx48_mpwqe_filler_strides: 0
rx48_buff_alloc_err: 0
rx48_cqe_compress_blks: 0
rx48_cqe_compress_pkts: 0
rx48_page_reuse: 0
rx48_cache_reuse: 0
rx48_cache_full: 0
rx48_cache_empty: 2560
rx48_cache_busy: 0
rx48_cache_waive: 0
rx48_congst_umr: 0
rx48_arfs_err: 0
rx48_xdp_tx_xmit: 0
rx48_xdp_tx_full: 0
rx48_xdp_tx_err: 0
rx48_xdp_tx_cqes: 0
rx49_packets: 0
rx49_bytes: 0
rx49_csum_complete: 0
rx49_csum_unnecessary: 0
rx49_csum_unnecessary_inner: 0
rx49_csum_none: 0
rx49_xdp_drop: 0
rx49_xdp_redirect: 0
rx49_lro_packets: 0
rx49_lro_bytes: 0
rx49_ecn_mark: 0
rx49_removed_vlan_packets: 0
rx49_wqe_err: 0
rx49_mpwqe_filler_cqes: 0
rx49_mpwqe_filler_strides: 0
rx49_buff_alloc_err: 0
rx49_cqe_compress_blks: 0
rx49_cqe_compress_pkts: 0
rx49_page_reuse: 0
rx49_cache_reuse: 0
rx49_cache_full: 0
rx49_cache_empty: 2560
rx49_cache_busy: 0
rx49_cache_waive: 0
rx49_congst_umr: 0
rx49_arfs_err: 0
rx49_xdp_tx_xmit: 0
rx49_xdp_tx_full: 0
rx49_xdp_tx_err: 0
rx49_xdp_tx_cqes: 0
rx50_packets: 0
rx50_bytes: 0
rx50_csum_complete: 0
rx50_csum_unnecessary: 0
rx50_csum_unnecessary_inner: 0
rx50_csum_none: 0
rx50_xdp_drop: 0
rx50_xdp_redirect: 0
rx50_lro_packets: 0
rx50_lro_bytes: 0
rx50_ecn_mark: 0
rx50_removed_vlan_packets: 0
rx50_wqe_err: 0
rx50_mpwqe_filler_cqes: 0
rx50_mpwqe_filler_strides: 0
rx50_buff_alloc_err: 0
rx50_cqe_compress_blks: 0
rx50_cqe_compress_pkts: 0
rx50_page_reuse: 0
rx50_cache_reuse: 0
rx50_cache_full: 0
rx50_cache_empty: 2560
rx50_cache_busy: 0
rx50_cache_waive: 0
rx50_congst_umr: 0
rx50_arfs_err: 0
rx50_xdp_tx_xmit: 0
rx50_xdp_tx_full: 0
rx50_xdp_tx_err: 0
rx50_xdp_tx_cqes: 0
rx51_packets: 0
rx51_bytes: 0
rx51_csum_complete: 0
rx51_csum_unnecessary: 0
rx51_csum_unnecessary_inner: 0
rx51_csum_none: 0
rx51_xdp_drop: 0
rx51_xdp_redirect: 0
rx51_lro_packets: 0
rx51_lro_bytes: 0
rx51_ecn_mark: 0
rx51_removed_vlan_packets: 0
rx51_wqe_err: 0
rx51_mpwqe_filler_cqes: 0
rx51_mpwqe_filler_strides: 0
rx51_buff_alloc_err: 0
rx51_cqe_compress_blks: 0
rx51_cqe_compress_pkts: 0
rx51_page_reuse: 0
rx51_cache_reuse: 0
rx51_cache_full: 0
rx51_cache_empty: 2560
rx51_cache_busy: 0
rx51_cache_waive: 0
rx51_congst_umr: 0
rx51_arfs_err: 0
rx51_xdp_tx_xmit: 0
rx51_xdp_tx_full: 0
rx51_xdp_tx_err: 0
rx51_xdp_tx_cqes: 0
rx52_packets: 0
rx52_bytes: 0
rx52_csum_complete: 0
rx52_csum_unnecessary: 0
rx52_csum_unnecessary_inner: 0
rx52_csum_none: 0
rx52_xdp_drop: 0
rx52_xdp_redirect: 0
rx52_lro_packets: 0
rx52_lro_bytes: 0
rx52_ecn_mark: 0
rx52_removed_vlan_packets: 0
rx52_wqe_err: 0
rx52_mpwqe_filler_cqes: 0
rx52_mpwqe_filler_strides: 0
rx52_buff_alloc_err: 0
rx52_cqe_compress_blks: 0
rx52_cqe_compress_pkts: 0
rx52_page_reuse: 0
rx52_cache_reuse: 0
rx52_cache_full: 0
rx52_cache_empty: 2560
rx52_cache_busy: 0
rx52_cache_waive: 0
rx52_congst_umr: 0
rx52_arfs_err: 0
rx52_xdp_tx_xmit: 0
rx52_xdp_tx_full: 0
rx52_xdp_tx_err: 0
rx52_xdp_tx_cqes: 0
rx53_packets: 0
rx53_bytes: 0
rx53_csum_complete: 0
rx53_csum_unnecessary: 0
rx53_csum_unnecessary_inner: 0
rx53_csum_none: 0
rx53_xdp_drop: 0
rx53_xdp_redirect: 0
rx53_lro_packets: 0
rx53_lro_bytes: 0
rx53_ecn_mark: 0
rx53_removed_vlan_packets: 0
rx53_wqe_err: 0
rx53_mpwqe_filler_cqes: 0
rx53_mpwqe_filler_strides: 0
rx53_buff_alloc_err: 0
rx53_cqe_compress_blks: 0
rx53_cqe_compress_pkts: 0
rx53_page_reuse: 0
rx53_cache_reuse: 0
rx53_cache_full: 0
rx53_cache_empty: 2560
rx53_cache_busy: 0
rx53_cache_waive: 0
rx53_congst_umr: 0
rx53_arfs_err: 0
rx53_xdp_tx_xmit: 0
rx53_xdp_tx_full: 0
rx53_xdp_tx_err: 0
rx53_xdp_tx_cqes: 0
rx54_packets: 0
rx54_bytes: 0
rx54_csum_complete: 0
rx54_csum_unnecessary: 0
rx54_csum_unnecessary_inner: 0
rx54_csum_none: 0
rx54_xdp_drop: 0
rx54_xdp_redirect: 0
rx54_lro_packets: 0
rx54_lro_bytes: 0
rx54_ecn_mark: 0
rx54_removed_vlan_packets: 0
rx54_wqe_err: 0
rx54_mpwqe_filler_cqes: 0
rx54_mpwqe_filler_strides: 0
rx54_buff_alloc_err: 0
rx54_cqe_compress_blks: 0
rx54_cqe_compress_pkts: 0
rx54_page_reuse: 0
rx54_cache_reuse: 0
rx54_cache_full: 0
rx54_cache_empty: 2560
rx54_cache_busy: 0
rx54_cache_waive: 0
rx54_congst_umr: 0
rx54_arfs_err: 0
rx54_xdp_tx_xmit: 0
rx54_xdp_tx_full: 0
rx54_xdp_tx_err: 0
rx54_xdp_tx_cqes: 0
rx55_packets: 0
rx55_bytes: 0
rx55_csum_complete: 0
rx55_csum_unnecessary: 0
rx55_csum_unnecessary_inner: 0
rx55_csum_none: 0
rx55_xdp_drop: 0
rx55_xdp_redirect: 0
rx55_lro_packets: 0
rx55_lro_bytes: 0
rx55_ecn_mark: 0
rx55_removed_vlan_packets: 0
rx55_wqe_err: 0
rx55_mpwqe_filler_cqes: 0
rx55_mpwqe_filler_strides: 0
rx55_buff_alloc_err: 0
rx55_cqe_compress_blks: 0
rx55_cqe_compress_pkts: 0
rx55_page_reuse: 0
rx55_cache_reuse: 0
rx55_cache_full: 0
rx55_cache_empty: 2560
rx55_cache_busy: 0
rx55_cache_waive: 0
rx55_congst_umr: 0
rx55_arfs_err: 0
rx55_xdp_tx_xmit: 0
rx55_xdp_tx_full: 0
rx55_xdp_tx_err: 0
rx55_xdp_tx_cqes: 0
tx0_packets: 24512439668
tx0_bytes: 15287569052791
tx0_tso_packets: 1536157106
tx0_tso_bytes: 8571753637944
tx0_tso_inner_packets: 0
tx0_tso_inner_bytes: 0
tx0_csum_partial: 2132156117
tx0_csum_partial_inner: 0
tx0_added_vlan_packets: 19906601448
tx0_nop: 308098536
tx0_csum_none: 17774445331
tx0_stopped: 19625
tx0_dropped: 0
tx0_xmit_more: 67864870
tx0_recover: 0
tx0_cqes: 19838744246
tx0_wake: 19624
tx0_cqe_err: 0
tx1_packets: 22598557053
tx1_bytes: 13568850145010
tx1_tso_packets: 1369529475
tx1_tso_bytes: 7661777265382
tx1_tso_inner_packets: 0
tx1_tso_inner_bytes: 0
tx1_csum_partial: 1884639496
tx1_csum_partial_inner: 0
tx1_added_vlan_packets: 18468333696
tx1_nop: 281301783
tx1_csum_none: 16583694200
tx1_stopped: 19457
tx1_dropped: 0
tx1_xmit_more: 55170875
tx1_recover: 0
tx1_cqes: 18413169824
tx1_wake: 19455
tx1_cqe_err: 0
tx2_packets: 22821611433
tx2_bytes: 13752535163683
tx2_tso_packets: 1396978825
tx2_tso_bytes: 7774704508463
tx2_tso_inner_packets: 0
tx2_tso_inner_bytes: 0
tx2_csum_partial: 1897834558
tx2_csum_partial_inner: 0
tx2_added_vlan_packets: 18641958085
tx2_nop: 286934891
tx2_csum_none: 16744123527
tx2_stopped: 13214
tx2_dropped: 0
tx2_xmit_more: 61749446
tx2_recover: 0
tx2_cqes: 18580215654
tx2_wake: 13214
tx2_cqe_err: 0
tx3_packets: 22580809948
tx3_bytes: 13730542936609
tx3_tso_packets: 1370434579
tx3_tso_bytes: 7605636711455
tx3_tso_inner_packets: 0
tx3_tso_inner_bytes: 0
tx3_csum_partial: 1865573748
tx3_csum_partial_inner: 0
tx3_added_vlan_packets: 18491873644
tx3_nop: 281195875
tx3_csum_none: 16626299896
tx3_stopped: 12542
tx3_dropped: 0
tx3_xmit_more: 57681647
tx3_recover: 0
tx3_cqes: 18434198757
tx3_wake: 12540
tx3_cqe_err: 0
tx4_packets: 27801801208
tx4_bytes: 17058453171137
tx4_tso_packets: 1740500105
tx4_tso_bytes: 9474905622036
tx4_tso_inner_packets: 0
tx4_tso_inner_bytes: 0
tx4_csum_partial: 2279225376
tx4_csum_partial_inner: 0
tx4_added_vlan_packets: 22744081633
tx4_nop: 349753979
tx4_csum_none: 20464856257
tx4_stopped: 14816
tx4_dropped: 0
tx4_xmit_more: 65469322
tx4_recover: 0
tx4_cqes: 22678618972
tx4_wake: 14816
tx4_cqe_err: 0
tx5_packets: 25099783024
tx5_bytes: 14917740698381
tx5_tso_packets: 1512988013
tx5_tso_bytes: 8571208921023
tx5_tso_inner_packets: 0
tx5_tso_inner_bytes: 0
tx5_csum_partial: 2078498561
tx5_csum_partial_inner: 0
tx5_added_vlan_packets: 20465533760
tx5_nop: 312614719
tx5_csum_none: 18387035199
tx5_stopped: 4605
tx5_dropped: 0
tx5_xmit_more: 64188936
tx5_recover: 0
tx5_cqes: 20401350718
tx5_wake: 4604
tx5_cqe_err: 0
tx6_packets: 25025504896
tx6_bytes: 14908021946070
tx6_tso_packets: 1515718977
tx6_tso_bytes: 8511442522461
tx6_tso_inner_packets: 0
tx6_tso_inner_bytes: 0
tx6_csum_partial: 2056378610
tx6_csum_partial_inner: 0
tx6_added_vlan_packets: 20434066400
tx6_nop: 310594020
tx6_csum_none: 18377687790
tx6_stopped: 15234
tx6_dropped: 0
tx6_xmit_more: 61130422
tx6_recover: 0
tx6_cqes: 20372943611
tx6_wake: 15234
tx6_cqe_err: 0
tx7_packets: 25457096169
tx7_bytes: 15456289446172
tx7_tso_packets: 1553342799
tx7_tso_bytes: 8764550988105
tx7_tso_inner_packets: 0
tx7_tso_inner_bytes: 0
tx7_csum_partial: 2105765233
tx7_csum_partial_inner: 0
tx7_added_vlan_packets: 20720382377
tx7_nop: 319044853
tx7_csum_none: 18614617145
tx7_stopped: 18745
tx7_dropped: 0
tx7_xmit_more: 57050107
tx7_recover: 0
tx7_cqes: 20663340775
tx7_wake: 18746
tx7_cqe_err: 0
tx8_packets: 25389771649
tx8_bytes: 15225503883962
tx8_tso_packets: 1563367648
tx8_tso_bytes: 8710384514258
tx8_tso_inner_packets: 0
tx8_tso_inner_bytes: 0
tx8_csum_partial: 2106586634
tx8_csum_partial_inner: 0
tx8_added_vlan_packets: 20704676274
tx8_nop: 318149261
tx8_csum_none: 18598089640
tx8_stopped: 4733
tx8_dropped: 0
tx8_xmit_more: 61014317
tx8_recover: 0
tx8_cqes: 20643667301
tx8_wake: 4735
tx8_cqe_err: 0
tx9_packets: 25521500166
tx9_bytes: 15302293145755
tx9_tso_packets: 1546316697
tx9_tso_bytes: 8770688145926
tx9_tso_inner_packets: 0
tx9_tso_inner_bytes: 0
tx9_csum_partial: 2097652880
tx9_csum_partial_inner: 0
tx9_added_vlan_packets: 20778408432
tx9_nop: 318538543
tx9_csum_none: 18680755556
tx9_stopped: 16118
tx9_dropped: 0
tx9_xmit_more: 68509728
tx9_recover: 0
tx9_cqes: 20709906498
tx9_wake: 16118
tx9_cqe_err: 0
tx10_packets: 25451605829
tx10_bytes: 15386896170792
tx10_tso_packets: 1576473520
tx10_tso_bytes: 8880888676383
tx10_tso_inner_packets: 0
tx10_tso_inner_bytes: 0
tx10_csum_partial: 2129796141
tx10_csum_partial_inner: 0
tx10_added_vlan_packets: 20659622590
tx10_nop: 319117433
tx10_csum_none: 18529826450
tx10_stopped: 20187
tx10_dropped: 0
tx10_xmit_more: 58892184
tx10_recover: 0
tx10_cqes: 20600737739
tx10_wake: 20188
tx10_cqe_err: 0
tx11_packets: 27008919793
tx11_bytes: 16587719213058
tx11_tso_packets: 1734884654
tx11_tso_bytes: 9475681471870
tx11_tso_inner_packets: 0
tx11_tso_inner_bytes: 0
tx11_csum_partial: 2296162292
tx11_csum_partial_inner: 0
tx11_added_vlan_packets: 21943096263
tx11_nop: 344188182
tx11_csum_none: 19646933971
tx11_stopped: 9703
tx11_dropped: 0
tx11_xmit_more: 66530718
tx11_recover: 0
tx11_cqes: 21876571667
tx11_wake: 9704
tx11_cqe_err: 0
tx12_packets: 25969493269
tx12_bytes: 15980767963416
tx12_tso_packets: 1671396456
tx12_tso_bytes: 9268973672821
tx12_tso_inner_packets: 0
tx12_tso_inner_bytes: 0
tx12_csum_partial: 2243809182
tx12_csum_partial_inner: 0
tx12_added_vlan_packets: 20980642456
tx12_nop: 330241007
tx12_csum_none: 18736833276
tx12_stopped: 10341
tx12_dropped: 0
tx12_xmit_more: 57834100
tx12_recover: 0
tx12_cqes: 20922815079
tx12_wake: 10342
tx12_cqe_err: 0
tx13_packets: 25332762261
tx13_bytes: 15353213283280
tx13_tso_packets: 1577433599
tx13_tso_bytes: 8785240284281
tx13_tso_inner_packets: 0
tx13_tso_inner_bytes: 0
tx13_csum_partial: 2110640515
tx13_csum_partial_inner: 0
tx13_added_vlan_packets: 20605670910
tx13_nop: 319805741
tx13_csum_none: 18495030395
tx13_stopped: 7006
tx13_dropped: 0
tx13_xmit_more: 58314402
tx13_recover: 0
tx13_cqes: 20547362770
tx13_wake: 7008
tx13_cqe_err: 0
tx14_packets: 26333743548
tx14_bytes: 16070719060573
tx14_tso_packets: 1677922970
tx14_tso_bytes: 9240299765487
tx14_tso_inner_packets: 0
tx14_tso_inner_bytes: 0
tx14_csum_partial: 2215668906
tx14_csum_partial_inner: 0
tx14_added_vlan_packets: 21384410786
tx14_nop: 332734939
tx14_csum_none: 19168741880
tx14_stopped: 13160
tx14_dropped: 0
tx14_xmit_more: 57650391
tx14_recover: 0
tx14_cqes: 21326767783
tx14_wake: 13161
tx14_cqe_err: 0
tx15_packets: 26824968971
tx15_bytes: 16687994233452
tx15_tso_packets: 1755745052
tx15_tso_bytes: 9533814012441
tx15_tso_inner_packets: 0
tx15_tso_inner_bytes: 0
tx15_csum_partial: 2304778064
tx15_csum_partial_inner: 0
tx15_added_vlan_packets: 21740906107
tx15_nop: 344143287
tx15_csum_none: 19436128058
tx15_stopped: 75
tx15_dropped: 0
tx15_xmit_more: 63325832
tx15_recover: 0
tx15_cqes: 21677585345
tx15_wake: 74
tx15_cqe_err: 0
tx16_packets: 24488158946
tx16_bytes: 15027415004570
tx16_tso_packets: 1559127391
tx16_tso_bytes: 8658691917845
tx16_tso_inner_packets: 0
tx16_tso_inner_bytes: 0
tx16_csum_partial: 2075856395
tx16_csum_partial_inner: 0
tx16_added_vlan_packets: 19835695731
tx16_nop: 308464189
tx16_csum_none: 17759839340
tx16_stopped: 4567
tx16_dropped: 0
tx16_xmit_more: 62631422
tx16_recover: 0
tx16_cqes: 19773070012
tx16_wake: 4568
tx16_cqe_err: 0
tx17_packets: 24700413784
tx17_bytes: 15216529713715
tx17_tso_packets: 1597555108
tx17_tso_bytes: 8773728661243
tx17_tso_inner_packets: 0
tx17_tso_inner_bytes: 0
tx17_csum_partial: 2127177297
tx17_csum_partial_inner: 0
tx17_added_vlan_packets: 20003144561
tx17_nop: 313356918
tx17_csum_none: 17875967264
tx17_stopped: 12572
tx17_dropped: 0
tx17_xmit_more: 62742980
tx17_recover: 0
tx17_cqes: 19940407615
tx17_wake: 12573
tx17_cqe_err: 0
tx18_packets: 24887710046
tx18_bytes: 15245034034664
tx18_tso_packets: 1582550520
tx18_tso_bytes: 8782692335483
tx18_tso_inner_packets: 0
tx18_tso_inner_bytes: 0
tx18_csum_partial: 2084514331
tx18_csum_partial_inner: 0
tx18_added_vlan_packets: 20173879181
tx18_nop: 314818702
tx18_csum_none: 18089364850
tx18_stopped: 21366
tx18_dropped: 0
tx18_xmit_more: 62485819
tx18_recover: 0
tx18_cqes: 20111400935
tx18_wake: 21366
tx18_cqe_err: 0
tx19_packets: 24831057648
tx19_bytes: 15164663890576
tx19_tso_packets: 1599135489
tx19_tso_bytes: 8756045449746
tx19_tso_inner_packets: 0
tx19_tso_inner_bytes: 0
tx19_csum_partial: 2119746608
tx19_csum_partial_inner: 0
tx19_added_vlan_packets: 20143573903
tx19_nop: 316966450
tx19_csum_none: 18023827295
tx19_stopped: 11431
tx19_dropped: 0
tx19_xmit_more: 57535904
tx19_recover: 0
tx19_cqes: 20086045325
tx19_wake: 11431
tx19_cqe_err: 0
tx20_packets: 21943735263
tx20_bytes: 13528749492187
tx20_tso_packets: 1390048103
tx20_tso_bytes: 7629058809637
tx20_tso_inner_packets: 0
tx20_tso_inner_bytes: 0
tx20_csum_partial: 1848533941
tx20_csum_partial_inner: 0
tx20_added_vlan_packets: 17861417651
tx20_nop: 276840365
tx20_csum_none: 16012883710
tx20_stopped: 38457
tx20_dropped: 0
tx20_xmit_more: 57042753
tx20_recover: 0
tx20_cqes: 17804384839
tx20_wake: 38457
tx20_cqe_err: 0
tx21_packets: 21476926958
tx21_bytes: 13096410597896
tx21_tso_packets: 1367724090
tx21_tso_bytes: 7568364585127
tx21_tso_inner_packets: 0
tx21_tso_inner_bytes: 0
tx21_csum_partial: 1830570727
tx21_csum_partial_inner: 0
tx21_added_vlan_packets: 17421087814
tx21_nop: 270611519
tx21_csum_none: 15590517087
tx21_stopped: 31213
tx21_dropped: 0
tx21_xmit_more: 60305389
tx21_recover: 0
tx21_cqes: 17360791205
tx21_wake: 31213
tx21_cqe_err: 0
tx22_packets: 21819106444
tx22_bytes: 13492871887100
tx22_tso_packets: 1387002018
tx22_tso_bytes: 7617705727669
tx22_tso_inner_packets: 0
tx22_tso_inner_bytes: 0
tx22_csum_partial: 1853632107
tx22_csum_partial_inner: 0
tx22_added_vlan_packets: 17743255447
tx22_nop: 274820992
tx22_csum_none: 15889623340
tx22_stopped: 24814
tx22_dropped: 0
tx22_xmit_more: 60811304
tx22_recover: 0
tx22_cqes: 17682451111
tx22_wake: 24815
tx22_cqe_err: 0
tx23_packets: 21830455800
tx23_bytes: 13427551902532
tx23_tso_packets: 1388556038
tx23_tso_bytes: 7604040587125
tx23_tso_inner_packets: 0
tx23_tso_inner_bytes: 0
tx23_csum_partial: 1850819694
tx23_csum_partial_inner: 0
tx23_added_vlan_packets: 17761271122
tx23_nop: 275142775
tx23_csum_none: 15910451428
tx23_stopped: 29899
tx23_dropped: 0
tx23_xmit_more: 58924909
tx23_recover: 0
tx23_cqes: 17702355187
tx23_wake: 29898
tx23_cqe_err: 0
tx24_packets: 21961484213
tx24_bytes: 13531373062497
tx24_tso_packets: 1394697504
tx24_tso_bytes: 7663866609308
tx24_tso_inner_packets: 0
tx24_tso_inner_bytes: 0
tx24_csum_partial: 1857072074
tx24_csum_partial_inner: 0
tx24_added_vlan_packets: 17856887568
tx24_nop: 276352855
tx24_csum_none: 15999815494
tx24_stopped: 33924
tx24_dropped: 0
tx24_xmit_more: 63992426
tx24_recover: 0
tx24_cqes: 17792905243
tx24_wake: 33923
tx24_cqe_err: 0
tx25_packets: 21853593838
tx25_bytes: 13357487830519
tx25_tso_packets: 1398822411
tx25_tso_bytes: 7691191518838
tx25_tso_inner_packets: 0
tx25_tso_inner_bytes: 0
tx25_csum_partial: 1869483109
tx25_csum_partial_inner: 0
tx25_added_vlan_packets: 17734634614
tx25_nop: 276327643
tx25_csum_none: 15865151505
tx25_stopped: 38651
tx25_dropped: 0
tx25_xmit_more: 56410535
tx25_recover: 0
tx25_cqes: 17678234537
tx25_wake: 38650
tx25_cqe_err: 0
tx26_packets: 21480261205
tx26_bytes: 13148973015935
tx26_tso_packets: 1348132284
tx26_tso_bytes: 7523489481775
tx26_tso_inner_packets: 0
tx26_tso_inner_bytes: 0
tx26_csum_partial: 1839740745
tx26_csum_partial_inner: 0
tx26_added_vlan_packets: 17430592911
tx26_nop: 270367836
tx26_csum_none: 15590852166
tx26_stopped: 34044
tx26_dropped: 0
tx26_xmit_more: 59870114
tx26_recover: 0
tx26_cqes: 17370736612
tx26_wake: 34043
tx26_cqe_err: 0
tx27_packets: 22694273108
tx27_bytes: 14135473431004
tx27_tso_packets: 1418371875
tx27_tso_bytes: 7784842263038
tx27_tso_inner_packets: 0
tx27_tso_inner_bytes: 0
tx27_csum_partial: 1919170584
tx27_csum_partial_inner: 0
tx27_added_vlan_packets: 18520826023
tx27_nop: 286296272
tx27_csum_none: 16601655439
tx27_stopped: 38125
tx27_dropped: 0
tx27_xmit_more: 72749775
tx27_recover: 0
tx27_cqes: 18448090270
tx27_wake: 38127
tx27_cqe_err: 0
tx28_packets: 0
tx28_bytes: 0
tx28_tso_packets: 0
tx28_tso_bytes: 0
tx28_tso_inner_packets: 0
tx28_tso_inner_bytes: 0
tx28_csum_partial: 0
tx28_csum_partial_inner: 0
tx28_added_vlan_packets: 0
tx28_nop: 0
tx28_csum_none: 0
tx28_stopped: 0
tx28_dropped: 0
tx28_xmit_more: 0
tx28_recover: 0
tx28_cqes: 0
tx28_wake: 0
tx28_cqe_err: 0
tx29_packets: 3
tx29_bytes: 266
tx29_tso_packets: 0
tx29_tso_bytes: 0
tx29_tso_inner_packets: 0
tx29_tso_inner_bytes: 0
tx29_csum_partial: 0
tx29_csum_partial_inner: 0
tx29_added_vlan_packets: 0
tx29_nop: 0
tx29_csum_none: 3
tx29_stopped: 0
tx29_dropped: 0
tx29_xmit_more: 1
tx29_recover: 0
tx29_cqes: 2
tx29_wake: 0
tx29_cqe_err: 0
tx30_packets: 0
tx30_bytes: 0
tx30_tso_packets: 0
tx30_tso_bytes: 0
tx30_tso_inner_packets: 0
tx30_tso_inner_bytes: 0
tx30_csum_partial: 0
tx30_csum_partial_inner: 0
tx30_added_vlan_packets: 0
tx30_nop: 0
tx30_csum_none: 0
tx30_stopped: 0
tx30_dropped: 0
tx30_xmit_more: 0
tx30_recover: 0
tx30_cqes: 0
tx30_wake: 0
tx30_cqe_err: 0
tx31_packets: 0
tx31_bytes: 0
tx31_tso_packets: 0
tx31_tso_bytes: 0
tx31_tso_inner_packets: 0
tx31_tso_inner_bytes: 0
tx31_csum_partial: 0
tx31_csum_partial_inner: 0
tx31_added_vlan_packets: 0
tx31_nop: 0
tx31_csum_none: 0
tx31_stopped: 0
tx31_dropped: 0
tx31_xmit_more: 0
tx31_recover: 0
tx31_cqes: 0
tx31_wake: 0
tx31_cqe_err: 0
tx32_packets: 0
tx32_bytes: 0
tx32_tso_packets: 0
tx32_tso_bytes: 0
tx32_tso_inner_packets: 0
tx32_tso_inner_bytes: 0
tx32_csum_partial: 0
tx32_csum_partial_inner: 0
tx32_added_vlan_packets: 0
tx32_nop: 0
tx32_csum_none: 0
tx32_stopped: 0
tx32_dropped: 0
tx32_xmit_more: 0
tx32_recover: 0
tx32_cqes: 0
tx32_wake: 0
tx32_cqe_err: 0
tx33_packets: 0
tx33_bytes: 0
tx33_tso_packets: 0
tx33_tso_bytes: 0
tx33_tso_inner_packets: 0
tx33_tso_inner_bytes: 0
tx33_csum_partial: 0
tx33_csum_partial_inner: 0
tx33_added_vlan_packets: 0
tx33_nop: 0
tx33_csum_none: 0
tx33_stopped: 0
tx33_dropped: 0
tx33_xmit_more: 0
tx33_recover: 0
tx33_cqes: 0
tx33_wake: 0
tx33_cqe_err: 0
tx34_packets: 0
tx34_bytes: 0
tx34_tso_packets: 0
tx34_tso_bytes: 0
tx34_tso_inner_packets: 0
tx34_tso_inner_bytes: 0
tx34_csum_partial: 0
tx34_csum_partial_inner: 0
tx34_added_vlan_packets: 0
tx34_nop: 0
tx34_csum_none: 0
tx34_stopped: 0
tx34_dropped: 0
tx34_xmit_more: 0
tx34_recover: 0
tx34_cqes: 0
tx34_wake: 0
tx34_cqe_err: 0
tx35_packets: 0
tx35_bytes: 0
tx35_tso_packets: 0
tx35_tso_bytes: 0
tx35_tso_inner_packets: 0
tx35_tso_inner_bytes: 0
tx35_csum_partial: 0
tx35_csum_partial_inner: 0
tx35_added_vlan_packets: 0
tx35_nop: 0
tx35_csum_none: 0
tx35_stopped: 0
tx35_dropped: 0
tx35_xmit_more: 0
tx35_recover: 0
tx35_cqes: 0
tx35_wake: 0
tx35_cqe_err: 0
tx36_packets: 0
tx36_bytes: 0
tx36_tso_packets: 0
tx36_tso_bytes: 0
tx36_tso_inner_packets: 0
tx36_tso_inner_bytes: 0
tx36_csum_partial: 0
tx36_csum_partial_inner: 0
tx36_added_vlan_packets: 0
tx36_nop: 0
tx36_csum_none: 0
tx36_stopped: 0
tx36_dropped: 0
tx36_xmit_more: 0
tx36_recover: 0
tx36_cqes: 0
tx36_wake: 0
tx36_cqe_err: 0
tx37_packets: 0
tx37_bytes: 0
tx37_tso_packets: 0
tx37_tso_bytes: 0
tx37_tso_inner_packets: 0
tx37_tso_inner_bytes: 0
tx37_csum_partial: 0
tx37_csum_partial_inner: 0
tx37_added_vlan_packets: 0
tx37_nop: 0
tx37_csum_none: 0
tx37_stopped: 0
tx37_dropped: 0
tx37_xmit_more: 0
tx37_recover: 0
tx37_cqes: 0
tx37_wake: 0
tx37_cqe_err: 0
tx38_packets: 0
tx38_bytes: 0
tx38_tso_packets: 0
tx38_tso_bytes: 0
tx38_tso_inner_packets: 0
tx38_tso_inner_bytes: 0
tx38_csum_partial: 0
tx38_csum_partial_inner: 0
tx38_added_vlan_packets: 0
tx38_nop: 0
tx38_csum_none: 0
tx38_stopped: 0
tx38_dropped: 0
tx38_xmit_more: 0
tx38_recover: 0
tx38_cqes: 0
tx38_wake: 0
tx38_cqe_err: 0
tx39_packets: 0
tx39_bytes: 0
tx39_tso_packets: 0
tx39_tso_bytes: 0
tx39_tso_inner_packets: 0
tx39_tso_inner_bytes: 0
tx39_csum_partial: 0
tx39_csum_partial_inner: 0
tx39_added_vlan_packets: 0
tx39_nop: 0
tx39_csum_none: 0
tx39_stopped: 0
tx39_dropped: 0
tx39_xmit_more: 0
tx39_recover: 0
tx39_cqes: 0
tx39_wake: 0
tx39_cqe_err: 0
tx40_packets: 0
tx40_bytes: 0
tx40_tso_packets: 0
tx40_tso_bytes: 0
tx40_tso_inner_packets: 0
tx40_tso_inner_bytes: 0
tx40_csum_partial: 0
tx40_csum_partial_inner: 0
tx40_added_vlan_packets: 0
tx40_nop: 0
tx40_csum_none: 0
tx40_stopped: 0
tx40_dropped: 0
tx40_xmit_more: 0
tx40_recover: 0
tx40_cqes: 0
tx40_wake: 0
tx40_cqe_err: 0
tx41_packets: 0
tx41_bytes: 0
tx41_tso_packets: 0
tx41_tso_bytes: 0
tx41_tso_inner_packets: 0
tx41_tso_inner_bytes: 0
tx41_csum_partial: 0
tx41_csum_partial_inner: 0
tx41_added_vlan_packets: 0
tx41_nop: 0
tx41_csum_none: 0
tx41_stopped: 0
tx41_dropped: 0
tx41_xmit_more: 0
tx41_recover: 0
tx41_cqes: 0
tx41_wake: 0
tx41_cqe_err: 0
tx42_packets: 0
tx42_bytes: 0
tx42_tso_packets: 0
tx42_tso_bytes: 0
tx42_tso_inner_packets: 0
tx42_tso_inner_bytes: 0
tx42_csum_partial: 0
tx42_csum_partial_inner: 0
tx42_added_vlan_packets: 0
tx42_nop: 0
tx42_csum_none: 0
tx42_stopped: 0
tx42_dropped: 0
tx42_xmit_more: 0
tx42_recover: 0
tx42_cqes: 0
tx42_wake: 0
tx42_cqe_err: 0
tx43_packets: 0
tx43_bytes: 0
tx43_tso_packets: 0
tx43_tso_bytes: 0
tx43_tso_inner_packets: 0
tx43_tso_inner_bytes: 0
tx43_csum_partial: 0
tx43_csum_partial_inner: 0
tx43_added_vlan_packets: 0
tx43_nop: 0
tx43_csum_none: 0
tx43_stopped: 0
tx43_dropped: 0
tx43_xmit_more: 0
tx43_recover: 0
tx43_cqes: 0
tx43_wake: 0
tx43_cqe_err: 0
tx44_packets: 0
tx44_bytes: 0
tx44_tso_packets: 0
tx44_tso_bytes: 0
tx44_tso_inner_packets: 0
tx44_tso_inner_bytes: 0
tx44_csum_partial: 0
tx44_csum_partial_inner: 0
tx44_added_vlan_packets: 0
tx44_nop: 0
tx44_csum_none: 0
tx44_stopped: 0
tx44_dropped: 0
tx44_xmit_more: 0
tx44_recover: 0
tx44_cqes: 0
tx44_wake: 0
tx44_cqe_err: 0
tx45_packets: 0
tx45_bytes: 0
tx45_tso_packets: 0
tx45_tso_bytes: 0
tx45_tso_inner_packets: 0
tx45_tso_inner_bytes: 0
tx45_csum_partial: 0
tx45_csum_partial_inner: 0
tx45_added_vlan_packets: 0
tx45_nop: 0
tx45_csum_none: 0
tx45_stopped: 0
tx45_dropped: 0
tx45_xmit_more: 0
tx45_recover: 0
tx45_cqes: 0
tx45_wake: 0
tx45_cqe_err: 0
tx46_packets: 0
tx46_bytes: 0
tx46_tso_packets: 0
tx46_tso_bytes: 0
tx46_tso_inner_packets: 0
tx46_tso_inner_bytes: 0
tx46_csum_partial: 0
tx46_csum_partial_inner: 0
tx46_added_vlan_packets: 0
tx46_nop: 0
tx46_csum_none: 0
tx46_stopped: 0
tx46_dropped: 0
tx46_xmit_more: 0
tx46_recover: 0
tx46_cqes: 0
tx46_wake: 0
tx46_cqe_err: 0
tx47_packets: 0
tx47_bytes: 0
tx47_tso_packets: 0
tx47_tso_bytes: 0
tx47_tso_inner_packets: 0
tx47_tso_inner_bytes: 0
tx47_csum_partial: 0
tx47_csum_partial_inner: 0
tx47_added_vlan_packets: 0
tx47_nop: 0
tx47_csum_none: 0
tx47_stopped: 0
tx47_dropped: 0
tx47_xmit_more: 0
tx47_recover: 0
tx47_cqes: 0
tx47_wake: 0
tx47_cqe_err: 0
tx48_packets: 0
tx48_bytes: 0
tx48_tso_packets: 0
tx48_tso_bytes: 0
tx48_tso_inner_packets: 0
tx48_tso_inner_bytes: 0
tx48_csum_partial: 0
tx48_csum_partial_inner: 0
tx48_added_vlan_packets: 0
tx48_nop: 0
tx48_csum_none: 0
tx48_stopped: 0
tx48_dropped: 0
tx48_xmit_more: 0
tx48_recover: 0
tx48_cqes: 0
tx48_wake: 0
tx48_cqe_err: 0
tx49_packets: 0
tx49_bytes: 0
tx49_tso_packets: 0
tx49_tso_bytes: 0
tx49_tso_inner_packets: 0
tx49_tso_inner_bytes: 0
tx49_csum_partial: 0
tx49_csum_partial_inner: 0
tx49_added_vlan_packets: 0
tx49_nop: 0
tx49_csum_none: 0
tx49_stopped: 0
tx49_dropped: 0
tx49_xmit_more: 0
tx49_recover: 0
tx49_cqes: 0
tx49_wake: 0
tx49_cqe_err: 0
tx50_packets: 0
tx50_bytes: 0
tx50_tso_packets: 0
tx50_tso_bytes: 0
tx50_tso_inner_packets: 0
tx50_tso_inner_bytes: 0
tx50_csum_partial: 0
tx50_csum_partial_inner: 0
tx50_added_vlan_packets: 0
tx50_nop: 0
tx50_csum_none: 0
tx50_stopped: 0
tx50_dropped: 0
tx50_xmit_more: 0
tx50_recover: 0
tx50_cqes: 0
tx50_wake: 0
tx50_cqe_err: 0
tx51_packets: 0
tx51_bytes: 0
tx51_tso_packets: 0
tx51_tso_bytes: 0
tx51_tso_inner_packets: 0
tx51_tso_inner_bytes: 0
tx51_csum_partial: 0
tx51_csum_partial_inner: 0
tx51_added_vlan_packets: 0
tx51_nop: 0
tx51_csum_none: 0
tx51_stopped: 0
tx51_dropped: 0
tx51_xmit_more: 0
tx51_recover: 0
tx51_cqes: 0
tx51_wake: 0
tx51_cqe_err: 0
tx52_packets: 0
tx52_bytes: 0
tx52_tso_packets: 0
tx52_tso_bytes: 0
tx52_tso_inner_packets: 0
tx52_tso_inner_bytes: 0
tx52_csum_partial: 0
tx52_csum_partial_inner: 0
tx52_added_vlan_packets: 0
tx52_nop: 0
tx52_csum_none: 0
tx52_stopped: 0
tx52_dropped: 0
tx52_xmit_more: 0
tx52_recover: 0
tx52_cqes: 0
tx52_wake: 0
tx52_cqe_err: 0
tx53_packets: 0
tx53_bytes: 0
tx53_tso_packets: 0
tx53_tso_bytes: 0
tx53_tso_inner_packets: 0
tx53_tso_inner_bytes: 0
tx53_csum_partial: 0
tx53_csum_partial_inner: 0
tx53_added_vlan_packets: 0
tx53_nop: 0
tx53_csum_none: 0
tx53_stopped: 0
tx53_dropped: 0
tx53_xmit_more: 0
tx53_recover: 0
tx53_cqes: 0
tx53_wake: 0
tx53_cqe_err: 0
tx54_packets: 0
tx54_bytes: 0
tx54_tso_packets: 0
tx54_tso_bytes: 0
tx54_tso_inner_packets: 0
tx54_tso_inner_bytes: 0
tx54_csum_partial: 0
tx54_csum_partial_inner: 0
tx54_added_vlan_packets: 0
tx54_nop: 0
tx54_csum_none: 0
tx54_stopped: 0
tx54_dropped: 0
tx54_xmit_more: 0
tx54_recover: 0
tx54_cqes: 0
tx54_wake: 0
tx54_cqe_err: 0
tx55_packets: 0
tx55_bytes: 0
tx55_tso_packets: 0
tx55_tso_bytes: 0
tx55_tso_inner_packets: 0
tx55_tso_inner_bytes: 0
tx55_csum_partial: 0
tx55_csum_partial_inner: 0
tx55_added_vlan_packets: 0
tx55_nop: 0
tx55_csum_none: 0
tx55_stopped: 0
tx55_dropped: 0
tx55_xmit_more: 0
tx55_recover: 0
tx55_cqes: 0
tx55_wake: 0
tx55_cqe_err: 0
tx0_xdp_xmit: 0
tx0_xdp_full: 0
tx0_xdp_err: 0
tx0_xdp_cqes: 0
tx1_xdp_xmit: 0
tx1_xdp_full: 0
tx1_xdp_err: 0
tx1_xdp_cqes: 0
tx2_xdp_xmit: 0
tx2_xdp_full: 0
tx2_xdp_err: 0
tx2_xdp_cqes: 0
tx3_xdp_xmit: 0
tx3_xdp_full: 0
tx3_xdp_err: 0
tx3_xdp_cqes: 0
tx4_xdp_xmit: 0
tx4_xdp_full: 0
tx4_xdp_err: 0
tx4_xdp_cqes: 0
tx5_xdp_xmit: 0
tx5_xdp_full: 0
tx5_xdp_err: 0
tx5_xdp_cqes: 0
tx6_xdp_xmit: 0
tx6_xdp_full: 0
tx6_xdp_err: 0
tx6_xdp_cqes: 0
tx7_xdp_xmit: 0
tx7_xdp_full: 0
tx7_xdp_err: 0
tx7_xdp_cqes: 0
tx8_xdp_xmit: 0
tx8_xdp_full: 0
tx8_xdp_err: 0
tx8_xdp_cqes: 0
tx9_xdp_xmit: 0
tx9_xdp_full: 0
tx9_xdp_err: 0
tx9_xdp_cqes: 0
tx10_xdp_xmit: 0
tx10_xdp_full: 0
tx10_xdp_err: 0
tx10_xdp_cqes: 0
tx11_xdp_xmit: 0
tx11_xdp_full: 0
tx11_xdp_err: 0
tx11_xdp_cqes: 0
tx12_xdp_xmit: 0
tx12_xdp_full: 0
tx12_xdp_err: 0
tx12_xdp_cqes: 0
tx13_xdp_xmit: 0
tx13_xdp_full: 0
tx13_xdp_err: 0
tx13_xdp_cqes: 0
tx14_xdp_xmit: 0
tx14_xdp_full: 0
tx14_xdp_err: 0
tx14_xdp_cqes: 0
tx15_xdp_xmit: 0
tx15_xdp_full: 0
tx15_xdp_err: 0
tx15_xdp_cqes: 0
tx16_xdp_xmit: 0
tx16_xdp_full: 0
tx16_xdp_err: 0
tx16_xdp_cqes: 0
tx17_xdp_xmit: 0
tx17_xdp_full: 0
tx17_xdp_err: 0
tx17_xdp_cqes: 0
tx18_xdp_xmit: 0
tx18_xdp_full: 0
tx18_xdp_err: 0
tx18_xdp_cqes: 0
tx19_xdp_xmit: 0
tx19_xdp_full: 0
tx19_xdp_err: 0
tx19_xdp_cqes: 0
tx20_xdp_xmit: 0
tx20_xdp_full: 0
tx20_xdp_err: 0
tx20_xdp_cqes: 0
tx21_xdp_xmit: 0
tx21_xdp_full: 0
tx21_xdp_err: 0
tx21_xdp_cqes: 0
tx22_xdp_xmit: 0
tx22_xdp_full: 0
tx22_xdp_err: 0
tx22_xdp_cqes: 0
tx23_xdp_xmit: 0
tx23_xdp_full: 0
tx23_xdp_err: 0
tx23_xdp_cqes: 0
tx24_xdp_xmit: 0
tx24_xdp_full: 0
tx24_xdp_err: 0
tx24_xdp_cqes: 0
tx25_xdp_xmit: 0
tx25_xdp_full: 0
tx25_xdp_err: 0
tx25_xdp_cqes: 0
tx26_xdp_xmit: 0
tx26_xdp_full: 0
tx26_xdp_err: 0
tx26_xdp_cqes: 0
tx27_xdp_xmit: 0
tx27_xdp_full: 0
tx27_xdp_err: 0
tx27_xdp_cqes: 0
tx28_xdp_xmit: 0
tx28_xdp_full: 0
tx28_xdp_err: 0
tx28_xdp_cqes: 0
tx29_xdp_xmit: 0
tx29_xdp_full: 0
tx29_xdp_err: 0
tx29_xdp_cqes: 0
tx30_xdp_xmit: 0
tx30_xdp_full: 0
tx30_xdp_err: 0
tx30_xdp_cqes: 0
tx31_xdp_xmit: 0
tx31_xdp_full: 0
tx31_xdp_err: 0
tx31_xdp_cqes: 0
tx32_xdp_xmit: 0
tx32_xdp_full: 0
tx32_xdp_err: 0
tx32_xdp_cqes: 0
tx33_xdp_xmit: 0
tx33_xdp_full: 0
tx33_xdp_err: 0
tx33_xdp_cqes: 0
tx34_xdp_xmit: 0
tx34_xdp_full: 0
tx34_xdp_err: 0
tx34_xdp_cqes: 0
tx35_xdp_xmit: 0
tx35_xdp_full: 0
tx35_xdp_err: 0
tx35_xdp_cqes: 0
tx36_xdp_xmit: 0
tx36_xdp_full: 0
tx36_xdp_err: 0
tx36_xdp_cqes: 0
tx37_xdp_xmit: 0
tx37_xdp_full: 0
tx37_xdp_err: 0
tx37_xdp_cqes: 0
tx38_xdp_xmit: 0
tx38_xdp_full: 0
tx38_xdp_err: 0
tx38_xdp_cqes: 0
tx39_xdp_xmit: 0
tx39_xdp_full: 0
tx39_xdp_err: 0
tx39_xdp_cqes: 0
tx40_xdp_xmit: 0
tx40_xdp_full: 0
tx40_xdp_err: 0
tx40_xdp_cqes: 0
tx41_xdp_xmit: 0
tx41_xdp_full: 0
tx41_xdp_err: 0
tx41_xdp_cqes: 0
tx42_xdp_xmit: 0
tx42_xdp_full: 0
tx42_xdp_err: 0
tx42_xdp_cqes: 0
tx43_xdp_xmit: 0
tx43_xdp_full: 0
tx43_xdp_err: 0
tx43_xdp_cqes: 0
tx44_xdp_xmit: 0
tx44_xdp_full: 0
tx44_xdp_err: 0
tx44_xdp_cqes: 0
tx45_xdp_xmit: 0
tx45_xdp_full: 0
tx45_xdp_err: 0
tx45_xdp_cqes: 0
tx46_xdp_xmit: 0
tx46_xdp_full: 0
tx46_xdp_err: 0
tx46_xdp_cqes: 0
tx47_xdp_xmit: 0
tx47_xdp_full: 0
tx47_xdp_err: 0
tx47_xdp_cqes: 0
tx48_xdp_xmit: 0
tx48_xdp_full: 0
tx48_xdp_err: 0
tx48_xdp_cqes: 0
tx49_xdp_xmit: 0
tx49_xdp_full: 0
tx49_xdp_err: 0
tx49_xdp_cqes: 0
tx50_xdp_xmit: 0
tx50_xdp_full: 0
tx50_xdp_err: 0
tx50_xdp_cqes: 0
tx51_xdp_xmit: 0
tx51_xdp_full: 0
tx51_xdp_err: 0
tx51_xdp_cqes: 0
tx52_xdp_xmit: 0
tx52_xdp_full: 0
tx52_xdp_err: 0
tx52_xdp_cqes: 0
tx53_xdp_xmit: 0
tx53_xdp_full: 0
tx53_xdp_err: 0
tx53_xdp_cqes: 0
tx54_xdp_xmit: 0
tx54_xdp_full: 0
tx54_xdp_err: 0
tx54_xdp_cqes: 0
tx55_xdp_xmit: 0
tx55_xdp_full: 0
tx55_xdp_err: 0
tx55_xdp_cqes: 0
> [...]
>
>>>> ethtool -S enp175s0f0
>>>> NIC statistics:
>>>> rx_packets: 141574897253
>>>> rx_bytes: 184445040406258
>>>> tx_packets: 172569543894
>>>> tx_bytes: 99486882076365
>>>> tx_tso_packets: 9367664195
>>>> tx_tso_bytes: 56435233992948
>>>> tx_tso_inner_packets: 0
>>>> tx_tso_inner_bytes: 0
>>>> tx_added_vlan_packets: 141297671626
>>>> tx_nop: 2102916272
>>>> rx_lro_packets: 0
>>>> rx_lro_bytes: 0
>>>> rx_ecn_mark: 0
>>>> rx_removed_vlan_packets: 141574897252
>>>> rx_csum_unnecessary: 0
>>>> rx_csum_none: 23135854
>>>> rx_csum_complete: 141551761398
>>>> rx_csum_unnecessary_inner: 0
>>>> rx_xdp_drop: 0
>>>> rx_xdp_redirect: 0
>>>> rx_xdp_tx_xmit: 0
>>>> rx_xdp_tx_full: 0
>>>> rx_xdp_tx_err: 0
>>>> rx_xdp_tx_cqe: 0
>>>> tx_csum_none: 127934791664
>>> It is a good idea to look into this, tx is not requesting hw tx
>>> csumming for a lot of packets, maybe you are wasting a lot of cpu
>>> on
>>> calculating csum, or maybe this is just the rx csum complete..
>>>
>>>> tx_csum_partial: 13362879974
>>>> tx_csum_partial_inner: 0
>>>> tx_queue_stopped: 232561
>>> TX queues are stalling, could be an indentation for the pcie
>>> bottelneck.
>>>
>>>> tx_queue_dropped: 0
>>>> tx_xmit_more: 1266021946
>>>> tx_recover: 0
>>>> tx_cqes: 140031716469
>>>> tx_queue_wake: 232561
>>>> tx_udp_seg_rem: 0
>>>> tx_cqe_err: 0
>>>> tx_xdp_xmit: 0
>>>> tx_xdp_full: 0
>>>> tx_xdp_err: 0
>>>> tx_xdp_cqes: 0
>>>> rx_wqe_err: 0
>>>> rx_mpwqe_filler_cqes: 0
>>>> rx_mpwqe_filler_strides: 0
>>>> rx_buff_alloc_err: 0
>>>> rx_cqe_compress_blks: 0
>>>> rx_cqe_compress_pkts: 0
>>>> rx_page_reuse: 0
>>>> rx_cache_reuse: 16625975793
>>>> rx_cache_full: 54161465914
>>>> rx_cache_empty: 258048
>>>> rx_cache_busy: 54161472735
>>>> rx_cache_waive: 0
>>>> rx_congst_umr: 0
>>>> rx_arfs_err: 0
>>>> ch_events: 40572621887
>>>> ch_poll: 40885650979
>>>> ch_arm: 40429276692
>>>> ch_aff_change: 0
>>>> ch_eq_rearm: 0
>>>> rx_out_of_buffer: 2791690
>>>> rx_if_down_packets: 74
>>>> rx_vport_unicast_packets: 141843476308
>>>> rx_vport_unicast_bytes: 185421265403318
>>>> tx_vport_unicast_packets: 172569484005
>>>> tx_vport_unicast_bytes: 100019940094298
>>>> rx_vport_multicast_packets: 85122935
>>>> rx_vport_multicast_bytes: 5761316431
>>>> tx_vport_multicast_packets: 6452
>>>> tx_vport_multicast_bytes: 643540
>>>> rx_vport_broadcast_packets: 22423624
>>>> rx_vport_broadcast_bytes: 1390127090
>>>> tx_vport_broadcast_packets: 22024
>>>> tx_vport_broadcast_bytes: 1321440
>>>> rx_vport_rdma_unicast_packets: 0
>>>> rx_vport_rdma_unicast_bytes: 0
>>>> tx_vport_rdma_unicast_packets: 0
>>>> tx_vport_rdma_unicast_bytes: 0
>>>> rx_vport_rdma_multicast_packets: 0
>>>> rx_vport_rdma_multicast_bytes: 0
>>>> tx_vport_rdma_multicast_packets: 0
>>>> tx_vport_rdma_multicast_bytes: 0
>>>> tx_packets_phy: 172569501577
>>>> rx_packets_phy: 142871314588
>>>> rx_crc_errors_phy: 0
>>>> tx_bytes_phy: 100710212814151
>>>> rx_bytes_phy: 187209224289564
>>>> tx_multicast_phy: 6452
>>>> tx_broadcast_phy: 22024
>>>> rx_multicast_phy: 85122933
>>>> rx_broadcast_phy: 22423623
>>>> rx_in_range_len_errors_phy: 2
>>>> rx_out_of_range_len_phy: 0
>>>> rx_oversize_pkts_phy: 0
>>>> rx_symbol_err_phy: 0
>>>> tx_mac_control_phy: 0
>>>> rx_mac_control_phy: 0
>>>> rx_unsupported_op_phy: 0
>>>> rx_pause_ctrl_phy: 0
>>>> tx_pause_ctrl_phy: 0
>>>> rx_discards_phy: 920161423
>>> Ok, this port seem to be suffering more, RX is congested, maybe due
>>> to
>>> the pcie bottleneck.
>> Yes this side is receiving more traffic - second port is +10G more tx
>>
> [...]
>
>
>>>> Average: 17 0.00 0.00 16.60 0.00 0.00 52.10
>>>> 0.00 0.00 0.00 31.30
>>>> Average: 18 0.00 0.00 13.90 0.00 0.00 61.20
>>>> 0.00 0.00 0.00 24.90
>>>> Average: 19 0.00 0.00 9.99 0.00 0.00 70.33
>>>> 0.00 0.00 0.00 19.68
>>>> Average: 20 0.00 0.00 9.00 0.00 0.00 73.00
>>>> 0.00 0.00 0.00 18.00
>>>> Average: 21 0.00 0.00 8.70 0.00 0.00 73.90
>>>> 0.00 0.00 0.00 17.40
>>>> Average: 22 0.00 0.00 15.42 0.00 0.00 58.56
>>>> 0.00 0.00 0.00 26.03
>>>> Average: 23 0.00 0.00 10.81 0.00 0.00 71.67
>>>> 0.00 0.00 0.00 17.52
>>>> Average: 24 0.00 0.00 10.00 0.00 0.00 71.80
>>>> 0.00 0.00 0.00 18.20
>>>> Average: 25 0.00 0.00 11.19 0.00 0.00 71.13
>>>> 0.00 0.00 0.00 17.68
>>>> Average: 26 0.00 0.00 11.00 0.00 0.00 70.80
>>>> 0.00 0.00 0.00 18.20
>>>> Average: 27 0.00 0.00 10.01 0.00 0.00 69.57
>>>> 0.00 0.00 0.00 20.42
>>> The numa cores are not at 100% util, you have around 20% of idle on
>>> each one.
>> Yes - no 100% cpu - but the difference between 80% and 100% is like
>> push
>> aditional 1-2Gbit/s
>>
> yes but, it doens't look like the bottleneck is the cpu, although it is
> close to be :)..
>
>>>> Average: 28 0.00 0.00 0.00 0.00 0.00 0.00
>>>> 0.00
>>>> 0.00 0.00 100.00
>>>> Average: 29 0.00 0.00 0.00 0.00 0.00 0.00
>>>> 0.00
>>>> 0.00 0.00 100.00
>>>> Average: 30 0.00 0.00 0.00 0.00 0.00 0.00
>>>> 0.00
>>>> 0.00 0.00 100.00
>>>> Average: 31 0.00 0.00 0.00 0.00 0.00 0.00
>>>> 0.00
>>>> 0.00 0.00 100.00
>>>> Average: 32 0.00 0.00 0.00 0.00 0.00 0.00
>>>> 0.00
>>>> 0.00 0.00 100.00
>>>> Average: 33 0.00 0.00 3.90 0.00 0.00 0.00
>>>> 0.00
>>>> 0.00 0.00 96.10
>>>> Average: 34 0.00 0.00 0.00 0.00 0.00 0.00
>>>> 0.00
>>>> 0.00 0.00 100.00
>>>> Average: 35 0.00 0.00 0.00 0.00 0.00 0.00
>>>> 0.00
>>>> 0.00 0.00 100.00
>>>> Average: 36 0.10 0.00 0.20 0.00 0.00 0.00
>>>> 0.00
>>>> 0.00 0.00 99.70
>>>> Average: 37 0.20 0.00 0.30 0.00 0.00 0.00
>>>> 0.00
>>>> 0.00 0.00 99.50
>>>> Average: 38 0.00 0.00 0.00 0.00 0.00 0.00
>>>> 0.00
>>>> 0.00 0.00 100.00
>>>> Average: 39 0.00 0.00 2.60 0.00 0.00 0.00
>>>> 0.00
>>>> 0.00 0.00 97.40
>>>> Average: 40 0.00 0.00 0.90 0.00 0.00 0.00
>>>> 0.00
>>>> 0.00 0.00 99.10
>>>> Average: 41 0.10 0.00 0.50 0.00 0.00 0.00
>>>> 0.00
>>>> 0.00 0.00 99.40
>>>> Average: 42 0.00 0.00 9.91 0.00 0.00 70.67
>>>> 0.00 0.00 0.00 19.42
>>>> Average: 43 0.00 0.00 15.90 0.00 0.00 57.50
>>>> 0.00 0.00 0.00 26.60
>>>> Average: 44 0.00 0.00 12.20 0.00 0.00 66.20
>>>> 0.00 0.00 0.00 21.60
>>>> Average: 45 0.00 0.00 12.00 0.00 0.00 67.50
>>>> 0.00 0.00 0.00 20.50
>>>> Average: 46 0.00 0.00 12.90 0.00 0.00 65.50
>>>> 0.00 0.00 0.00 21.60
>>>> Average: 47 0.00 0.00 14.59 0.00 0.00 60.84
>>>> 0.00 0.00 0.00 24.58
>>>> Average: 48 0.00 0.00 13.59 0.00 0.00 61.74
>>>> 0.00 0.00 0.00 24.68
>>>> Average: 49 0.00 0.00 18.36 0.00 0.00 53.29
>>>> 0.00 0.00 0.00 28.34
>>>> Average: 50 0.00 0.00 15.32 0.00 0.00 58.86
>>>> 0.00 0.00 0.00 25.83
>>>> Average: 51 0.00 0.00 17.60 0.00 0.00 55.20
>>>> 0.00 0.00 0.00 27.20
>>>> Average: 52 0.00 0.00 15.92 0.00 0.00 56.06
>>>> 0.00 0.00 0.00 28.03
>>>> Average: 53 0.00 0.00 13.00 0.00 0.00 62.30
>>>> 0.00 0.00 0.00 24.70
>>>> Average: 54 0.00 0.00 13.20 0.00 0.00 61.50
>>>> 0.00 0.00 0.00 25.30
>>>> Average: 55 0.00 0.00 14.59 0.00 0.00 58.64
>>>> 0.00 0.00 0.00 26.77
>>>>
>>>>
>>>> ethtool -k enp175s0f0
>>>> Features for enp175s0f0:
>>>> rx-checksumming: on
>>>> tx-checksumming: on
>>>> tx-checksum-ipv4: on
>>>> tx-checksum-ip-generic: off [fixed]
>>>> tx-checksum-ipv6: on
>>>> tx-checksum-fcoe-crc: off [fixed]
>>>> tx-checksum-sctp: off [fixed]
>>>> scatter-gather: on
>>>> tx-scatter-gather: on
>>>> tx-scatter-gather-fraglist: off [fixed]
>>>> tcp-segmentation-offload: on
>>>> tx-tcp-segmentation: on
>>>> tx-tcp-ecn-segmentation: off [fixed]
>>>> tx-tcp-mangleid-segmentation: off
>>>> tx-tcp6-segmentation: on
>>>> udp-fragmentation-offload: off
>>>> generic-segmentation-offload: on
>>>> generic-receive-offload: on
>>>> large-receive-offload: off [fixed]
>>>> rx-vlan-offload: on
>>>> tx-vlan-offload: on
>>>> ntuple-filters: off
>>>> receive-hashing: on
>>>> highdma: on [fixed]
>>>> rx-vlan-filter: on
>>>> vlan-challenged: off [fixed]
>>>> tx-lockless: off [fixed]
>>>> netns-local: off [fixed]
>>>> tx-gso-robust: off [fixed]
>>>> tx-fcoe-segmentation: off [fixed]
>>>> tx-gre-segmentation: on
>>>> tx-gre-csum-segmentation: on
>>>> tx-ipxip4-segmentation: off [fixed]
>>>> tx-ipxip6-segmentation: off [fixed]
>>>> tx-udp_tnl-segmentation: on
>>>> tx-udp_tnl-csum-segmentation: on
>>>> tx-gso-partial: on
>>>> tx-sctp-segmentation: off [fixed]
>>>> tx-esp-segmentation: off [fixed]
>>>> tx-udp-segmentation: on
>>>> fcoe-mtu: off [fixed]
>>>> tx-nocache-copy: off
>>>> loopback: off [fixed]
>>>> rx-fcs: off
>>>> rx-all: off
>>>> tx-vlan-stag-hw-insert: on
>>>> rx-vlan-stag-hw-parse: off [fixed]
>>>> rx-vlan-stag-filter: on [fixed]
>>>> l2-fwd-offload: off [fixed]
>>>> hw-tc-offload: off
>>>> esp-hw-offload: off [fixed]
>>>> esp-tx-csum-hw-offload: off [fixed]
>>>> rx-udp_tunnel-port-offload: on
>>>> tls-hw-tx-offload: off [fixed]
>>>> tls-hw-rx-offload: off [fixed]
>>>> rx-gro-hw: off [fixed]
>>>> tls-hw-record: off [fixed]
>>>>
>>>> ethtool -c enp175s0f0
>>>> Coalesce parameters for enp175s0f0:
>>>> Adaptive RX: off TX: on
>>>> stats-block-usecs: 0
>>>> sample-interval: 0
>>>> pkt-rate-low: 0
>>>> pkt-rate-high: 0
>>>> dmac: 32703
>>>>
>>>> rx-usecs: 256
>>>> rx-frames: 128
>>>> rx-usecs-irq: 0
>>>> rx-frames-irq: 0
>>>>
>>>> tx-usecs: 8
>>>> tx-frames: 128
>>>> tx-usecs-irq: 0
>>>> tx-frames-irq: 0
>>>>
>>>> rx-usecs-low: 0
>>>> rx-frame-low: 0
>>>> tx-usecs-low: 0
>>>> tx-frame-low: 0
>>>>
>>>> rx-usecs-high: 0
>>>> rx-frame-high: 0
>>>> tx-usecs-high: 0
>>>> tx-frame-high: 0
>>>>
>>>> ethtool -g enp175s0f0
>>>> Ring parameters for enp175s0f0:
>>>> Pre-set maximums:
>>>> RX: 8192
>>>> RX Mini: 0
>>>> RX Jumbo: 0
>>>> TX: 8192
>>>> Current hardware settings:
>>>> RX: 4096
>>>> RX Mini: 0
>>>> RX Jumbo: 0
>>>> TX: 4096
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>> Also changed a little coalesce params - and best for this config are:
>> ethtool -c enp175s0f0
>> Coalesce parameters for enp175s0f0:
>> Adaptive RX: off TX: off
>> stats-block-usecs: 0
>> sample-interval: 0
>> pkt-rate-low: 0
>> pkt-rate-high: 0
>> dmac: 32573
>>
>> rx-usecs: 40
>> rx-frames: 128
>> rx-usecs-irq: 0
>> rx-frames-irq: 0
>>
>> tx-usecs: 8
>> tx-frames: 8
>> tx-usecs-irq: 0
>> tx-frames-irq: 0
>>
>> rx-usecs-low: 0
>> rx-frame-low: 0
>> tx-usecs-low: 0
>> tx-frame-low: 0
>>
>> rx-usecs-high: 0
>> rx-frame-high: 0
>> tx-usecs-high: 0
>> tx-frame-high: 0
>>
>>
>> Less drops on RX side - and more pps in overall forwarded.
>>
> how much improvement ? maybe we can improve our adaptive rx coal to be
> efficient for this work load.
>
>
I can prepare more stats with ethtool maybee to compare - but normally
tested with simple icmp forwarded from interface to interface
- before change coalescence params:
adaptive-rx off rx-usecs 384 rx-frames 128
3% loss for icmp
- after change to:
adaptive-rx off rx-usecs 40 rx-frames 128 adaptive-tx off tx-usecs 8
tx-frames 8
2% loss for icmp
But yes - to know better will need to compare rx/tx counters from
ethtool + /proc/net/dev
Was trying to turn on adaptative-tx+rx - but 100% saturation at 43Gbit/s
RX / 43Gbit/s TX
Powered by blists - more mailing lists