| lists.openwall.net | lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC | |
|
Open Source and information security mailing list archives
| ||
|
Message-ID: <915a2418-cc09-902c-e7de-74ab92e064f2@itcare.pl>
Date: Fri, 9 Nov 2018 20:59:45 +0100
From: Paweł Staszewski <pstaszewski@...are.pl>
To: David Ahern <dsahern@...il.com>,
Jesper Dangaard Brouer <brouer@...hat.com>
Cc: netdev <netdev@...r.kernel.org>, Yoel Caspersen <yoel@...knet.dk>
Subject: Re: Kernel 4.19 network performance - forwarding/routing normal users
traffic
W dniu 09.11.2018 o 17:21, David Ahern pisze:
> On 11/9/18 3:20 AM, Paweł Staszewski wrote:
>> I just catch some weird behavior :)
>> All was working fine for about 20k packets
>>
>> Then after xdp start to forward every 10 packets
> Interesting. Any counter showing drops?
nothing that will fit
NIC statistics:
rx_packets: 187041
rx_bytes: 10600954
tx_packets: 40316
tx_bytes: 16526844
tx_tso_packets: 797
tx_tso_bytes: 3876084
tx_tso_inner_packets: 0
tx_tso_inner_bytes: 0
tx_added_vlan_packets: 38391
tx_nop: 2
rx_lro_packets: 0
rx_lro_bytes: 0
rx_ecn_mark: 0
rx_removed_vlan_packets: 187041
rx_csum_unnecessary: 0
rx_csum_none: 150011
rx_csum_complete: 37030
rx_csum_unnecessary_inner: 0
rx_xdp_drop: 0
rx_xdp_redirect: 64893
rx_xdp_tx_xmit: 0
rx_xdp_tx_full: 0
rx_xdp_tx_err: 0
rx_xdp_tx_cqe: 0
tx_csum_none: 2468
tx_csum_partial: 35955
tx_csum_partial_inner: 0
tx_queue_stopped: 0
tx_queue_dropped: 0
tx_xmit_more: 0
tx_recover: 0
tx_cqes: 38423
tx_queue_wake: 0
tx_udp_seg_rem: 0
tx_cqe_err: 0
tx_xdp_xmit: 0
tx_xdp_full: 0
tx_xdp_err: 0
tx_xdp_cqes: 0
rx_wqe_err: 0
rx_mpwqe_filler_cqes: 0
rx_mpwqe_filler_strides: 0
rx_buff_alloc_err: 0
rx_cqe_compress_blks: 0
rx_cqe_compress_pkts: 0
rx_page_reuse: 0
rx_cache_reuse: 186302
rx_cache_full: 0
rx_cache_empty: 666768
rx_cache_busy: 174
rx_cache_waive: 0
rx_congst_umr: 0
rx_arfs_err: 0
ch_events: 249320
ch_poll: 249321
ch_arm: 249001
ch_aff_change: 0
ch_eq_rearm: 0
rx_out_of_buffer: 0
rx_if_down_packets: 57
rx_vport_unicast_packets: 142659
rx_vport_unicast_bytes: 42706914
tx_vport_unicast_packets: 40167
tx_vport_unicast_bytes: 16668096
rx_vport_multicast_packets: 39188170
rx_vport_multicast_bytes: 3466527450
tx_vport_multicast_packets: 58
tx_vport_multicast_bytes: 4556
rx_vport_broadcast_packets: 16343520
rx_vport_broadcast_bytes: 1031334602
tx_vport_broadcast_packets: 91
tx_vport_broadcast_bytes: 5460
rx_vport_rdma_unicast_packets: 0
rx_vport_rdma_unicast_bytes: 0
tx_vport_rdma_unicast_packets: 0
tx_vport_rdma_unicast_bytes: 0
rx_vport_rdma_multicast_packets: 0
rx_vport_rdma_multicast_bytes: 0
tx_vport_rdma_multicast_packets: 0
tx_vport_rdma_multicast_bytes: 0
tx_packets_phy: 40316
rx_packets_phy: 55674361
rx_crc_errors_phy: 0
tx_bytes_phy: 16839376
rx_bytes_phy: 4763267396
tx_multicast_phy: 58
tx_broadcast_phy: 91
rx_multicast_phy: 39188180
rx_broadcast_phy: 16343521
rx_in_range_len_errors_phy: 0
rx_out_of_range_len_phy: 0
rx_oversize_pkts_phy: 0
rx_symbol_err_phy: 0
tx_mac_control_phy: 0
rx_mac_control_phy: 0
rx_unsupported_op_phy: 0
rx_pause_ctrl_phy: 0
tx_pause_ctrl_phy: 0
rx_discards_phy: 1
tx_discards_phy: 0
tx_errors_phy: 0
rx_undersize_pkts_phy: 0
rx_fragments_phy: 0
rx_jabbers_phy: 0
rx_64_bytes_phy: 3792455
rx_65_to_127_bytes_phy: 51821620
rx_128_to_255_bytes_phy: 37669
rx_256_to_511_bytes_phy: 1481
rx_512_to_1023_bytes_phy: 434
rx_1024_to_1518_bytes_phy: 694
rx_1519_to_2047_bytes_phy: 20008
rx_2048_to_4095_bytes_phy: 0
rx_4096_to_8191_bytes_phy: 0
rx_8192_to_10239_bytes_phy: 0
link_down_events_phy: 0
rx_pcs_symbol_err_phy: 0
rx_corrected_bits_phy: 6
rx_err_lane_0_phy: 0
rx_err_lane_1_phy: 0
rx_err_lane_2_phy: 0
rx_err_lane_3_phy: 6
rx_buffer_passed_thres_phy: 0
rx_pci_signal_integrity: 0
tx_pci_signal_integrity: 82
outbound_pci_stalled_rd: 0
outbound_pci_stalled_wr: 0
outbound_pci_stalled_rd_events: 0
outbound_pci_stalled_wr_events: 0
rx_prio0_bytes: 4144920388
rx_prio0_packets: 48310037
tx_prio0_bytes: 16839376
tx_prio0_packets: 40316
rx_prio1_bytes: 481032
rx_prio1_packets: 7074
tx_prio1_bytes: 0
tx_prio1_packets: 0
rx_prio2_bytes: 9074194
rx_prio2_packets: 106207
tx_prio2_bytes: 0
tx_prio2_packets: 0
rx_prio3_bytes: 0
rx_prio3_packets: 0
tx_prio3_bytes: 0
tx_prio3_packets: 0
rx_prio4_bytes: 0
rx_prio4_packets: 0
tx_prio4_bytes: 0
tx_prio4_packets: 0
rx_prio5_bytes: 0
rx_prio5_packets: 0
tx_prio5_bytes: 0
tx_prio5_packets: 0
rx_prio6_bytes: 371961810
rx_prio6_packets: 4006281
tx_prio6_bytes: 0
tx_prio6_packets: 0
rx_prio7_bytes: 236830040
rx_prio7_packets: 3244761
tx_prio7_bytes: 0
tx_prio7_packets: 0
tx_pause_storm_warning_events : 0
tx_pause_storm_error_events: 0
module_unplug: 0
module_bus_stuck: 0
module_high_temp: 0
module_bad_shorted: 0
NIC statistics:
rx_packets: 843
rx_bytes: 58889
tx_packets: 324
tx_bytes: 23324
tx_tso_packets: 0
tx_tso_bytes: 0
tx_tso_inner_packets: 0
tx_tso_inner_bytes: 0
tx_added_vlan_packets: 293
tx_nop: 0
rx_lro_packets: 0
rx_lro_bytes: 0
rx_ecn_mark: 0
rx_removed_vlan_packets: 843
rx_csum_unnecessary: 0
rx_csum_none: 190
rx_csum_complete: 653
rx_csum_unnecessary_inner: 0
rx_xdp_drop: 0
rx_xdp_redirect: 0
rx_xdp_tx_xmit: 0
rx_xdp_tx_full: 0
rx_xdp_tx_err: 0
rx_xdp_tx_cqe: 0
tx_csum_none: 324
tx_csum_partial: 0
tx_csum_partial_inner: 0
tx_queue_stopped: 0
tx_queue_dropped: 0
tx_xmit_more: 1
tx_recover: 0
tx_cqes: 323
tx_queue_wake: 0
tx_udp_seg_rem: 0
tx_cqe_err: 0
tx_xdp_xmit: 64926
tx_xdp_full: 0
tx_xdp_err: 0
tx_xdp_cqes: 47958
rx_wqe_err: 0
rx_mpwqe_filler_cqes: 0
rx_mpwqe_filler_strides: 0
rx_buff_alloc_err: 0
rx_cqe_compress_blks: 0
rx_cqe_compress_pkts: 0
rx_page_reuse: 0
rx_cache_reuse: 648
rx_cache_full: 0
rx_cache_empty: 602112
rx_cache_busy: 0
rx_cache_waive: 0
rx_congst_umr: 0
rx_arfs_err: 0
ch_events: 49628
ch_poll: 49628
ch_arm: 49626
ch_aff_change: 0
ch_eq_rearm: 0
rx_out_of_buffer: 0
rx_if_down_packets: 46
rx_vport_unicast_packets: 5953
rx_vport_unicast_bytes: 4927049
tx_vport_unicast_packets: 65194
tx_vport_unicast_bytes: 31820150
rx_vport_multicast_packets: 37085249
rx_vport_multicast_bytes: 2449620421
tx_vport_multicast_packets: 55
tx_vport_multicast_bytes: 4278
rx_vport_broadcast_packets: 434654
rx_vport_broadcast_bytes: 31881063
tx_vport_broadcast_packets: 1
tx_vport_broadcast_bytes: 60
rx_vport_rdma_unicast_packets: 0
rx_vport_rdma_unicast_bytes: 0
tx_vport_rdma_unicast_packets: 0
tx_vport_rdma_unicast_bytes: 0
rx_vport_rdma_multicast_packets: 0
rx_vport_rdma_multicast_bytes: 0
tx_vport_rdma_multicast_packets: 0
tx_vport_rdma_multicast_bytes: 0
tx_packets_phy: 65250
rx_packets_phy: 37525857
rx_crc_errors_phy: 0
tx_bytes_phy: 32085488
rx_bytes_phy: 2636532027
tx_multicast_phy: 55
tx_broadcast_phy: 1
rx_multicast_phy: 37085250
rx_broadcast_phy: 434654
rx_in_range_len_errors_phy: 0
rx_out_of_range_len_phy: 0
rx_oversize_pkts_phy: 0
rx_symbol_err_phy: 0
tx_mac_control_phy: 0
rx_mac_control_phy: 0
rx_unsupported_op_phy: 0
rx_pause_ctrl_phy: 0
tx_pause_ctrl_phy: 0
rx_discards_phy: 0
tx_discards_phy: 0
tx_errors_phy: 0
rx_undersize_pkts_phy: 0
rx_fragments_phy: 0
rx_jabbers_phy: 0
rx_64_bytes_phy: 63346
rx_65_to_127_bytes_phy: 37434768
rx_128_to_255_bytes_phy: 14088
rx_256_to_511_bytes_phy: 10461
rx_512_to_1023_bytes_phy: 96
rx_1024_to_1518_bytes_phy: 1933
rx_1519_to_2047_bytes_phy: 1165
rx_2048_to_4095_bytes_phy: 0
rx_4096_to_8191_bytes_phy: 0
rx_8192_to_10239_bytes_phy: 0
link_down_events_phy: 0
rx_pcs_symbol_err_phy: 0
rx_corrected_bits_phy: 5
rx_err_lane_0_phy: 1
rx_err_lane_1_phy: 0
rx_err_lane_2_phy: 0
rx_err_lane_3_phy: 4
rx_buffer_passed_thres_phy: 0
rx_pci_signal_integrity: 0
tx_pci_signal_integrity: 82
outbound_pci_stalled_rd: 0
outbound_pci_stalled_wr: 0
outbound_pci_stalled_rd_events: 0
outbound_pci_stalled_wr_events: 0
rx_prio0_bytes: 23157221
rx_prio0_packets: 195789
tx_prio0_bytes: 32085488
tx_prio0_packets: 65250
rx_prio1_bytes: 0
rx_prio1_packets: 0
tx_prio1_bytes: 0
tx_prio1_packets: 0
rx_prio2_bytes: 0
rx_prio2_packets: 0
tx_prio2_bytes: 0
tx_prio2_packets: 0
rx_prio3_bytes: 23397578
rx_prio3_packets: 343182
tx_prio3_bytes: 0
tx_prio3_packets: 0
rx_prio4_bytes: 0
rx_prio4_packets: 0
tx_prio4_bytes: 0
tx_prio4_packets: 0
rx_prio5_bytes: 0
rx_prio5_packets: 0
tx_prio5_bytes: 0
tx_prio5_packets: 0
rx_prio6_bytes: 14643472
rx_prio6_packets: 203589
tx_prio6_bytes: 0
tx_prio6_packets: 0
rx_prio7_bytes: 2575333474
rx_prio7_packets: 36783293
tx_prio7_bytes: 0
tx_prio7_packets: 0
tx_pause_storm_warning_events : 0
tx_pause_storm_error_events: 0
module_unplug: 0
module_bus_stuck: 0
module_high_temp: 0
module_bad_shorted: 0
But wondering if any offloading now can do some things that we dont want
for xdp
currently all offloads are enabled.
ethtool -k enp175s0f0
Features for enp175s0f0:
rx-checksumming: on
tx-checksumming: on
tx-checksum-ipv4: on
tx-checksum-ip-generic: off [fixed]
tx-checksum-ipv6: on
tx-checksum-fcoe-crc: off [fixed]
tx-checksum-sctp: off [fixed]
scatter-gather: on
tx-scatter-gather: on
tx-scatter-gather-fraglist: off [fixed]
tcp-segmentation-offload: on
tx-tcp-segmentation: on
tx-tcp-ecn-segmentation: off [fixed]
tx-tcp-mangleid-segmentation: off
tx-tcp6-segmentation: on
udp-fragmentation-offload: off
generic-segmentation-offload: on
generic-receive-offload: on
large-receive-offload: off [fixed]
rx-vlan-offload: on
tx-vlan-offload: on
ntuple-filters: off
receive-hashing: on
highdma: on [fixed]
rx-vlan-filter: on
vlan-challenged: off [fixed]
tx-lockless: off [fixed]
netns-local: off [fixed]
tx-gso-robust: off [fixed]
tx-fcoe-segmentation: off [fixed]
tx-gre-segmentation: on
tx-gre-csum-segmentation: on
tx-ipxip4-segmentation: off [fixed]
tx-ipxip6-segmentation: off [fixed]
tx-udp_tnl-segmentation: on
tx-udp_tnl-csum-segmentation: on
tx-gso-partial: on
tx-sctp-segmentation: off [fixed]
tx-esp-segmentation: off [fixed]
tx-udp-segmentation: on
fcoe-mtu: off [fixed]
tx-nocache-copy: off
loopback: off [fixed]
rx-fcs: off
rx-all: off
tx-vlan-stag-hw-insert: on
rx-vlan-stag-hw-parse: off [fixed]
rx-vlan-stag-filter: on [fixed]
l2-fwd-offload: off [fixed]
hw-tc-offload: off
esp-hw-offload: off [fixed]
esp-tx-csum-hw-offload: off [fixed]
rx-udp_tunnel-port-offload: on
tls-hw-tx-offload: off [fixed]
tls-hw-rx-offload: off [fixed]
rx-gro-hw: off [fixed]
tls-hw-record: off [fixed]
Also at the time when xdp is forwarding 1/10 frame - same problem is
with local input/output traffic - testing server is also responding to
1/10 icmp request
>
>
>> ping 172.16.0.2 -i 0.1
>> PING 172.16.0.2 (172.16.0.2) 56(84) bytes of data.
>> 64 bytes from 172.16.0.2: icmp_seq=1 ttl=64 time=5.12 ms
>> 64 bytes from 172.16.0.2: icmp_seq=9 ttl=64 time=5.20 ms
>> 64 bytes from 172.16.0.2: icmp_seq=19 ttl=64 time=4.85 ms
>> 64 bytes from 172.16.0.2: icmp_seq=29 ttl=64 time=4.91 ms
>> 64 bytes from 172.16.0.2: icmp_seq=38 ttl=64 time=4.85 ms
>> 64 bytes from 172.16.0.2: icmp_seq=48 ttl=64 time=5.00 ms
>> ^C
>> --- 172.16.0.2 ping statistics ---
>> 55 packets transmitted, 6 received, 89% packet loss, time 5655ms
>> rtt min/avg/max/mdev = 4.850/4.992/5.203/0.145 ms
>>
>>
>> And again after some time back to normal
>>
>> ping 172.16.0.2 -i 0.1
>> PING 172.16.0.2 (172.16.0.2) 56(84) bytes of data.
>> 64 bytes from 172.16.0.2: icmp_seq=1 ttl=64 time=5.02 ms
>> 64 bytes from 172.16.0.2: icmp_seq=2 ttl=64 time=5.06 ms
>> 64 bytes from 172.16.0.2: icmp_seq=3 ttl=64 time=5.19 ms
>> 64 bytes from 172.16.0.2: icmp_seq=4 ttl=64 time=5.07 ms
>> 64 bytes from 172.16.0.2: icmp_seq=5 ttl=64 time=5.08 ms
>> 64 bytes from 172.16.0.2: icmp_seq=6 ttl=64 time=5.14 ms
>> 64 bytes from 172.16.0.2: icmp_seq=7 ttl=64 time=5.08 ms
>> 64 bytes from 172.16.0.2: icmp_seq=8 ttl=64 time=5.17 ms
>> 64 bytes from 172.16.0.2: icmp_seq=9 ttl=64 time=5.04 ms
>> 64 bytes from 172.16.0.2: icmp_seq=10 ttl=64 time=5.10 ms
>> 64 bytes from 172.16.0.2: icmp_seq=11 ttl=64 time=5.11 ms
>> 64 bytes from 172.16.0.2: icmp_seq=12 ttl=64 time=5.13 ms
>> 64 bytes from 172.16.0.2: icmp_seq=13 ttl=64 time=5.12 ms
>> 64 bytes from 172.16.0.2: icmp_seq=14 ttl=64 time=5.15 ms
>> 64 bytes from 172.16.0.2: icmp_seq=15 ttl=64 time=5.13 ms
>> 64 bytes from 172.16.0.2: icmp_seq=16 ttl=64 time=5.04 ms
>> 64 bytes from 172.16.0.2: icmp_seq=17 ttl=64 time=5.12 ms
>> 64 bytes from 172.16.0.2: icmp_seq=18 ttl=64 time=5.07 ms
>> 64 bytes from 172.16.0.2: icmp_seq=19 ttl=64 time=5.06 ms
>> 64 bytes from 172.16.0.2: icmp_seq=20 ttl=64 time=5.12 ms
>> 64 bytes from 172.16.0.2: icmp_seq=21 ttl=64 time=5.21 ms
>> 64 bytes from 172.16.0.2: icmp_seq=22 ttl=64 time=4.98 ms
>> ^C
>> --- 172.16.0.2 ping statistics ---
>> 22 packets transmitted, 22 received, 0% packet loss, time 2105ms
>> rtt min/avg/max/mdev = 4.988/5.104/5.210/0.089 ms
>>
>>
>> I will try to catch this with debug enabled
>>
>>
>>
>>
>>
>> Wondering also - cause xdp will bypass now vlan counters and other stuff
>> like tcpdump
> yes, xdp is before tcpdump based sockets.
>
> And the counters (vlan just being the current example) is another
> problem to be solved. The vlan net_device never sees the packet and you
> can not arbitrarily bump the counters just because the device lookups
> reference them.
Ok.
>> Is there possible to add only counters from xdp for vlans ?
>> This will help me in testing.
> I will take a look today at adding counters that you can dump using
> bpftool. It will be a temporary solution for this xdp program only.
Yes anything that can give me counters to check traffic lvls
>>
>> And also - for non lab scenario there should be possible to sniff
>> sometimes on interface :)
> Yes, sampling is another problem.
>
>
>> Soo wondering if need to attack another xdp program to interface or all
>> this can be done by one
>>
>> I think this is time where i will need to learn more about xdp :)
>>
>>
>
Powered by blists - more mailing lists