| lists.openwall.net | lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC | |
|
Open Source and information security mailing list archives
| ||
|
Date: Wed, 6 Mar 2019 12:45:07 +0100
From: Simon Huelck <simonmail@....de>
To: Jose Abreu <jose.abreu@...opsys.com>,
Sebastian Gottschall <s.gottschall@...media-net.de>,
Jerome Brunet <jbrunet@...libre.com>,
Martin Blumenstingl <martin.blumenstingl@...glemail.com>
Cc: linux-amlogic@...ts.infradead.org, Gpeppe.cavallaro@...com,
alexandre.torgue@...com,
Emiliano Ingrassia <ingrassia@...genesys.com>,
netdev@...r.kernel.org
Subject: Re: stmmac / meson8b-dwmac
Hi,
RX is starving TX i guess ... so the other way around. a transfer was
ongoing from 10.10.11.100 to 10.10.11.1 when i triggered this command on
10.10.11.1
root@...oidc2:~# iperf3 -c 10.10.11.100 -i1
Connecting to host 10.10.11.100, port 5201
[ 5] local 10.10.11.1 port 51652 connected to 10.10.11.100 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 371 KBytes 3.04 Mbits/sec 0 14.3 KBytes
[ 5] 1.00-2.00 sec 331 KBytes 2.71 Mbits/sec 0 22.8 KBytes
[ 5] 2.00-3.00 sec 314 KBytes 2.57 Mbits/sec 0 22.8 KBytes
[ 5] 3.00-4.00 sec 314 KBytes 2.57 Mbits/sec 0 22.8 KBytes
[ 5] 4.00-5.00 sec 314 KBytes 2.57 Mbits/sec 0 22.8 KBytes
[ 5] 5.00-6.00 sec 314 KBytes 2.57 Mbits/sec 0 22.8 KBytes
[ 5] 6.00-7.01 sec 48.0 MBytes 400 Mbits/sec 0 552 KBytes
[ 5] 7.01-8.00 sec 106 MBytes 896 Mbits/sec 0 874 KBytes
[ 5] 8.00-9.01 sec 110 MBytes 917 Mbits/sec 0 874 KBytes
[ 5] 9.01-10.00 sec 98.8 MBytes 833 Mbits/sec 94 637 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 365 MBytes 306 Mbits/sec 94 sender
[ 5] 0.00-10.02 sec 362 MBytes 304 Mbits/sec
receiver
--
root@...oidc2:~# ethtool -S eth0
NIC statistics:
mmc_tx_octetcount_gb: 0
mmc_tx_framecount_gb: 0
mmc_tx_broadcastframe_g: 0
mmc_tx_multicastframe_g: 0
mmc_tx_64_octets_gb: 0
mmc_tx_65_to_127_octets_gb: 0
mmc_tx_128_to_255_octets_gb: 0
mmc_tx_256_to_511_octets_gb: 0
mmc_tx_512_to_1023_octets_gb: 0
mmc_tx_1024_to_max_octets_gb: 0
mmc_tx_unicast_gb: 0
mmc_tx_multicast_gb: 0
mmc_tx_broadcast_gb: 0
mmc_tx_underflow_error: 0
mmc_tx_singlecol_g: 0
mmc_tx_multicol_g: 0
mmc_tx_deferred: 0
mmc_tx_latecol: 0
mmc_tx_exesscol: 0
mmc_tx_carrier_error: 0
mmc_tx_octetcount_g: 0
mmc_tx_framecount_g: 0
mmc_tx_excessdef: 0
mmc_tx_pause_frame: 0
mmc_tx_vlan_frame_g: 0
mmc_rx_framecount_gb: 2039534
mmc_rx_octetcount_gb: 2808094903
mmc_rx_octetcount_g: 2808094903
mmc_rx_broadcastframe_g: 29673
mmc_rx_multicastframe_g: 67
mmc_rx_crc_error: 0
mmc_rx_align_error: 0
mmc_rx_run_error: 0
mmc_rx_jabber_error: 0
mmc_rx_undersize_g: 0
mmc_rx_oversize_g: 0
mmc_rx_64_octets_gb: 0
mmc_rx_65_to_127_octets_gb: 201563
mmc_rx_128_to_255_octets_gb: 656
mmc_rx_256_to_511_octets_gb: 1020
mmc_rx_512_to_1023_octets_gb: 448
mmc_rx_1024_to_max_octets_gb: 1835847
mmc_rx_unicast_g: 2009794
mmc_rx_length_error: 0
mmc_rx_autofrangetype: 0
mmc_rx_pause_frames: 0
mmc_rx_fifo_overflow: 315
mmc_rx_vlan_frames_gb: 2039517
mmc_rx_watchdog_error: 0
mmc_rx_ipc_intr_mask: 1073692671
mmc_rx_ipc_intr: 0
mmc_rx_ipv4_gd: 2010329
mmc_rx_ipv4_hderr: 0
mmc_rx_ipv4_nopay: 0
mmc_rx_ipv4_frag: 0
mmc_rx_ipv4_udsbl: 0
mmc_rx_ipv4_gd_octets: 2760880445
mmc_rx_ipv4_hderr_octets: 0
mmc_rx_ipv4_nopay_octets: 0
mmc_rx_ipv4_frag_octets: 0
mmc_rx_ipv4_udsbl_octets: 0
mmc_rx_ipv6_gd_octets: 10047
mmc_rx_ipv6_hderr_octets: 0
mmc_rx_ipv6_nopay_octets: 0
mmc_rx_ipv6_gd: 22
mmc_rx_ipv6_hderr: 0
mmc_rx_ipv6_nopay: 0
mmc_rx_udp_gd: 7642
mmc_rx_udp_err: 0
mmc_rx_tcp_gd: 2002692
mmc_rx_tcp_err: 0
mmc_rx_icmp_gd: 17
mmc_rx_icmp_err: 0
mmc_rx_udp_gd_octets: 666460
mmc_rx_udp_err_octets: 0
mmc_rx_tcp_gd_octets: 2720015720
mmc_rx_tcp_err_octets: 0
mmc_rx_icmp_gd_octets: 852
mmc_rx_icmp_err_octets: 0
tx_underflow: 0
tx_carrier: 0
tx_losscarrier: 0
vlan_tag: 2039202
tx_deferred: 0
tx_vlan: 1114175
tx_jabber: 0
tx_frame_flushed: 0
tx_payload_error: 0
tx_ip_header_error: 0
rx_desc: 0
sa_filter_fail: 0
overflow_error: 0
ipc_csum_error: 0
rx_collision: 0
rx_crc_errors: 0
dribbling_bit: 0
rx_length: 0
rx_mii: 0
rx_multicast: 0
rx_gmac_overflow: 0
rx_watchdog: 0
da_rx_filter_fail: 0
sa_rx_filter_fail: 0
rx_missed_cntr: 0
rx_overflow_cntr: 0
rx_vlan: 0
tx_undeflow_irq: 0
tx_process_stopped_irq: 0
tx_jabber_irq: 0
rx_overflow_irq: 0
rx_buf_unav_irq: 0
rx_process_stopped_irq: 0
rx_watchdog_irq: 0
tx_early_irq: 0
fatal_bus_error_irq: 0
rx_early_irq: 29321
threshold: 1
tx_pkt_n: 1114175
rx_pkt_n: 2039219
normal_irq_n: 167679
rx_normal_irq_n: 128772
napi_poll: 184569
tx_normal_irq_n: 42982
tx_clean: 55788
tx_set_ic_bit: 44567
irq_receive_pmt_irq_n: 0
mmc_tx_irq_n: 0
mmc_rx_irq_n: 0
mmc_rx_csum_offload_irq_n: 0
irq_tx_path_in_lpi_mode_n: 15749
irq_tx_path_exit_lpi_mode_n: 15748
irq_rx_path_in_lpi_mode_n: 0
irq_rx_path_exit_lpi_mode_n: 0
phy_eee_wakeup_error_n: 0
ip_hdr_err: 0
ip_payload_err: 0
ip_csum_bypassed: 0
ipv4_pkt_rcvd: 0
ipv6_pkt_rcvd: 0
no_ptp_rx_msg_type_ext: 0
ptp_rx_msg_type_sync: 0
ptp_rx_msg_type_follow_up: 0
ptp_rx_msg_type_delay_req: 0
ptp_rx_msg_type_delay_resp: 0
ptp_rx_msg_type_pdelay_req: 0
ptp_rx_msg_type_pdelay_resp: 0
ptp_rx_msg_type_pdelay_follow_up: 0
ptp_rx_msg_type_announce: 0
ptp_rx_msg_type_management: 0
ptp_rx_msg_pkt_reserved_type: 0
ptp_frame_type: 0
ptp_ver: 0
timestamp_dropped: 0
av_pkt_rcvd: 0
av_tagged_pkt_rcvd: 0
vlan_tag_priority_val: 0
l3_filter_match: 0
l4_filter_match: 0
l3_l4_filter_no_match: 0
irq_pcs_ane_n: 0
irq_pcs_link_n: 0
irq_rgmii_n: 0
mtl_tx_status_fifo_full: 0
mtl_tx_fifo_not_empty: 0
mmtl_fifo_ctrl: 0
mtl_tx_fifo_read_ctrl_write: 0
mtl_tx_fifo_read_ctrl_wait: 0
mtl_tx_fifo_read_ctrl_read: 0
mtl_tx_fifo_read_ctrl_idle: 0
mac_tx_in_pause: 0
mac_tx_frame_ctrl_xfer: 0
mac_tx_frame_ctrl_idle: 0
mac_tx_frame_ctrl_wait: 0
mac_tx_frame_ctrl_pause: 0
mac_gmii_tx_proto_engine: 0
mtl_rx_fifo_fill_level_full: 0
mtl_rx_fifo_fill_above_thresh: 0
mtl_rx_fifo_fill_below_thresh: 0
mtl_rx_fifo_fill_level_empty: 0
mtl_rx_fifo_read_ctrl_flush: 0
mtl_rx_fifo_read_ctrl_read_data: 0
mtl_rx_fifo_read_ctrl_status: 0
mtl_rx_fifo_read_ctrl_idle: 0
mtl_rx_fifo_ctrl_active: 0
mac_rx_frame_ctrl_fifo: 0
mac_gmii_rx_proto_engine: 0
tx_tso_frames: 0
tx_tso_nfrags: 0
regards,
Simon
Am 06.03.2019 um 12:35 schrieb Simon Huelck:
> Hi,
>
> i sorted out some more things:
>
> - i did not activate tcp window scaling , with this , iperf3 is reaching
> 930MBits now, this was related to my firewall and therefore my uplink.
>
> so the remaining topic is ( and im currently testing with next-20190306 )
>
> TX is starving RX and the total bandwith seem to be limited to 930MBits
> instead of something like 930MBits * 2 for duplex.
>
> @Jose, do you have some hints on that ? i saw that you introduced
> patches for that , but somehow RX/TX are not equally sharing the NAPI
> budget. But i wonder why the TX queue and RX queue collide at all,
> shouldnt they be indipendent ?
>
> regards,
> Simon
>> Hi guys,
>>
>>
>> 1. i discovered something strange. when i never configure my external
>> VLAN interface UP ( firewall doesnt start, traffic shaper CAKE doesnt
>> start), my non duplex iperf bandwidth increases from 600MBits to 930.
>>
>> 2. duplex isnt working, TX is totally starving RX, the total bandwidth
>> is 900MBits, whats going on there ?
>>
>> 3. i had a MTU issue ( was set to 1500, but due to VLANs etc 1450 would
>> be better ) but this didnt change performance
>> 4. even when i up eth0.4, then i down eth0.4, then flush iptables and
>> shaper were never added, i drop to 600MBits ....
>>
>> questions:
>> - why is duplex still not working even so the kernel says so ?
>> - why is TX totally starving RX, even so duplex is "on"
>> - when i flush all my iptable rules, and the traffic shaper, still im
>> bond to 600MBits ... very strange, someone got an idea ? upping eth0.4
>> is cutting the performance, even when other VLAN IFs like eth0.3,
>> eth0.2, eth0.5 are up and bridged ( eth0.4 isnt bridged somewhere )
>>
>>
>>
>> my setup:
>>
>> br-dmz 8000.7ef0fd9b157f no eth0.2
>> br-guest 8000.001f1fbbbd60 no wlan0
>> br-iot 8000.7ef0fd9b157f no eth0.5
>> br-lan 8000.001f1fbbbd61 no eth0.3
>> wlan0_1
>> wlan2
>>
>> eth0.4 is my uplink
>>
>> all the bridges are internally , eth0.4 is externally
>>
>>
>> C:\Users\Simon\Downloads\iperf3.6_64bit\iperf3.6_64bit>iperf3.exe -c
>> 10.10.11.1 -i1
>> warning: Ignoring nonsense TCP MSS 0
>> Connecting to host 10.10.11.1, port 5201
>> [ 5] local 10.10.11.100 port 52173 connected to 10.10.11.1 port 5201
>> [ ID] Interval Transfer Bitrate
>> [ 5] 0.00-1.00 sec 384 KBytes 3.14 Mbits/sec
>> [ 5] 1.00-2.00 sec 384 KBytes 3.15 Mbits/sec
>> [ 5] 2.00-3.00 sec 1.12 MBytes 9.44 Mbits/sec
>> [ 5] 3.00-4.00 sec 2.00 MBytes 16.8 Mbits/sec
>> [ 5] 4.00-5.00 sec 2.38 MBytes 19.9 Mbits/sec
>> [ 5] 5.00-6.00 sec 3.12 MBytes 26.2 Mbits/sec
>> [ 5] 6.00-7.00 sec 4.75 MBytes 39.8 Mbits/sec
>> [ 5] 7.00-8.00 sec 68.4 MBytes 574 Mbits/sec
>> [ 5] 8.00-9.00 sec 104 MBytes 875 Mbits/sec
>> [ 5] 9.00-10.00 sec 105 MBytes 881 Mbits/sec
>> - - - - - - - - - - - - - - - - - - - - - - - - -
>> [ ID] Interval Transfer Bitrate
>> [ 5] 0.00-10.00 sec 292 MBytes 245 Mbits/sec sender
>> [ 5] 0.00-10.04 sec 292 MBytes 244 Mbits/sec
>> receiver
>>
>> iperf Done.
>>
>>
>> root@...oidc2:~# iperf3 -c 10.10.11.100 -i1
>> Connecting to host 10.10.11.100, port 5201
>> [ 5] local 10.10.11.1 port 60022 connected to 10.10.11.100 port 5201
>> [ ID] Interval Transfer Bitrate Retr Cwnd
>> [ 5] 0.00-1.00 sec 112 MBytes 941 Mbits/sec 0 417 KBytes
>> [ 5] 1.00-2.00 sec 113 MBytes 945 Mbits/sec 0 487 KBytes
>> [ 5] 2.00-3.00 sec 112 MBytes 940 Mbits/sec 0 487 KBytes
>> [ 5] 3.00-4.00 sec 112 MBytes 941 Mbits/sec 0 512 KBytes
>> [ 5] 4.00-5.01 sec 109 MBytes 907 Mbits/sec 0 543 KBytes
>> [ 5] 5.01-6.01 sec 109 MBytes 911 Mbits/sec 0 543 KBytes
>> [ 5] 6.01-7.01 sec 108 MBytes 902 Mbits/sec 0 543 KBytes
>> [ 5] 7.01-8.01 sec 108 MBytes 905 Mbits/sec 0 543 KBytes
>> [ 5] 8.01-9.00 sec 106 MBytes 895 Mbits/sec 0 543 KBytes
>> [ 5] 9.00-10.00 sec 106 MBytes 891 Mbits/sec 0 543 KBytes
>> - - - - - - - - - - - - - - - - - - - - - - - - -
>> [ ID] Interval Transfer Bitrate Retr
>> [ 5] 0.00-10.00 sec 1.07 GBytes 918 Mbits/sec 0 sender
>> [ 5] 0.00-10.04 sec 1.07 GBytes 915 Mbits/sec
>> receiver
>>
>> --
>>
>>
>> Mar 5 09:46:03 localhost kernel: [ 105.534204] meson8b-dwmac
>> c9410000.ethernet eth0: Link is Up - 1Gbps/Full - flow control off
>>
>> iperf Done.
>> root@...oidc2:~#
>>
>>
>>
>>
>> regards,
>> Simon
>>
>> Am 01.03.2019 um 10:23 schrieb Jose Abreu:
>>> Hi Simon,
>>>
>>> On 2/27/2019 7:02 PM, Simon Huelck wrote:
>>>> Hi,
>>>>
>>>>
>>>> the thing is , that im not a stmmac developer. Yes , maybe i can bissect
>>>> it and yes you are lucky since im a C-developer since a long time for
>>>> embedded systems.
>>>>
>>>> The problem is that i dont understand the structure of stmmac and im not
>>>> aware of any documentation about the driver structure nor the underlying
>>>> ethernet hardware ( even though im used to ethernet hardware in embedded
>>>> environment ). So how shall i recognize the relevant change between
>>>> 4.14.29 and 5.0rc8 ?
>>>>
>>>>
>>>> Is it in the DTS/DTB, wrong hardware description ? Is it in the code ?
>>>> how is the duplex hardware working on this piece ?
>>>>
>>>> I can try to support you the best i can, but i have little chances to
>>>> analyze it myself. At which measurements / counters is it possible to
>>>> see that duplex is fully working ? Why did even the non-duplex
>>>> bandwidth regress from 900MBits to 650 ? Why is that 650 MBits dividing
>>>> up to TX and RX in summary when doing duplex ? Why is TX not starving in
>>>> duplex but RX ?
>>>>
>>>> From my point of view should be the following things given:
>>>> - the non duplex bandwidth should be somewhere around 900MBits , the HW
>>>> is capable of that
>>>> - TX should not influence RX or vice versa in duplex
>>>> - the duplex bandwidth should be 900MBits in both directions ( maybe a
>>>> bit asymetric when buffers in both dirs are not same )
>>>>
>>>> I guess we need some profiling on stmmac and ( at least i need ) more
>>>> knowledge of the hardware and stmmac itself. Can someone point me to the
>>>> driver documentation, describing the functions in the code and the
>>>> structure ? How can i profile stmmac ( usually im using hardware / JTAG
>>>> debuggers at work, but here @home i got nothing like that )
>>>>
>>>> So how do we continue ?
>>> When I said bissect I was meaning GIT Bissect [1]. You shouldn't
>>> need any development background for this. You just have to start
>>> bissect, compile, test and check if commit is good or not.
>>>
>>> I'm not very familiar with this feature but I think you can
>>> bissect pretty fast if you say you just want stmmac commits,
>>> check ("Cutting down bisection by giving more parameters to
>>> bisect start") on previous link ... In your case it would be
>>> stmmac changes, dts, and phy.
>>>
>>> [1] https://git-scm.com/docs/git-bisect
>>>
>>> Thanks,
>>> Jose Miguel Abreu
Powered by blists - more mailing lists