netdev - Throughtput regression due to [v2,net,2/2] tcp: fix delayed ACKs for MSS boundary condition

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [thread-next>] [day] [month] [year] [list]

Message-ID: <d653597e-9b1f-4eb7-af36-8d79cc5146b7@esteem.com>
Date: Fri, 14 Jun 2024 06:49:08 -0700
From: Neil Hellfeldt <hellfeldt@...eem.com>
To: netdev@...r.kernel.org
Subject: Throughtput regression due to [v2,net,2/2] tcp: fix delayed ACKs for
 MSS boundary condition

Hi,

So I believe I found a regression due to:
patch: [v2,net,2/2] tcp: fix delayed ACKs for MSS boundary condition
commit: 4720852ed9afb1c5ab84e96135cb5b73d5afde6f

I recently upgraded our production machines from Ubuntu 16.04 all the 
way up to 24.04.

In the process I noticed that iperf3 was no longer able to get the 
throughput that it was able to on 16.04
I found that Ubuntu 22.04 is when it broke. Then I found that Ubuntu's 
kernel version 5.15.0-92 worked
fine and version 5.15.0-93 did not. After that I narrowed it down to the 
patch:

patch: [v2,net,2/2] tcp: fix delayed ACKs for MSS boundary condition
commit: 4720852ed9afb1c5ab84e96135cb5b73d5afde6f

diff --git a/net/ipv4/tcp_input.c b/net/ipv4/tcp_input.c
index 06fe1cf645d5a..8afb0950a6979 100644
--- a/net/ipv4/tcp_input.c
+++ b/net/ipv4/tcp_input.c
@@ -253,6 +253,19 @@  static void tcp_measure_rcv_mss(struct sock *sk, const struct sk_buff *skb)
  		if (unlikely(len > icsk->icsk_ack.rcv_mss +
  				   MAX_TCP_OPTION_SPACE))
  			tcp_gro_dev_warn(sk, skb, len);
+ /* If the skb has a len of exactly 1*MSS and has the PSH bit
+ * set then it is likely the end of an application write. So
+ * more data may not be arriving soon, and yet the data sender
+ * may be waiting for an ACK if cwnd-bound or using TX zero
+ * copy. So we set ICSK_ACK_PUSHED here so that
+ * tcp_cleanup_rbuf() will send an ACK immediately if the app
+ * reads all of the data and is not ping-pong. If len > MSS
+ * then this logic does not matter (and does not hurt) because
+ * tcp_cleanup_rbuf() will always ACK immediately if the app
+ * reads data and there is more than an MSS of unACKed data.
+ */
+ if (TCP_SKB_CB(skb)->tcp_flags & TCPHDR_PSH)
+ icsk->icsk_ack.pending |= ICSK_ACK_PUSHED;
  	} else {
  		/* Otherwise, we make more careful check taking into account,
  		 * that SACKs block is variable.

After I removed the patches I was able to get the expected speeds. I then reverted the patched on most current
version of the kernel for Ubuntu which is 6.8.0-35 and I was able to get the expected speeds again.

The device unit under test is a embedded device with lwip and built in iperf3 server. It has 100mbit network port.
It is also a low data rate wireless radio. The expected rate for the Ethernet is ~52000Kbps avg with the patch applied
it was getting ~38000Kbps. The expected throughput for wireless is ~328kbps with the patch applied were are getting ~292Kbps.
That a ~27% throughput regression for Ethernet and a ~11% for the wireless.

command used iperf3 -c 172.18.8.134 -P 1 -i 1 -f m -t 10 -4 -w 64K -R
Iperf version is 3.0.11 from Ubuntu. The newer version of iperf3 3.16 show the correct speeds but shows a saw tooth plot for the wireless.