netdev - [PATCH net-next] tcp: tsq: avoid using a tasklet if possible

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [thread-next>] [day] [month] [year] [list]

Date:   Thu, 30 Mar 2023 19:25:34 +0000
From:   Eric Dumazet <edumazet@...gle.com>
To:     "David S . Miller" <davem@...emloft.net>,
        Jakub Kicinski <kuba@...nel.org>,
        Paolo Abeni <pabeni@...hat.com>
Cc:     netdev@...r.kernel.org, eric.dumazet@...il.com,
        Eric Dumazet <edumazet@...gle.com>
Subject: [PATCH net-next] tcp: tsq: avoid using a tasklet if possible

Instead of always defer TCP Small Queue handling to a tasklet
we can try to lock the socket and immediately call tcp_tsq_handler()

In my experiments on a 100Gbit NIC with FQ qdisc, number of times
tcp_tasklet_func() is fired per second goes from ~70,000 to ~1,000.

This reduces number of ksoftirqd wakeups and thus extra cpu
costs and latencies.

Signed-off-by: Eric Dumazet <edumazet@...gle.com>
---
 net/ipv4/tcp_output.c | 16 ++++++++++++++--
 1 file changed, 14 insertions(+), 2 deletions(-)

diff --git a/net/ipv4/tcp_output.c b/net/ipv4/tcp_output.c
index cfe128b81a010339b486dd6a40b077cee9570d08..470bb840bb6b6fb457f96d5c00fdfd11b414482f 100644
--- a/net/ipv4/tcp_output.c
+++ b/net/ipv4/tcp_output.c
@@ -1023,13 +1023,18 @@ static void tcp_tsq_write(struct sock *sk)
 	}
 }
 
-static void tcp_tsq_handler(struct sock *sk)
+static void tcp_tsq_handler_locked(struct sock *sk)
 {
-	bh_lock_sock(sk);
 	if (!sock_owned_by_user(sk))
 		tcp_tsq_write(sk);
 	else if (!test_and_set_bit(TCP_TSQ_DEFERRED, &sk->sk_tsq_flags))
 		sock_hold(sk);
+}
+
+static void tcp_tsq_handler(struct sock *sk)
+{
+	bh_lock_sock(sk);
+	tcp_tsq_handler_locked(sk);
 	bh_unlock_sock(sk);
 }
 /*
@@ -1165,6 +1170,13 @@ void tcp_wfree(struct sk_buff *skb)
 		nval = (oval & ~TSQF_THROTTLED) | TSQF_QUEUED;
 	} while (!try_cmpxchg(&sk->sk_tsq_flags, &oval, nval));
 
+	/* attempt to grab socket lock. */
+	if (spin_trylock_bh(&sk->sk_lock.slock)) {
+		clear_bit(TSQ_QUEUED, &sk->sk_tsq_flags);
+		tcp_tsq_handler_locked(sk);
+		spin_unlock_bh(&sk->sk_lock.slock);
+		goto out;
+	}
 	/* queue this socket to tasklet queue */
 	local_irq_save(flags);
 	tsq = this_cpu_ptr(&tsq_tasklet);
-- 
2.40.0.348.gf938b09366-goog