[<prev] [next>] [thread-next>] [day] [month] [year] [list]
Message-ID: <7a01e4c7ddb84292cc284b6664c794b9a6e713a8.1707759574.git.asml.silence@gmail.com>
Date: Tue, 13 Feb 2024 13:33:08 +0000
From: Pavel Begunkov <asml.silence@...il.com>
To: netdev@...r.kernel.org,
edumazet@...gle.com,
davem@...emloft.net,
dsahern@...nel.org,
pabeni@...hat.com,
kuba@...nel.org
Cc: Pavel Begunkov <asml.silence@...il.com>
Subject: [PATCH v2] net: cache for same cpu skb_attempt_defer_free
Optimise skb_attempt_defer_free() executed by the CPU the skb was
allocated on. Instead of __kfree_skb() -> kmem_cache_free() we can
disable softirqs and put the buffer into cpu local caches.
Trying it with a TCP CPU bound ping pong benchmark (i.e. netbench), it
showed a 1% throughput improvement (392.2 -> 396.4 Krps). Cross checking
with profiles, the total CPU share of skb_attempt_defer_free() dropped by
0.6%. Note, I'd expect the win doubled with rx only benchmarks, as the
optimisation is for the receive path, but the test spends >55% of CPU
doing writes.
Signed-off-by: Pavel Begunkov <asml.silence@...il.com>
---
v2: remove in_hardirq()
net/core/skbuff.c | 16 +++++++++++++++-
1 file changed, 15 insertions(+), 1 deletion(-)
diff --git a/net/core/skbuff.c b/net/core/skbuff.c
index 9b790994da0c..f32f358ef1d8 100644
--- a/net/core/skbuff.c
+++ b/net/core/skbuff.c
@@ -6947,6 +6947,20 @@ void __skb_ext_put(struct skb_ext *ext)
EXPORT_SYMBOL(__skb_ext_put);
#endif /* CONFIG_SKB_EXTENSIONS */
+static void kfree_skb_napi_cache(struct sk_buff *skb)
+{
+ /* if SKB is a clone, don't handle this case */
+ if (skb->fclone != SKB_FCLONE_UNAVAILABLE) {
+ __kfree_skb(skb);
+ return;
+ }
+
+ local_bh_disable();
+ skb_release_all(skb, SKB_DROP_REASON_NOT_SPECIFIED, false);
+ napi_skb_cache_put(skb);
+ local_bh_enable();
+}
+
/**
* skb_attempt_defer_free - queue skb for remote freeing
* @skb: buffer
@@ -6965,7 +6979,7 @@ void skb_attempt_defer_free(struct sk_buff *skb)
if (WARN_ON_ONCE(cpu >= nr_cpu_ids) ||
!cpu_online(cpu) ||
cpu == raw_smp_processor_id()) {
-nodefer: __kfree_skb(skb);
+nodefer: kfree_skb_napi_cache(skb);
return;
}
--
2.43.0
Powered by blists - more mailing lists