Message-ID: <6F20808D-445D-48DA-8481-4466D9A77659@nutanix.com>
Date: Fri, 5 Dec 2025 16:56:24 +0000
From: Jon Kohler <jon@...anix.com>
To: Jesper Dangaard Brouer <hawk@...nel.org>
CC: Sebastian Andrzej Siewior <bigeasy@...utronix.de>,
        Jason Wang
	<jasowang@...hat.com>,
        "netdev@...r.kernel.org" <netdev@...r.kernel.org>,
        Willem de Bruijn <willemdebruijn.kernel@...il.com>,
        Andrew Lunn
	<andrew+netdev@...n.ch>,
        "David S. Miller" <davem@...emloft.net>,
        Eric
 Dumazet <edumazet@...gle.com>, Jakub Kicinski <kuba@...nel.org>,
        Paolo Abeni
	<pabeni@...hat.com>, Alexei Starovoitov <ast@...nel.org>,
        Daniel Borkmann
	<daniel@...earbox.net>,
        John Fastabend <john.fastabend@...il.com>,
        Stanislav
 Fomichev <sdf@...ichev.me>,
        open list <linux-kernel@...r.kernel.org>,
        "bpf@...r.kernel.org" <bpf@...r.kernel.org>,
        Alexander Lobakin
	<aleksander.lobakin@...el.com>
Subject: Re: [PATCH net-next v2 5/9] tun: use bulk NAPI cache allocation in
 tun_xdp_one



> On Dec 5, 2025, at 8:21 AM, Jesper Dangaard Brouer <hawk@...nel.org> wrote:
> 
> 
> 
> On 05/12/2025 08.58, Sebastian Andrzej Siewior wrote:
>> On 2025-12-03 15:35:24 [+0000], Jon Kohler wrote:
>>> Thanks, Sebastian - so if I’m reading this correct, it *is* fine to do
>>> the two following patterns, outside of NAPI:
>>> 
>>>    local_bh_disable();
>>>    skb = napi_build_skb(buf, len);
>>>    local_bh_enable();
>>> 
>>>    local_bh_disable();
>>>    napi_consume_skb(skb, 1);
>>>    local_bh_enable();
>>> 
>>> If so, I wonder if it would be cleaner to have something like
>>>    build_skb_bh(buf, len);
>>> 
>>>    consume_skb_bh(skb, 1);
>>> 
>>> Then have those methods handle the local_bh enable/disable, so that
>>> the toggle was a property of a call, not a requirement of the call?
>> Having budget = 0 would be for non-NAPI users. So passing the 1 is
>> superfluous. You goal seems to be to re-use napi_alloc_cache. Right? And
>> this is better than skb_pool?

Yes, the goal is to avoid the allocation / deallocation overhead, which is
very visible in flamegraphs, and doing it the way I’ve done it here does
benefit the benchmark I used; however, as Jason pointed out, I need to
look more broadly, so I’ll do that for the next time around.

>> There is already napi_alloc_skb() which expects BH to be disabled and
>> netdev_alloc_skb() (and friends) which do disable BH if needed. I don't
>> see an equivalent for non-NAPI users. Haven't checked if any of these
>> could replace your napi_build_skb().
>> Historically non-NAPI users would be IRQ users and those can't do
>> local_bh_disable(). Therefore there is dev_kfree_skb_irq_reason() for
>> them. You need to delay the free for two reasons.
>> It seems pure software implementations didn't bother so far.
>> It might make sense to do napi_consume_skb() similar to
>> __netdev_alloc_skb() so that also budget=0 users fill the pool if this
>> is really a benefit.
> 
> I'm not convinced that this "optimization" will be an actual benefit on
> a busy system.  Let me explain the side-effect of local_bh_enable().
> 
> Calling local_bh_enable() is adding a re-scheduling opportunity, e.g.
> for processing softirq.  For a benchmark this might not be noticeable as
> this is the main workload.  If there isn't any pending softirq this is
> also not noticeable.  In a more mixed workload (or packet storm) this
> re-scheduling will allow others to "steal" CPU cycles from you.
> 
> Thus, you might not actually save any cycles via this short BH-disable
> section.  I remember that I was saving around 19ns / 68cycles on a
> 3.6GHz E5-1650 CPU, by using this SKB recycle cache.  The cost of a re-
> scheduling event is like more.
> 
> My advice is to use the napi_* function when already running within a
> BH-disabled section, as it makes sense to save those cycles
> (essentially reducing the time spend with BH-disabled).  Wrapping these
> napi_* function with BH-disabled just to use them outside NAPI feels
> wrong in so many ways.
> 
> The another reason why these napi_* functions belongs with NAPI is that
> netstack NIC drivers will (almost) always do TX completion first, that
> will free/consume some SKBs, and afterwards do RX processing that need
> to allocate SKBs for the incoming data frames.  Thus, keeping a cache of
> SKBs just released/consumed makes sense.  (p.s. in the past we always
> bulk free'ed all SKBs in the napi cache when exiting NAPI, as they would
> not be cache hot for next round).
> 
> --Jesper

Hey Jesper, thanks for the commentary, I appreciate it. I’ll chew on that
for the next revision of this work and see where it lands.

Thanks again - Jon

Powered by blists - more mailing lists