[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]
Message-ID: <CAGXJAmyb8s5xu9W1dXxhwnQfeY4=P21FquBymonUseM_OpaU2w@mail.gmail.com>
Date: Tue, 28 Jan 2025 17:23:02 -0800
From: John Ousterhout <ouster@...stanford.edu>
To: Paolo Abeni <pabeni@...hat.com>
Cc: netdev@...r.kernel.org, edumazet@...gle.com, horms@...nel.org,
kuba@...nel.org
Subject: Re: [PATCH net-next v6 05/12] net: homa: create homa_rpc.h and homa_rpc.c
On Tue, Jan 28, 2025 at 12:20 AM Paolo Abeni <pabeni@...hat.com> wrote:
> ...
> > I think that might work, but it would suffer from the slow reclamation
> > problem I mentioned with RCU. It would also create more complexity in
> > the code (e.g. the allocation might still turn out to be redundant, so
> > there would need to be additional code to check for that: the lookup
> > would essentially have to be done twice in the case of creating a new
> > RPC). I'd rather not incur this complexity until there's evidence that
> > GFP_ATOMIC is causing problems.
>
> Have a look at tcp established socket lookup and the
> SLAB_TYPESAFE_BY_RCU flag usage for slab-based allocations. A combo of
> such flag for RPC allocation (using a dedicated kmem_cache) and RCU
> lookup should improve consistently the performances, with a consolidate
> code layout and no unmanageable problems with large number of objects
> waiting for the grace period.
I will check that out.
> >>> Homa needs to handle a very high rate of RPCs, so this would result in
> >>> too much accumulated memory (in particular, skbs don't get reclaimed
> >>> until the RPC is reclaimed).
> >>
> >> For the RPC struct, that above is a fair point, but why skbs need to be
> >> freed together with the RCP struct? if you have skbs i.e. sitting in a
> >> RX queue, you can flush such queue when the RPC goes out of scope,
> >> without any additional delay.
> >
> > Reclaiming the skbs inline would be too expensive;
>
> Why? For other protocols the main skb free cost is due to memory
> accounting, that homa is currently not implementing, so I don't see why
> it should be critically expansive at this point (note that homa should
> performat least rmem/wmem accounting, but let put this aside for a
> moment). Could you please elaborate on this topic?
In my measurements, skb freeing is by far the largest cost in RPC
reaping. I'm not currently in a good position to remeasure this, but
my recollection is that it takes a few hundred ns to free an skb. A
large RPC (1 MByte is Homa's current limit) will have at least 100
skbs (with jumbo frames) and more than 600 skbs with 1500B frames:
that's 20-100 usec. The problem is that this can occur in a place
where it delays the processing of a packet for an unrelated short
message. Under good conditions, Homa can handle a short RPC in 15 usec
(end-to-end round-trip) so delaying a packet by 100 usec is a severe
penalty.
> [...]
> > Note that the bucket locks would be needed even with RCU usage, in
> > order to permit concurrent RPC creation in different buckets. Thus
> > Homa's locking scheme doesn't introduce additional locks; it
> > eliminates locks that would otherwise be needed on individual RPCs and
> > uses the bucket locks for 2 purposes.
>
> It depends on the relative frequency of RPC lookup vs RPC
> insertion/deletion. i.e. for TCP connections the lookup frequency is
> expected to be significantly higher than the socket creation and
> destruction.
>
> I understand the expected patter in quite different with homa RPC? If so
> you should at least consider a dedicated kmem_cache for such structs.
Right: Homa creates sockets even less often than TCP, but it creates
new RPCs all the time. For an RPC with short messages (very common)
the client will do one insertion and one lookup; the server will do an
insertion but never a lookup. Thus the relative frequency of lookup
vs. insertion is quite different in Homa from TCP. It might well be
worth looking at a kmem_cache for the RPC structs. I don't yet know
much about kmem_caches, but I'll put it on my "to do" list.
-John-
Powered by blists - more mailing lists