lists.openwall.net   lists  /  announce  owl-users  owl-dev  john-users  john-dev  passwdqc-users  yescrypt  popa3d-users  /  oss-security  kernel-hardening  musl  sabotage  tlsify  passwords  /  crypt-dev  xvendor  /  Bugtraq  Full-Disclosure  linux-kernel  linux-netdev  linux-ext4  linux-hardening  linux-cve-announce  PHC 
Open Source and information security mailing list archives
 
Hash Suite: Windows password security audit tool. GUI, reports in PDF.
[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]
Message-ID: <CAGXJAmyb8s5xu9W1dXxhwnQfeY4=P21FquBymonUseM_OpaU2w@mail.gmail.com>
Date: Tue, 28 Jan 2025 17:23:02 -0800
From: John Ousterhout <ouster@...stanford.edu>
To: Paolo Abeni <pabeni@...hat.com>
Cc: netdev@...r.kernel.org, edumazet@...gle.com, horms@...nel.org, 
	kuba@...nel.org
Subject: Re: [PATCH net-next v6 05/12] net: homa: create homa_rpc.h and homa_rpc.c

On Tue, Jan 28, 2025 at 12:20 AM Paolo Abeni <pabeni@...hat.com> wrote:
> ...
> > I think that might work, but it would suffer from the slow reclamation
> > problem I mentioned with RCU. It would also create more complexity in
> > the code (e.g. the allocation might still turn out to be redundant, so
> > there would need to be additional code to check for that: the lookup
> > would essentially have to be done twice in the case of creating a new
> > RPC). I'd rather not incur this complexity until there's evidence that
> > GFP_ATOMIC is causing problems.
>
> Have a look at tcp established socket lookup and the
> SLAB_TYPESAFE_BY_RCU flag usage for slab-based allocations. A combo of
> such flag for RPC allocation (using a dedicated kmem_cache) and RCU
> lookup should improve consistently the performances, with a consolidate
> code layout and no unmanageable problems with large number of objects
> waiting for the grace period.

I will check that out.

> >>> Homa needs to handle a very high rate of RPCs, so this would result in
> >>> too much accumulated memory  (in particular, skbs don't get reclaimed
> >>> until the RPC is reclaimed).
> >>
> >> For the RPC struct, that above is a fair point, but why skbs need to be
> >> freed together with the RCP struct? if you have skbs i.e. sitting in a
> >> RX queue, you can flush such queue when the RPC goes out of scope,
> >> without any additional delay.
> >
> > Reclaiming the skbs inline would be too expensive;
>
> Why? For other protocols the main skb free cost is due to memory
> accounting, that homa is currently not implementing, so I don't see why
> it should be critically expansive at this point (note that homa should
> performat least rmem/wmem accounting, but let put this aside for a
> moment). Could you please elaborate on this topic?

In my measurements, skb freeing is by far the largest cost in RPC
reaping. I'm not currently in a good position to remeasure this, but
my recollection is that it takes a few hundred ns to free an skb. A
large RPC (1 MByte is Homa's current limit) will have at least 100
skbs (with jumbo frames) and more than 600 skbs with 1500B frames:
that's 20-100 usec. The problem is that this can occur in a place
where it delays the processing of a packet for an unrelated short
message. Under good conditions, Homa can handle a short RPC in 15 usec
(end-to-end round-trip) so delaying a packet by 100 usec is a severe
penalty.

> [...]
> > Note that the bucket locks would be needed even with RCU usage, in
> > order to permit concurrent RPC creation in different buckets. Thus
> > Homa's locking scheme doesn't introduce additional locks; it
> > eliminates locks that would otherwise be needed on individual RPCs and
> > uses the bucket locks for 2 purposes.
>
> It depends on the relative frequency of RPC lookup vs RPC
> insertion/deletion. i.e. for TCP connections the lookup frequency is
> expected to be significantly higher than the socket creation and
> destruction.
>
> I understand the expected patter in quite different with homa RPC? If so
> you should at least consider a dedicated kmem_cache for such structs.

Right: Homa creates sockets even less often than TCP, but it creates
new RPCs all the time. For an RPC with short messages (very common)
the client will do one insertion and one lookup; the server will do an
insertion but never a lookup. Thus the relative frequency of lookup
vs. insertion is quite different in Homa from TCP. It might well be
worth looking at a kmem_cache for the RPC structs. I don't yet know
much about kmem_caches, but I'll put it on my "to do" list.


-John-

Powered by blists - more mailing lists

Powered by Openwall GNU/*/Linux Powered by OpenVZ