Message-ID: <CANP3RGdMEYrbHWMEb-gTUgNRwje66FTihccVgrg6s4z0c0a+Kw@mail.gmail.com>
Date: Thu, 15 Aug 2024 17:49:42 -0700
From: Maciej Żenczykowski <maze@...gle.com>
To: Jakub Kicinski <kuba@...nel.org>
Cc: Florian Fainelli <f.fainelli@...il.com>, 
	Linux Network Development Mailing List <netdev@...r.kernel.org>, "David S . Miller" <davem@...emloft.net>, 
	Eric Dumazet <edumazet@...gle.com>, Paolo Abeni <pabeni@...hat.com>, 
	"Kory Maincent (Dent Project)" <kory.maincent@...tlin.com>, Ahmed Zaki <ahmed.zaki@...el.com>, 
	Edward Cree <ecree.xilinx@...il.com>, Yuyang Huang <yuyanghuang@...gle.com>, 
	Lorenzo Colitti <lorenzo@...gle.com>
Subject: Re: [PATCH net-next] ethtool: add tunable api to disable various
 firmware offloads

On Wed, Aug 14, 2024 at 5:32 PM Jakub Kicinski <kuba@...nel.org> wrote:
> Sounds sane, adding Florian, he mentioned MDNS at last netconf.
> Tho, wasn't there supposed to be a more granular API in Android
> to control such protocol offloads?

We're deprecating the 'new' but already old mdns offload logic in
favour of APFv6+ (unfortunately there was a race here, we weren't
ready with APFv6 in time...).

APF [Android Packet Filter] is a bytecode interpreter built into
[wifi] firmware, currently the bytecode is fed in via Android Wifi HAL
(we'd like to change this to ethtool or something as well) from a java
bytecode generator that knows the state of the system.

APFv2 (ingress packet filtering to prevent spurious wakeups) and APFv4
(adds drop counters) have been in use on Pixel for many many years now
(going back to Pixel 3? somewhere 2016-ish).
We require all new (launched on Android 14/U+) devices to have a
working APFv4 implementation on their wifi client/sta interface
(enforced via Android VTS).
APFv6 (only available on non publicly available hardware atm) adds
support for sending back replies (ie. arp offload & the like).  We
hope to require it for newly launched devices on (the future) Android
16/W (or whatever it ends up being) and to extend it to non-wifi (ie.
wired) as well.

The APFv6 interpreter source is here:
https://android.googlesource.com/platform/hardware/google/apf/+/refs/heads/main/v6/
for the curious.

APF is *not* eBPF simply because eBPF is *far* too complex (there's
not enough space for it in the firmware/ram of many low end wifi
chipsets).  The latest APFv6 interpreter is only ~5kB of compiled
code, and another ~2kB of bytecode (though space for 4kB is
recommended).  Some wifi chipset vendors are finding it very hard to
find room for even that.

eBPF bytecode is very space inefficient compared to APF.

> You gotta find an upstream driver which implements this for us to merge.

This came about due to a request from a major wifi chipset vendor to
notify them which offloads could/should be disabled due to being
reimplemented via APFv6 (Android Packet Filter).
So I believe (note: we're still discussing this) we likely have folks
willing to implement this for at least 1 wifi driver (not sure which
one, and if it is merged upstream or lives out-of-tree).

I am of course in a very hard position here, as I don't own any
drivers/firmware - AFAIK even Pixel's wifi/cell drivers aren't Google
owned/maintained, but rather Broadcom/Synaptics/Qualcomm/Mediatek/etc
as appropriate...

I do know there is a need for an api of this sort (not necessarily
exactly this patch though), and if we merge something (sane :-) ) into
Linux, we can then backport that (either to LTS or just to ACK), and
then we (as in Google) can require implementations (for new
hardware/drivers) in future versions of Android...

Presumably that would result in implementations in many drivers,
though not necessarily any in-tree ones (I have no idea what the
current state of in-vs-out-of-tree drivers is wrt. Android wifi/cell
hardware)

This is very much a chicken-and-egg problem though.  As long as there
is no 'public' API, the default approach is for per-vendor or even
per-chip / per-driver custom apis, hidden behind Android HALs.  For
example we have such an Android HAL api for disabling ND offload on at
least one of our devices.  Of course the HAL itself is backed by
calling into the driver - just over some driver specific netlink...

Similarly for APF bytecode installation: it all goes via Android HALs,
which then in practice immediately turn around and talk straight over
driver/vendor-specific netlink to the driver.

I'd *love* to simplify all this.  Indeed in an ideal world, we'd have
'native' Linux apis into drivers to disable offloads, load APF, and
could even teach the kernel to actually build the APF bytecode to load
in in the first place (this would fix lots of races we currently have,
where we need to fetch state out of the kernel to build the bytecode
to pass back into the kernel driver/firmware...).  Perhaps a BPF
program to build the APF bytecode during the suspend sequence --
seriously, this isn't even a joke.

That said... the problem is simply too large to bite all at once, so
I'm trying to find a way out of the logjam wrt. upstream Linux that
APF has been in for years.

If we can't settle on a 'public' Linux API for this (not saying *this*
patch is necessarily the right way) we'll likely end up implementing
this either via ethtool privflags (or something) or simply via an
Android HAL... again :-( Be aware HALs are what most of the org loves
and it's hard enough to explain why we don't want to go via a HAL....

This isn't meant to be a threat ;-) it's just an unavoidable statement
of fact... :-(

What I could offer to provide is some sort of tun/tap/veth/virtio-net
based APF implementation for testing (but that's not exactly relevant
to *this* patch, since they don't support offloads to disable in the
first place) - as I would love to have that available in cuttlefish
(Android VM) for integration testing.

> If Florian doesn't have any quick uses -- I think Intel ethernet drivers
> have private flags for enabling/disabling an LLDP agent. That could be
> another way..
> --
> pw-bot: cr

--
Maciej Żenczykowski, Kernel Networking Developer @ Google

Powered by blists - more mailing lists