Message-ID: <CADXeF1FYoXTixLFFhESDkCo2HXG3JAzzdMCfkFrr2dqmRVQcWg@mail.gmail.com>
Date: Wed, 13 Nov 2024 13:26:29 +0900
From: Yuyang Huang <yuyanghuang@...gle.com>
To: Andrew Lunn <andrew@...n.ch>
Cc: Hangbin Liu <liuhangbin@...il.com>, "David S. Miller" <davem@...emloft.net>, 
	Eric Dumazet <edumazet@...gle.com>, Jakub Kicinski <kuba@...nel.org>, Paolo Abeni <pabeni@...hat.com>, 
	Simon Horman <horms@...nel.org>, David Ahern <dsahern@...nel.org>, roopa@...ulusnetworks.com, 
	jiri@...nulli.us, stephen@...workplumber.org, netdev@...r.kernel.org, 
	Maciej Żenczykowski <maze@...gle.com>, 
	Lorenzo Colitti <lorenzo@...gle.com>
Subject: Re: [PATCH net-next] netlink: add igmp join/leave notifications

>O.K. WiFi is not my area. But i'm more interested in uAPIs, and
> ensuring you are not adding APIs which promote kernel bypass.

WiFi chipset vendors must implement the Android WiFi HAL to install
and read the APF program from WiFi firmware. The Android System Server
will talk to vendor HAL service using the WiFi HAL. The datapath is:
Network Stack process -> Android System Server -> vendor HAL service
-> WiFi driver -> WiFi firmware. The Android WiFi HAL is specific to
Android. The vendor HAL service, WiFi driver and WiFi firmware are all
vendor proprietary software.  In other words, those API are not in
mainline yet.

Feel free to referring to the following page for more details:
* APF public doc:
https://source.android.com/docs/core/connect/android-packet-filter
* APF interpreter:
https://cs.android.com/android/platform/superproject/main/+/main:hardware/google/apf/v6/
* APF program generator:
https://cs.android.com/android/platform/superproject/main/+/main:packages/modules/NetworkStack/src/android/net/apf/ApfFilter.java
* APF WiFi HAL:
https://cs.android.com/android/platform/superproject/main/+/main:hardware/interfaces/wifi/aidl/aidl_api/android.hardware.wifi/2/android/hardware/wifi/IWifiStaIface.aidl;l=50?q=installApfPacketFilter

>Is the API to pass this bytestream to the wifi chipset in mainline?

Currently, these APIs are only in Android HAL, which is not in
mainline. We are working on adding proper APF APIs into ethtool.
Please some previous discussion here:
https://lore.kernel.org/netdev/20240813223325.3522113-1-maze@google.com/T/

>And how does APF differ from BPF? Or P4? Why would we want APF when
>mainline has BPF?

Android has utilized APF on Pixel devices since approximately 2016
(potentially as early as Pixel 3). All new devices launching with
Android 14/U+ are required to implement APFv4 on their WiFi client/STA
interface, enforced through Android VTS. APFv6 introduces support for
reply sending (e.g., ARP offload). This version is planned to be a
requirement for new devices launching with Android 16/W (or its
eventual successor), extending to wired interfaces as well.

APF is distinct from eBPF due to the latter's complexity and larger
memory footprint.  Many low-end WiFi chipsets lack sufficient RAM in
firmware to accommodate eBPF. In contrast, the latest APFv6
interpreter requires only ~5kB of compiled code and ~2kB of bytecode
(4kB recommended). Even this minimal size presents integration
challenges for some WiFi chipset vendors.

In sum, eBPF bytecode is very space inefficient compared to APF.

We have collaborated with major WiFi chipset vendors (Broadcom,
Synaptics, Qualcomm, Mediatek etc.) across various device categories
(phones, watches, tablets, TV etc.). To my knowledge, none have
integrated eBPF or P4 into their chipsets. Instead, they have either
implemented vendor-specific offload solutions or adopted APF. One of
APF's goals is to defrag the hardware offloading within the Android
ecosystem.

Moving the APF program generator into kernel space would eliminate the
need for generating the program in userspace. Ideally, future Linux
APIs would directly interact with drivers to manage offloads and APF
loading. The kernel could even be responsible for building the APF
bytecode, resolving race conditions caused by fetching kernel state
for bytecode generation.  A BPF program could potentially build the
APF bytecode during suspend.

However, this is a long-term goal requiring significant design and
discussion with the upstream Linux community. In the short term, APF
program generation will remain in user space.

>Do the new netlink message make sense without APF? Can i write a user
>space IGMP snooping implementation and then call bridge mdb
>add/del/replace?

The RTM_NEWMULTICAST and RTM_DELMULTICAST events introduced in this
patch enable user space implementation of IGMP/MLD offloading and
IPv4/IPv6 multicast filtering. I have limited knowledge on how to
implement IGMP snooping correctly so I don't know if they are
sufficient.

These two events have broader applications beyond APF. Any user space
implementation of IGMP/MLD offloading requires kernel events to signal
when a multicast address is added or removed. In fact, in the original
attempt of this patch from "Patrick Ruddy", the commit message also
mentioned about "having userspace applications to program multicast
MAC filters in hardware". Which makes me believe APF won't be the only
potential use case. Please check the following thread for more
details:  https://lore.kernel.org/r/20180906091056.21109-1-pruddy@vyatta.att-mail.com

>Assuming i can, why are the WiFi card not using that API, same a switches do?

I'm unsure if I understand your suggestion to use
RTM_NEWMDB/RTM_DELMDB/RTNLGRP_MDB. Currently, the kernel doesn't send
these events outside the bridge context.
Previous patches attempted to reuse these events, but reviewers raised
concerns about potential confusion for existing users.  IMHO, I agree
with the reviewer that re-using the RTM_NEWMDB/RTM_DELMDB/RTNLGRP_MDB
might be confusing and might affect the current applications that use
those events. Those events seem to make more sense from a bridge
perspective. Our use cases are targeting the end device which is not a
bridge or router.

It might also make sense to consider whether to accept the proposed
APIs from an API completeness perspective. The current netlink API for
multicast addresses seems incomplete. While RTM_GETMULTICAST exists,
it only supports IPv6, not IPv4. This limitation forces tools like 'ip
maddr' to rely on parsing procfs instead of using netlink.
Additionally, 'ip monitor' cannot track multicast address additions or
removals. I feel it would make sense to have full netlink based
dumping and event notification support for both IPv4/IPv6 multicast
addresses as well.

Thanks,
Yuyang



On Wed, Nov 13, 2024 at 10:17 AM Andrew Lunn <andrew@...n.ch> wrote:
>
> On Wed, Nov 13, 2024 at 09:56:17AM +0900, Yuyang Huang wrote:
> > > Please could you say more about programming hardware offload filters?
> > > How is this done?
> >
> > Sure,  please let me explain a little bit further on how Android
> > Packet Filter (APF) works here.
> >
> > The Android Packet Filter (APF) has two parts:
> > * APF Interpreter: Runs on the Wi-Fi chipset and executes APF
> > programs(bytecodes) to decide whether to accept, drop, or reply to
> > incoming packets.
>
> O.K. WiFi is not my area. But i'm more interested in uAPIs, and
> ensuring you are not adding APIs which promote kernel bypass.
>
> Is the API to pass this bytestream to the wifi chipset in mainline?
> And how does APF differ from BPF? Or P4? Why would we want APF when
> mainline has BPF?
>
> Do the new netlink message make sense without APF? Can i write a user
> space IGMP snooping implementation and then call bridge mdb
> add/del/replace? Assuming i can, why are the WiFi card not using that
> API, same a switches do?
>
>         Andrew

Powered by blists - more mailing lists