| lists.openwall.net | lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC | |
|
Open Source and information security mailing list archives
| ||
|
Message-ID: <Z63B_UE61OdrgEJY@linaro.org> Date: Thu, 13 Feb 2025 10:57:17 +0100 From: Stephan Gerhold <stephan.gerhold@...aro.org> To: Wesley Cheng <quic_wcheng@...cinc.com> Cc: srinivas.kandagatla@...aro.org, mathias.nyman@...el.com, perex@...ex.cz, conor+dt@...nel.org, dmitry.torokhov@...il.com, corbet@....net, broonie@...nel.org, lgirdwood@...il.com, tiwai@...e.com, krzk+dt@...nel.org, pierre-louis.bossart@...ux.intel.com, Thinh.Nguyen@...opsys.com, robh@...nel.org, gregkh@...uxfoundation.org, linux-kernel@...r.kernel.org, devicetree@...r.kernel.org, linux-sound@...r.kernel.org, linux-input@...r.kernel.org, linux-usb@...r.kernel.org, linux-arm-msm@...r.kernel.org, linux-doc@...r.kernel.org Subject: Re: [PATCH v34 00/31] Introduce QC USB SND audio offloading support On Tue, Jan 21, 2025 at 01:04:47PM -0800, Wesley Cheng wrote: > Requesting to see if we can get some Acked-By tags, and merge on usb-next. > > Several Qualcomm based chipsets can support USB audio offloading to a > dedicated audio DSP, which can take over issuing transfers to the USB > host controller. The intention is to reduce the load on the main > processors in the SoC, and allow them to be placed into lower power modes. > There are several parts to this design: > 1. Adding ASoC binding layer > 2. Create a USB backend for Q6DSP > 3. Introduce XHCI interrupter support > 4. Create vendor ops for the USB SND driver > > USB | ASoC > -------------------------------------------------------------------- > | _________________________ > | |sm8250 platform card | > | |_________________________| > | | | > | ___V____ ____V____ > | |Q6USB | |Q6AFE | > | |"codec" | |"cpu" | > | |________| |_________| > | ^ ^ ^ > | | |________| > | ___V____ | > | |SOC-USB | | > ________ ________ | | | > |USB SND |<--->|QC offld|<------------>|________| | > |(card.c)| | |<---------- | > |________| |________|___ | | | > ^ ^ | | | ____________V_________ > | | | | | |APR/GLINK | > __ V_______________V_____ | | | |______________________| > |USB SND (endpoint.c) | | | | ^ > |_________________________| | | | | > ^ | | | ___________V___________ > | | | |->|audio DSP | > ___________V_____________ | | |_______________________| > |XHCI HCD |<- | > |_________________________| | > > > Adding ASoC binding layer > ========================= > soc-usb: Intention is to treat a USB port similar to a headphone jack. > The port is always present on the device, but cable/pin status can be > enabled/disabled. Expose mechanisms for USB backend ASoC drivers to > communicate with USB SND. > > Create a USB backend for Q6DSP > ============================== > q6usb: Basic backend driver that will be responsible for maintaining the > resources needed to initiate a playback stream using the Q6DSP. Will > be the entity that checks to make sure the connected USB audio device > supports the requested PCM format. If it does not, the PCM open call will > fail, and userspace ALSA can take action accordingly. > > Introduce XHCI interrupter support > ================================== > XHCI HCD supports multiple interrupters, which allows for events to be routed > to different event rings. This is determined by "Interrupter Target" field > specified in Section "6.4.1.1 Normal TRB" of the XHCI specification. > > Events in the offloading case will be routed to an event ring that is assigned > to the audio DSP. > > Create vendor ops for the USB SND driver > ======================================== > qc_audio_offload: This particular driver has several components associated > with it: > - QMI stream request handler > - XHCI interrupter and resource management > - audio DSP memory management > > When the audio DSP wants to enable a playback stream, the request is first > received by the ASoC platform sound card. Depending on the selected route, > ASoC will bring up the individual DAIs in the path. The Q6USB backend DAI > will send an AFE port start command (with enabling the USB playback path), and > the audio DSP will handle the request accordingly. > > Part of the AFE USB port start handling will have an exchange of control > messages using the QMI protocol. The qc_audio_offload driver will populate the > buffer information: > - Event ring base address > - EP transfer ring base address > > and pass it along to the audio DSP. All endpoint management will now be handed > over to the DSP, and the main processor is not involved in transfers. > > Overall, implementing this feature will still expose separate sound card and PCM > devices for both the platform card and USB audio device: > 0 [SM8250MTPWCD938]: sm8250 - SM8250-MTP-WCD9380-WSA8810-VA-D > SM8250-MTP-WCD9380-WSA8810-VA-DMIC > 1 [Audio ]: USB-Audio - USB Audio > Generic USB Audio at usb-xhci-hcd.1.auto-1.4, high speed > > This is to ensure that userspace ALSA entities can decide which route to take > when executing the audio playback. In the above, if card#1 is selected, then > USB audio data will take the legacy path over the USB PCM drivers, etc... > > The current limitation is that the latest USB audio device that is identified > will be automatically selected by the Q6USB BE DAI for offloading. Future > patches can be added to possibly add for more flexibility, but until the userpace > applications can be better defined, having these mechanisms will complicate the > overall implementation. > > USB offload Kcontrols > ===================== > Part of the vendor offload package will have a mixer driver associated with it > (mixer_usb_offload.c). This entity will be responsible for coordinating with > SOC USB and the Q6USB backend DAI to fetch information about the sound card > and PCM device indices associated with the offload path. The logic is done > based on the current implementation of how paths are controlled within the QC > ASoC implementation. > > QC ASoC Q6Routing > ----------------- > Within the Q6 ASOC design, the registered ASoC platform card will expose a set > of kcontrols for enabling the BE DAI links to the FE DAI link. For example: > > tinymix -D 0 contents > Number of controls: 1033 > ctl type num name value > ... > 1025 BOOL 1 USB Mixer MultiMedia1 Off > 1026 BOOL 1 USB Mixer MultiMedia2 Off > 1027 BOOL 1 USB Mixer MultiMedia3 Off > 1028 BOOL 1 USB Mixer MultiMedia4 Off > 1029 BOOL 1 USB Mixer MultiMedia5 Off > 1030 BOOL 1 USB Mixer MultiMedia6 Off > 1031 BOOL 1 USB Mixer MultiMedia7 Off > 1032 BOOL 1 USB Mixer MultiMedia8 Off > > Each of these kcontrols will enable the USB BE DAI link (q6usb) to be connected > to a FE DAI link (q6asm). Since each of these controls are DAPM widgets, when > it is enabled, the DAPM widget's "connect" flag is updated accordingly. > > USB Offload Mapping > ------------------- > Based on the Q6routing, the USB BE DAI link can determine which sound card and > PCM device is enabled for offloading. Fetching the ASoC platform sound card's > information is fairly straightforward, and the bulk of the work goes to finding > the corresponding PCM device index. As mentioned above, the USB BE DAI can > traverse the DAPM widgets to find the DAPM path that is related to the control > for the "USB Mixer." Based on which "USB Mixer" is enabled, it can find the > corresponding DAPM widget associated w/ the FE DAI link (Multimedia*). From there > it can find the PCM device created for the Multimedia* stream. > > Only one BE DAI link can be enabled per FE DAI. For example, if the HDMI path is > enabled for Multimedia1, the USB Mixer will be disabled and switched over. > > Examples of kcontrol > -------------------- > tinymix -D 0 contents > Number of controls: 1033 > ctl type num name > ... > 1025 BOOL 1 USB Mixer MultiMedia1 Off > 1026 BOOL 1 USB Mixer MultiMedia2 On > 1027 BOOL 1 USB Mixer MultiMedia3 Off > 1028 BOOL 1 USB Mixer MultiMedia4 Off > 1029 BOOL 1 USB Mixer MultiMedia5 Off > 1030 BOOL 1 USB Mixer MultiMedia6 Off > 1031 BOOL 1 USB Mixer MultiMedia7 Off > 1032 BOOL 1 USB Mixer MultiMedia8 Off > > tinymix -D 2 contents > Number of controls: 10 > ctl type num name value > 0 INT 2 Capture Channel Map 0, 0 (range 0->36) > 1 INT 2 Playback Channel Map 0, 0 (range 0->36) > 2 BOOL 1 Headset Capture Switch On > 3 INT 1 Headset Capture Volume 10 (range 0->13) > 4 BOOL 1 Sidetone Playback Switch On > 5 INT 1 Sidetone Playback Volume 4096 (range 0->8192) > 6 BOOL 1 Headset Playback Switch On > 7 INT 2 Headset Playback Volume 20, 20 (range 0->24) > 8 INT 1 USB Offload Playback Card Route PCM#0 0 (range -1->32) > 9 INT 1 USB Offload Playback PCM Route PCM#0 1 (range -1->255) > > The example highlights that the userspace/application can utilize the offload path > for the USB device on card#0 PCM device#1. > > When dealing with multiple USB audio devices, only the latest USB device identified > is going to be selected for offload capable. > > tinymix -D 1 contents > Number of controls: 9 > ctl type num name value > 0 INT 2 Capture Channel Map 0, 0 (range 0->36) > 1 INT 2 Playback Channel Map 0, 0 (range 0->36) > 2 BOOL 1 Headset Capture Switch On > 3 INT 1 Headset Capture Volume 1 (range 0->4) > 4 BOOL 1 Sidetone Playback Switch On > 5 INT 1 Sidetone Playback Volume 4096 (range 0->8192) > 6 BOOL 1 Headset Playback Switch On > 7 INT 2 Headset Playback Volume 20, 20 (range 0->24) > 8 INT 1 USB Offload Playback Card Route PCM#0 -1 (range -1->32) > 9 INT 1 USB Offload Playback PCM Route PCM#0 -1 (range -1->255) > > "-1, -1" shows that this device has no route to the offload path. > > This feature was validated using: > - tinymix: set/enable the multimedia path to route to USB backend > - tinyplay: issue playback on platform card > Could you share the device tree changes for a board upstream and any other changes needed to be able to test this series? E.g. for sm8250-mtp.dts, based on your examples above. There are examples for qcom,q6usb etc in the bindings you submitted, but having a full example would be much easier. Thanks, Stephan
Powered by blists - more mailing lists