Message-ID: <ccd61585-de58-4960-b08c-c1c53db8d110@quicinc.com>
Date:   Mon, 24 Jul 2023 19:38:25 -0700
From:   Wesley Cheng <quic_wcheng@...cinc.com>
To:     <agross@...nel.org>, <andersson@...nel.org>, <robh+dt@...nel.org>,
        <krzysztof.kozlowski+dt@...aro.org>, <conor+dt@...nel.org>,
        <catalin.marinas@....com>, <will@...nel.org>,
        <mathias.nyman@...el.com>, <gregkh@...uxfoundation.org>,
        <lgirdwood@...il.com>, <broonie@...nel.org>, <perex@...ex.cz>,
        <tiwai@...e.com>, <srinivas.kandagatla@...aro.org>,
        <bgoswami@...cinc.com>, <Thinh.Nguyen@...opsys.com>
CC:     <linux-arm-msm@...r.kernel.org>, <devicetree@...r.kernel.org>,
        <linux-kernel@...r.kernel.org>,
        <linux-arm-kernel@...ts.infradead.org>,
        <linux-usb@...r.kernel.org>, <alsa-devel@...a-project.org>,
        <quic_jackp@...cinc.com>, <pierre-louis.bossart@...ux.intel.com>,
        <oneukum@...e.com>, <albertccwang@...gle.com>,
        <o-takashi@...amocchi.jp>
Subject: Re: [PATCH v4 00/32] Introduce QC USB SND audio offloading support

Apologies for the long delay in addressing some of your comments and 
feedback.  I understand it is difficult to refresh a lot of the material 
after such a long gap, but hopefully I have addressed some of the 
concerns in the latest revision.  Appreciate all the suggestions/reviews!

Thanks
Wesley Cheng

On 7/24/2023 7:33 PM, Wesley Cheng wrote:
> Several Qualcomm based chipsets can support USB audio offloading to a
> dedicated audio DSP, which can take over issuing transfers to the USB
> host controller.  The intention is to reduce the load on the main
> processors in the SoC, and allow them to be placed into lower power modes.
> There are several parts to this design:
>    1. Adding ASoC binding layer
>    2. Create a USB backend for Q6DSP
>    3. Introduce XHCI interrupter support
>    4. Create vendor ops for the USB SND driver
> 
>        USB                          |            ASoC
> --------------------------------------------------------------------
>                                     |  _________________________
>                                     | |sm8250 platform card     |
>                                     | |_________________________|
>                                     |         |           |
>                                     |      ___V____   ____V____
>                                     |     |Q6USB   | |Q6AFE    |
>                                     |     |"codec" | |"cpu"    |
>                                     |     |________| |_________|
>                                     |         ^  ^        ^
>                                     |         |  |________|
>                                     |      ___V____    |
>                                     |     |SOC-USB |   |
>     ________       ________               |        |   |
>    |USB SND |<--->|QC offld|<------------>|________|   |
>    |(card.c)|     |        |<----------                |
>    |________|     |________|___     | |                |
>        ^               ^       |    | |    ____________V_________
>        |               |       |    | |   |APR/GLINK             |
>     __ V_______________V_____  |    | |   |______________________|
>    |USB SND (endpoint.c)     | |    | |              ^
>    |_________________________| |    | |              |
>                ^               |    | |   ___________V___________
>                |               |    | |->|audio DSP              |
>     ___________V_____________  |    |    |_______________________|
>    |XHCI HCD                 |<-    |
>    |_________________________|      |
> 
> 
> Adding ASoC binding layer:
> soc-usb: Intention is to treat a USB port similar to a headphone jack.
> The port is always present on the device, but cable/pin status can be
> enabled/disabled.  Expose mechanisms for USB backend ASoC drivers to
> communicate with USB SND.
> 
> Create a USB backend for Q6DSP:
> q6usb: Basic backend driver that will be responsible for maintaining the
> resources needed to initiate a playback stream using the Q6DSP.  Will
> be the entity that checks to make sure the connected USB audio device
> supports the requested PCM format.  If it does not, the PCM open call will
> fail, and userpsace ALSA can take action accordingly.
> 
> Introduce XHCI interrupter support:
> XHCI HCD supports multiple interrupters, which allows for events to be routed
> to different event rings.  This is determined by "Interrupter Target" field
> specified in Section "6.4.1.1 Normal TRB" of the XHCI specification.
> 
> Events in the offloading case will be routed to an event ring that is assigned
> to the audio DSP.
> 
> Create vendor ops for the USB SND driver:
> qc_audio_offload: This particular driver has several components associated
> with it:
> - QMI stream request handler
> - XHCI interrupter and resource management
> - audio DSP memory management
> 
> When the audio DSP wants to enable a playback stream, the request is first
> received by the ASoC platform sound card.  Depending on the selected route,
> ASoC will bring up the individual DAIs in the path.  The Q6USB backend DAI
> will send an AFE port start command (with enabling the USB playback path), and
> the audio DSP will handle the request accordingly.
> 
> Part of the AFE USB port start handling will have an exchange of control
> messages using the QMI protocol.  The qc_audio_offload driver will populate the
> buffer information:
> - Event ring base address
> - EP transfer ring base address
> 
> and pass it along to the audio DSP.  All endpoint management will now be handed
> over to the DSP, and the main processor is not involved in transfers.
> 
> Overall, implementing this feature will still expose separate sound card and PCM
> devices for both the platorm card and USB audio device:
>   0 [SM8250MTPWCD938]: sm8250 - SM8250-MTP-WCD9380-WSA8810-VA-D
>                        SM8250-MTP-WCD9380-WSA8810-VA-DMIC
>   1 [Audio          ]: USB-Audio - USB Audio
>                        Generic USB Audio at usb-xhci-hcd.1.auto-1.4, high speed
> 
> This is to ensure that userspace ALSA entities can decide which route to take
> when executing the audio playback.  In the above, if card#1 is selected, then
> USB audio data will take the legacy path over the USB PCM drivers, etc...
> 
> This feature was validated using:
> - tinymix: set/enable the multimedia path to route to USB backend
> - tinyplay: issue playback on platform card
> 
> Changelog
> --------------------------------------------
> Changes in v4:
> - Rebased to xhci/for-usb-next
> - Addressed some dt-bindings comments
> 
> XHCI:
> - Pulled in latest changes from Mathias' feature_interrupters branch:
> https://git.kernel.org/pub/scm/linux/kernel/git/mnyman/xhci.git/log/?h=feature_interrupters
> 
> - Fixed commit text and signage for the XHCI sideband/interrupter related changes
> - Added some logic to address the FIXME tags mentioned throughout the commits, such
> as handling multi segment rings and building the SGT, locking concerns, and ep
> cleanup operations.
> - Removed some fixme tags for conditions that may not be needed/addressed.
> - Repurposed the new endpoint stop sync API to be utilized in other places.
> - Fixed potential compile issue if XHCI sideband config is not defined.
> 
> ASoC:
> - Added sound jack control into the Q6USB driver.  Allows for userpsace to know when
> an offload capable device is connected.
> 
> USB SND:
> - Avoided exporting _snd_pcm_hw_param_set based on Takashi's recommendation.
> - Split USB QMI packet header definitions into a separate commit.  This is used to
> properly allow the QMI interface driver to parse and route QMI packets accordingly
> - Added a "depends on" entry when enabling QC audio offload to avoid compile time
> issues.
> 
> Changes in v3:
> - Changed prefix from RFC to PATCH
> - Rebased entire series to usb-next
> - Updated copyright years
> 
> XHCI:
> - Rebased changes on top of XHCI changes merged into usb-next, and only added
> changes that were still under discussion.
> - Added change to read in the "num-hc-interrupters" device property.
> 
> ASoC:
> - qusb6 USB backend
>    - Incorporated suggestions to fetch iommu information with existing APIs
>    - Added two new sound kcontrols to fetch offload status and offload device
>      selection.
>      - offload status - will return the card and pcm device in use
>          tinymix -D 0 get 1 --> 1, 0 (offload in progress on card#1 pcm#0)
> 
>      - device selection - set the card and pcm device to enable offload on. Ex.:
>          tinymix -D 0 set 1 2 0  --> sets offload on card#2 pcm#0
>                                      (this should be the USB card)
> 
> USB SND:
> - Fixed up some locking related concerns for registering platform ops.
>     - Moved callbacks under the register_mutex, so that
> - Modified APIs to properly pass more information about the USB SND device, so
> that the Q6USB backend can build a device list/map, in order to monitor offload
> status and device selection.
> 
> Changes in v2:
> 
> XHCI:
> - Replaced XHCI and HCD changes with Mathias' XHCI interrupter changes
> in his tree:
> https://git.kernel.org/pub/scm/linux/kernel/git/mnyman/xhci.git/log/?h=feature_interrupters
> 
> Adjustments made to Mathias' changes:
>    - Created xhci-intr.h to export/expose interrupter APIs versus exposing xhci.h.
>      Moved dependent structures to this file as well. (so clients can parse out
>      information from "struct xhci_interrupter")
>    - Added some basic locking when requesting interrupters.
>    - Fixed up some sanity checks.
>    - Removed clearing of the ERSTBA during freeing of the interrupter. (pending
>      issue where SMMU fault occurs if DMA addr returned is 64b - TODO)
> 
> - Clean up pending events in the XHCI secondary interrupter.  While testing USB
> bus suspend, it was seen that on bus resume, the xHCI HC would run into a command
> timeout.
> - Added offloading APIs to xHCI to fetch transfer and event ring information.
> 
> ASoC:
> - Modified soc-usb to allow for multiple USB port additions.  For this to work,
> the USB offload driver has to have a reference to the USB backend by adding
> a "usb-soc-be" DT entry to the device saved into XHCI sysdev.
> - Created separate dt-bindings for defining USB_RX port.
> - Increased APR timeout to accommodate the situation where the AFE port start
> command could be delayed due to having to issue a USB bus resume while
> handling the QMI stream start command.
> 
> USB SND:
> - Added a platform ops during usb_audio_suspend().  This allows for the USB
> offload driver to halt the audio stream when system enters PM suspend.  This
> ensures the audio DSP is not issuing transfers on the USB bus.
> - Do not override platform ops if they are already populated.
> - Introduce a shared status variable between the USB offload and USB SND layers,
> to ensure that only one path is active at a time.  If the USB bus is occupied,
> then userspace is notified that the path is busy.
> 
> Mathias Nyman (3):
>    xhci: add support to allocate several interrupters
>    xhci: add helper to stop endpoint and wait for completion
>    xhci: sideband: add initial api to register a sideband entity
> 
> Wesley Cheng (29):
>    usb: host: xhci-mem: Cleanup pending secondary event ring events
>    usb: host: xhci-mem: Allow for interrupter clients to choose specific
>      index
>    ASoC: Add SOC USB APIs for adding an USB backend
>    ASoC: dt-bindings: qcom,q6dsp-lpass-ports: Add USB_RX port
>    ASoC: qcom: qdsp6: Introduce USB AFE port to q6dsp
>    ASoC: qdsp6: q6afe: Increase APR timeout
>    ASoC: qcom: Add USB backend ASoC driver for Q6
>    sound: usb: card: Introduce USB SND platform op callbacks
>    sound: usb: Export USB SND APIs for modules
>    dt-bindings: usb: dwc3: Add snps,num-hc-interrupters definition
>    usb: dwc3: Add DT parameter to specify maximum number of interrupters
>    usb: host: xhci-plat: Set XHCI max interrupters if property is present
>    sound: usb: pcm: Export fixed rate check USB SND API
>    sound: usb: qcom: Add USB QMI definitions
>    sound: usb: Introduce QC USB SND offloading support
>    sound: usb: card: Check for support for requested audio format
>    sound: soc: soc-usb: Add PCM format check API for USB backend
>    sound: soc: qcom: qusb6: Ensure PCM format is supported by USB audio
>      device
>    sound: usb: Prevent starting of audio stream if in use
>    ASoC: dt-bindings: Add Q6USB backend bindings
>    ASoC: dt-bindings: Update example for enabling USB offload on SM8250
>    ASoC: qcom: qdsp6: q6afe: Split USB AFE dev_token param into separate
>      API
>    sound: Pass USB SND card and PCM information to SOC USB
>    sound: soc: qdsp6: Add SND kcontrol to select offload device
>    sound: soc: qdsp6: Add SND kcontrol for fetching offload status
>    sound: soc: qcom: q6usb: Add headphone jack for offload connection
>      status
>    sound: usb: qc_audio_offload: Use card and PCM index from QMI request
>    sound: usb: card: Allow for rediscovery of connected USB SND devices
>    sound: soc: soc-usb: Rediscover USB SND devices on USB port add
> 
>   .../bindings/sound/qcom,q6usb-dais.yaml       |   49 +
>   .../bindings/sound/qcom,sm8250.yaml           |   15 +
>   .../devicetree/bindings/usb/snps,dwc3.yaml    |   13 +
>   arch/arm64/boot/dts/qcom/sm8350.dtsi.rej      |   14 +
>   arch/arm64/configs/defconfig.rej              |   24 +
>   drivers/usb/dwc3/core.c                       |   12 +
>   drivers/usb/dwc3/core.h                       |    2 +
>   drivers/usb/dwc3/host.c                       |    5 +-
>   drivers/usb/host/Kconfig                      |    9 +
>   drivers/usb/host/Makefile                     |    4 +
>   drivers/usb/host/xhci-debugfs.c               |    2 +-
>   drivers/usb/host/xhci-hub.c                   |   29 +-
>   drivers/usb/host/xhci-mem.c                   |  167 +-
>   drivers/usb/host/xhci-plat.c                  |    2 +
>   drivers/usb/host/xhci-ring.c                  |    2 +-
>   drivers/usb/host/xhci-sideband.c              |  195 ++
>   drivers/usb/host/xhci.c                       |  109 +-
>   drivers/usb/host/xhci.h                       |  102 +-
>   .../sound/qcom,q6dsp-lpass-ports.h            |    1 +
>   include/linux/usb/xhci-intr.h                 |   86 +
>   include/linux/usb/xhci-sideband.h             |   90 +
>   include/sound/q6usboffload.h                  |   20 +
>   include/sound/soc-usb.h                       |   39 +
>   sound/core/oss/pcm_oss.c                      |    4 +-
>   sound/soc/Makefile                            |    2 +-
>   sound/soc/qcom/Kconfig                        |    4 +
>   sound/soc/qcom/qdsp6/Makefile                 |    1 +
>   sound/soc/qcom/qdsp6/q6afe-dai.c              |   50 +
>   sound/soc/qcom/qdsp6/q6afe.c                  |  208 +-
>   sound/soc/qcom/qdsp6/q6afe.h                  |   47 +-
>   sound/soc/qcom/qdsp6/q6dsp-lpass-ports.c      |   23 +
>   sound/soc/qcom/qdsp6/q6dsp-lpass-ports.h      |    1 +
>   sound/soc/qcom/qdsp6/q6routing.c              |    9 +
>   sound/soc/qcom/qdsp6/q6usb.c                  |  458 +++++
>   sound/soc/soc-usb.c                           |  197 ++
>   sound/usb/Kconfig                             |   15 +
>   sound/usb/Makefile                            |    2 +-
>   sound/usb/card.c                              |   85 +
>   sound/usb/card.h                              |   32 +
>   sound/usb/endpoint.c                          |    2 +
>   sound/usb/helper.c                            |    1 +
>   sound/usb/pcm.c                               |   29 +-
>   sound/usb/pcm.h                               |   12 +
>   sound/usb/qcom/Makefile                       |    2 +
>   sound/usb/qcom/qc_audio_offload.c             | 1820 +++++++++++++++++
>   sound/usb/qcom/usb_audio_qmi_v01.c            |  892 ++++++++
>   sound/usb/qcom/usb_audio_qmi_v01.h            |  162 ++
>   47 files changed, 4870 insertions(+), 179 deletions(-)
>   create mode 100644 Documentation/devicetree/bindings/sound/qcom,q6usb-dais.yaml
>   create mode 100644 arch/arm64/boot/dts/qcom/sm8350.dtsi.rej
>   create mode 100644 arch/arm64/configs/defconfig.rej
>   create mode 100644 drivers/usb/host/xhci-sideband.c
>   create mode 100644 include/linux/usb/xhci-intr.h
>   create mode 100644 include/linux/usb/xhci-sideband.h
>   create mode 100644 include/sound/q6usboffload.h
>   create mode 100644 include/sound/soc-usb.h
>   create mode 100644 sound/soc/qcom/qdsp6/q6usb.c
>   create mode 100644 sound/soc/soc-usb.c
>   create mode 100644 sound/usb/qcom/Makefile
>   create mode 100644 sound/usb/qcom/qc_audio_offload.c
>   create mode 100644 sound/usb/qcom/usb_audio_qmi_v01.c
>   create mode 100644 sound/usb/qcom/usb_audio_qmi_v01.h
> 

Powered by blists - more mailing lists