Message-ID: <PUZPR03MB613101A170B0034F55401121A1289@PUZPR03MB6131.apcprd03.prod.outlook.com>
Date:   Tue, 18 Oct 2022 20:49:28 +0000
From:   Jeffrey Vanhoof <jvanhoof@...orola.com>
To:     "thinh.nguyen@...opsys.com" <thinh.nguyen@...opsys.com>
CC:     "balbi@...nel.org" <balbi@...nel.org>,
        "corbet@....net" <corbet@....net>,
        "dan.scally@...asonboard.com" <dan.scally@...asonboard.com>,
        "gregkh@...uxfoundation.org" <gregkh@...uxfoundation.org>,
        "laurent.pinchart@...asonboard.com" 
        <laurent.pinchart@...asonboard.com>,
        "linux-doc@...r.kernel.org" <linux-doc@...r.kernel.org>,
        "linux-kernel@...r.kernel.org" <linux-kernel@...r.kernel.org>,
        "linux-usb@...r.kernel.org" <linux-usb@...r.kernel.org>,
        "m.grzeschik@...gutronix.de" <m.grzeschik@...gutronix.de>,
        "paul.elder@...asonboard.com" <paul.elder@...asonboard.com>,
        Jeffrey Vanhoof <jvanhoof@...orola.com>,
        "stable@...r.kernel.org" <stable@...r.kernel.org>,
        Dan Vacura <W36195@...orola.com>
Subject: Re: [PATCH v3 2/6] usb: dwc3: gadget: cancel requests instead of
 release after missed isoc

>From qjv001@...001-XeonWs Tue Oct 18 15:37:29 2022
From: qjv001 <qjv001@...001-XeonWs>
To: Thinh Nguyen <Thinh.Nguyen@...opsys.com>
Subject: Re: [PATCH v3 2/6] usb: dwc3: gadget: cancel requests instead of
 release after missed isoc
References: <20221017205446.523796-1-w36195@...orola.com>
 <20221017205446.523796-3-w36195@...orola.com>
 <20221017213031.tqb575hdzli7jlbh@...opsys.com>
 <Y04K/HoUigF5FYBA@...3>
 <20221018184535.3g3sm35picdeuajs@...opsys.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <20221018184535.3g3sm35picdeuajs@...opsys.com>
X-Mutt-References: <20221018184535.3g3sm35picdeuajs@...opsys.com>
X-Mutt-Fcc: ~/sent
Status: RO
Date: Tue, 18 Oct 2022 15:37:29 -0500
Content-Length: 5434
Lines: 124

Hi Thinh,

On Tue, Oct 18, 2022 at 06:45:40PM +0000, Thinh Nguyen wrote:
> Hi Dan,
> 
> On Mon, Oct 17, 2022, Dan Vacura wrote:
> > Hi Thinh,
> > 
> > On Mon, Oct 17, 2022 at 09:30:38PM +0000, Thinh Nguyen wrote:
> > > On Mon, Oct 17, 2022, Dan Vacura wrote:
> > > > From: Jeff Vanhoof <qjv001@...orola.com>
> > > > 
> > > > arm-smmu related crashes seen after a Missed ISOC interrupt when
> > > > no_interrupt=1 is used. This can happen if the hardware is still using
> > > > the data associated with a TRB after the usb_request's ->complete call
> > > > has been made.  Instead of immediately releasing a request when a Missed
> > > > ISOC interrupt has occurred, this change will add logic to cancel the
> > > > request instead where it will eventually be released when the
> > > > END_TRANSFER command has completed. This logic is similar to some of the
> > > > cleanup done in dwc3_gadget_ep_dequeue.
> > > 
> > > This doesn't sound right. How did you determine that the hardware is
> > > still using the data associated with the TRB? Did you check the TRB's
> > > HWO bit?
> > 
> > The problem we're seeing was mentioned in the summary of this patch
> > series, issue #1. Basically, with the following patch
> > https://urldefense.com/v3/__https://patchwork.kernel.org/project/linux-usb/patch/20210628155311.16762-6-m.grzeschik@pengutronix.de/__;!!A4F2R9G_pg!aSNZ-IjMcPgL47A4NR5qp9qhVlP91UGTuCxej5NRTv8-FmTrMkKK7CjNToQQVEgtpqbKzLU2HXET9O226AEN$  
> > integrated a smmu panic is occurring on our Android device with the 5.15
> > kernel which is:
> > 
> >     <3>[  718.314900][  T803] arm-smmu 15000000.apps-smmu: Unhandled arm-smmu context fault from a600000.dwc3!
> > 
> > The uvc gadget driver appears to be the first (and only) gadget that
> > uses the no_interrupt=1 logic, so this seems to be a new condition for
> > the dwc3 driver. In our configuration, we have up to 64 requests and the
> > no_interrupt=1 for up to 15 requests. The list size of dep->started_list
> > would get up to that amount when looping through to cleanup the
> > completed requests. From testing and debugging the smmu panic occurs
> > when a -EXDEV status shows up and right after
> > dwc3_gadget_ep_cleanup_completed_request() was visited. The conclusion
> > we had was the requests were getting returned to the gadget too early.
> 
> As I mentioned, if the status is updated to missed isoc, that means that
> the controller returned ownership of the TRB to the driver. At least for
> the particular request with -EXDEV, its TRBs are completed. I'm not
> clear on your conclusion.
> 
> Do we know where did the crash occur? Is it from dwc3 driver or from uvc
> driver, and at what line? It'd great if we can see the driver log.
>

To interject, what should happen in dwc3_gadget_ep_reclaim_completed_trb if the
IOC bit is not set (but the IMI bit is) and -EXDEV status is passed into it?
If the function returns 0, another attempt to reclaim may occur. If this
happens and the next request did have the HWO bit set, the function would
return 1 but dwc3_gadget_ep_cleanup_completed_request would still call
dwc3_gadget_giveback.

As a test (without this patch), I added a check to see if HWO bit was set in
dwc3_gadget_ep_cleanup_completed_requests(). If the usecase was ISOC and the
HWO bit was set I avoided calling dwc3_gadget_ep_cleanup_completed_request().
This seemed to also avoid the iommu related crash being seen.

Is there an issue in this area that needs to be corrected instead? Not having
interrupts set for each request may be causing some new issues to be uncovered.

As far as the crash seen without this patch, no good stacktrace is given. Line
provided for crash varied a bit, but tended to appear towards the end of
dwc3_stop_active_transfer() or dwc3_gadget_endpoint_trbs_complete().

Since dwc3_gadget_endpoint_trbs_complete() can be called from multiple
locations, I duplicated the function to help identify which path it was likely
being called from. At the time of the crashes seen,
dwc3_gadget_endpoint_transfer_in_progress() appeared to be the caller.

dwc3_gadget_endpoint_transfer_in_progress()
->dwc3_gadget_endpoint_trbs_complete() (crashed towards end of here)
->dwc3_stop_active_transfer() (sometimes crashed towards end of here)

I hope this clarifies things a bit.
 
> > 
> > > 
> > > The dwc3 driver would only give back the requests if the TRBs of the
> > > associated requests are completed or when the device is disconnected.
> > > If the TRB indicated missed isoc, that means that the TRB is completed
> > > and its status was updated.
> > 
> > Interesting, the device is not disconnected as we don't get the
> > -ESHUTDOWN status back and with this patch in place things continue
> > after a -EXDEV status is received.
> > 
> 
> Actually, minor correction here: a recent change
> b44c0e7fef51 ("usb: dwc3: gadget: conditionally remove requests")
> changed -ESHUTDOWN request status to -ECONNRESET when disable endpoint.
> This doesn't look right.
> 
> While disabling endpoint may also apply for other cases such as
> switching alternate interface in addition to disconnect, -ESHUTDOWN
> seems more fitting there.
> 
btw, we don't have "usb: dwc3: gadget: conditionally remove requests" in our baseline

> Hi Michael,
> 
> Can you help clarify for the change above? This changed the usage of
> requests. Now requests returned by disconnection won't be returned as
> -ESHUTDOWN.
> 
> > > 
> > > There's a special case which dwc3 may give back requests early is the
> > > case of the device disconnecting. The requests should be returned with
> > > -ESHUTDOWN, and the gadget driver shouldn't be re-using the requests on
> > > de-initialization anyway.
> > > 
> > > We should not issue End Transfer command just because of missed isoc. We
> > > may want issue End Transfer if the gadget driver is too slow and unable
> > > to feed requests in time (causing underrun and missed isoc) to resync
> > > with the host, but we already handle that.
> > 
> > Hmm, isn't that what happens when we get into this
> > condition in dwc3_gadget_endpoint_trbs_complete():
> > 
> > 	if (usb_endpoint_xfer_isoc(dep->endpoint.desc) &&
> > 		list_empty(&dep->started_list) &&
> > 		(list_empty(&dep->pending_list) || status == -EXDEV))
> > 		dwc3_stop_active_transfer(dep, true, true);
> > 
> 
> Yes, it's being handled there.
> 
> > > 
> > > I'm still not clear what's the problem you're seeing. Do you have the
> > > crash log? Tracepoints?
> > > 
> > 
> > Appreciate the support!
> > 
> 
> Thanks,
> Thinh

Thanks,
Jeff

Powered by blists - more mailing lists