Message-ID: <CAJZ5v0jn9-vsJZdH2v0yh8axmJmqOwkN=vuqs=Re=Nnf0WWkLA@mail.gmail.com>
Date:   Mon, 12 Jun 2023 17:37:18 +0200
From:   "Rafael J. Wysocki" <rafael@...nel.org>
To:     Alan Stern <stern@...land.harvard.edu>,
        Damien Le Moal <dlemoal@...nel.org>
Cc:     Kai-Heng Feng <kai.heng.feng@...onical.com>,
        Hannes Reinecke <hare@...e.de>,
        Bart Van Assche <bvanassche@....org>,
        Bagas Sanjaya <bagasdotme@...il.com>,
        Pavel Machek <pavel@....cz>,
        "Rafael J. Wysocki" <rafael@...nel.org>,
        Len Brown <len.brown@...el.com>,
        Greg Kroah-Hartman <gregkh@...uxfoundation.org>,
        Kees Cook <keescook@...omium.org>,
        Tony Luck <tony.luck@...el.com>,
        "Guilherme G. Piccoli" <gpiccoli@...lia.com>,
        Thorsten Leemhuis <linux@...mhuis.info>,
        "James E.J. Bottomley" <jejb@...ux.ibm.com>,
        "Martin K. Petersen" <martin.petersen@...cle.com>,
        Phillip Potter <phil@...lpotter.co.uk>,
        Joe Breuer <linux-kernel@...reuer.net>,
        Linux Power Management <linux-pm@...r.kernel.org>,
        Linux Kernel Mailing List <linux-kernel@...r.kernel.org>,
        Linux Hardening <linux-hardening@...r.kernel.org>,
        Linux Regressions <regressions@...ts.linux.dev>,
        Linux SCSI <linux-scsi@...r.kernel.org>,
        Dan Williams <dan.j.williams@...el.com>,
        Hannes Reinecke <hare@...e.com>,
        Adrian Hunter <adrian.hunter@...el.com>,
        Martin Kepplinger <martin.kepplinger@...i.sm>
Subject: Re: Fwd: Waking up from resume locks up on sr device

On Mon, Jun 12, 2023 at 4:33 PM Alan Stern <stern@...land.harvard.edu> wrote:
>
> On Mon, Jun 12, 2023 at 04:47:09PM +0900, Damien Le Moal wrote:
> > On 6/12/23 16:36, Kai-Heng Feng wrote:
> > > On Mon, Jun 12, 2023 at 3:22 PM Damien Le Moal <dlemoal@...nel.org> wrote:
> > >>
> > >> On 6/12/23 15:09, Hannes Reinecke wrote:
> > >>> On 6/12/23 05:09, Damien Le Moal wrote:
> > >>>> On 6/11/23 00:03, Bart Van Assche wrote:
> > >>>>> On 6/10/23 06:27, Bagas Sanjaya wrote:
> > >>>>>> On 6/10/23 15:55, Pavel Machek wrote:
> > >>>>>>>>> #regzbot introduced: v5.0..v6.4-rc5 https://bugzilla.kernel.org/show_bug.cgi?id=217530
> > >>>>>>>>> #regzbot title: Waking up from resume locks up on SCSI CD/DVD drive
> > >>>>>>>>>
> > >>>>>>>> The reporter had found the culprit (via bisection), so:
> > >>>>>>>>
> > >>>>>>>> #regzbot introduced: a19a93e4c6a98c
> > >>>>>>> Maybe cc the authors of that commit?
> > >>>>>>
> > >>>>>> Ah! I forgot to do that! Thanks anyway.
> > >>>>>
> > >>>>> Hi Damien,
> > >>>>>
> > >>>>> Why does the ATA code call scsi_rescan_device() before system resume has
> > >>>>> finished? Would ATA devices still work with the patch below applied?
> > >>>>
> > >>>> I do not know the PM code well at all, need to dig into it. But your patch
> > >>>> worries me as it seems it would prevent rescan of the device on a resume, which
> > >>>> can be an issue if the device has changed.
> > >>>>
> > >>>> I am not yet 100% clear on the root cause for this, but I think it comes from
> > >>>> the fact that ata_port_pm_resume() runs before the sci device resume is done, so
> > >>>> with scsi_dev->power.is_suspended still true. And ata_port_pm_resume() calls
> > >>>> ata_port_resume_async() which triggers EH (which will do reset + rescan)
> > >>>> asynchronously. So it looks like we have scsi device resume and libata EH for
> > >>>> rescan fighting each others for the scan mutex and device lock, leading to deadlock.
> > >>>>
> > >>>> Trying to recreate this issue now to confirm and debug further. But I suspect
> > >>>> the solution to this may be best implemented in libata, not in scsi.
> > >>>> This looks definitely related to this thread:
> > >>>>
> > >>>> https://lore.kernel.org/linux-scsi/7b553268-69d3-913a-f9de-28f8d45bdb1e@acm.org/
> > >>>>
> > >>>> Similaraly to your comment on that thread, having to look at
> > >>>> dev->power.is_suspended is not ideal I think. What we need is to have ata and
> > >>>> scsi pm resume be synchronized, but I am not yet 100% clear on the scsi layer side.
> > >>>>
> > >>> Which is my feeling, too.
> > >>> libata runs rescan as part of the device discovery, so really it will
> > >>> run after resume. And consequently resume really cannot wait for rescan
> > >>> to finish.
> > >>>
> > >>> What I would be looking at is to decouple resume from libata device
> > >>> rescan, and have resume to complete before libata EH runs.
> > >>
> > >> That is the case now, for the ata port at least, even though that is not super
> > >> explicit, and not reliable. See ata_port_pm_resume(): I think that the call to
> > >> EH in ata_port_pm_resume() -> ata_port_resume_async() -> ata_port_request_pm()
> > >> -> ata_port_schedule_eh() should instead use a sync resume, leading to a sync EH
> > >> call.
> > >>
> > >> That EH execution essentially does ata_eh_handle_port_resume(), which calls into
> > >> the adapter resume operation. That in itself does not do much beside some
> > >> registers accesses to wakeup the port. There should be no issues doing that
> > >> synchronously.
> > >>
> > >> The problem is that after that is done, ata EH calls ata_std_error_handler() ->
> > >> ata_do_eh() -> ata_eh_recover() -> ata_eh_revalidate_and_attach() ->
> > >> schedule_work(&(ap->scsi_rescan_task)). And the rescan work calls
> > >> scsi_rescan_device() (yet in another context than EH) which causes the problem
> > >> when the scsi disk device has not been resumed yet (dev->power_is_suspended
> > >> still true).
> > >>
> > >> So it really looks like the solution should be to have ata_scsi_dev_rescan()
> > >> wait for the scsi device to resume first, but not sure how to do that with the
> > >> pm API. Digging...
> > >
> > > Probably use dpm_wait_for_children()? Right now it's an internal PM API.
> >
> > But I am not sure if there is a relationship between ata_device and its
> > scsi_device (dev->sdev)... Need to clarify that.
> > >
> > > Rafael,
> > > What do you think?
>
> Look into the device_pm_wait_for_dev() API.  It's the appropriate thing
> to use when you want to wait for another device to complete a system PM
> transition.  (However, it's not appropriate for runtime PM.)
>
> Of course, if there is a parent-child relationship between two devices
> then waiting is never necessary.  The PM core guarantees that a parent
> will always be at full power when a child changes its power state,
> unless pm_suspend_ignore_children() has been called for the parent
> device.

There are also device links that go beyond the parent-child handling.
Please see Documentation/driver-api/device_link.rst and the kerneldoc
description of device_link_add() for more information.

Powered by blists - more mailing lists