Message-ID: <CAJZ5v0ju58LxvRckv2T=H0D=aDooGUoGfqFze5zWQ1ARAAJcag@mail.gmail.com>
Date:   Fri, 26 Jun 2020 13:27:16 +0200
From:   "Rafael J. Wysocki" <rafael@...nel.org>
To:     Saravana Kannan <saravanak@...gle.com>
Cc:     "Rafael J. Wysocki" <rafael@...nel.org>,
        Greg Kroah-Hartman <gregkh@...uxfoundation.org>,
        Geert Uytterhoeven <geert@...ux-m68k.org>,
        "Cc: Android Kernel" <kernel-team@...roid.com>,
        "Rafael J. Wysocki" <rafael.j.wysocki@...el.com>,
        Linux Kernel Mailing List <linux-kernel@...r.kernel.org>
Subject: Re: [PATCH v1] driver core: Fix suspend/resume order issue with
 deferred probe

On Thu, Jun 25, 2020 at 7:52 PM Saravana Kannan <saravanak@...gle.com> wrote:
>
> On Thu, Jun 25, 2020 at 10:47 AM Rafael J. Wysocki <rafael@...nel.org> wrote:
> >
> > On Thu, Jun 25, 2020 at 7:09 PM Saravana Kannan <saravanak@...gle.com> wrote:
> > >
> > > On Thu, Jun 25, 2020 at 10:03 AM Rafael J. Wysocki <rafael@...nel.org> wrote:
> > > >
> > > > On Thu, Jun 25, 2020 at 7:01 PM Saravana Kannan <saravanak@...gle.com> wrote:
> > > > >
> > > > > On Thu, Jun 25, 2020 at 9:58 AM Rafael J. Wysocki <rafael@...nel.org> wrote:
> > > > > >
> > > > > > On Thu, Jun 25, 2020 at 6:49 PM Saravana Kannan <saravanak@...gle.com> wrote:
> > > > > > >
> > > > > > > Dropping Feng Kan <fkan@....com> and Toan Le <toanle@....com> because
> > > > > > > their mails are bouncing.
> > > > > > >
> > > > > > > On Thu, Jun 25, 2020 at 8:19 AM Rafael J. Wysocki <rafael@...nel.org> wrote:
> > > > > > > >
> > > > > > > > On Thu, Jun 25, 2020 at 5:24 AM Saravana Kannan <saravanak@...gle.com> wrote:
> > > > > > > > >
> > > > > > > > > Under the following conditions:
> > > > > > > > > - driver A is built in and can probe device-A
> > > > > > > > > - driver B is a module and can probe device-B
> > > > > > > > > - device-A is supplier of device-B
> > > > > > > > >
> > > > > > > > > Without this patch:
> > > > > > > > > 1. device-A is added.
> > > > > > > > > 2. device-B is added.
> > > > > > > > > 3. dpm_list is now [device-A, device-B].
> > > > > > > > > 4. driver-A defers probe of device-A.
> > > > > > > > > 5. deferred probe of device-A is reattempted
> > > > > > > > > 6. device-A is moved to end of dpm_list.
> > > > > > > > > 6. dpm_list is now [device-B, device-A].
> > > > > > > > > 7. driver-B is loaded and probes device-B.
> > > > > > > > > 8. dpm_list stays as [device-B, device-A].
> > > > > > > > >
> > > > > > > > > Suspend (which goes in the reverse order of dpm_list) fails because
> > > > > > > > > device-A (supplier) is suspended before device-B (consumer).
> > > > > > > > >
> > > > > > > > > With this patch:
> > > > > > > > > 1. device-A is added.
> > > > > > > > > 2. device-B is added.
> > > > > > > > > 3. dpm_list is now [device-A, device-B].
> > > > > > > > > 4. driver-A defers probe of device-A.
> > > > > > > > > 5. deferred probe of device-A is reattempted later.
> > > > > > > > > 6. dpm_list is now [device-B, device-A].
> > > > > > > > > 7. driver-B is loaded and probes device-B.
> > > > > > > > > 8. dpm_list is now [device-A, device-B].
> > > > > > > > >
> > > > > > > > > Suspend works because device-B (consumer) is suspended before device-A
> > > > > > > > > (supplier).
> > > > > > > > >
> > > > > > > > > Fixes: 494fd7b7ad10 ("PM / core: fix deferred probe breaking suspend resume order")
> > > > > > > > > Fixes: 716a7a259690 ("driver core: fw_devlink: Add support for batching fwnode parsing")
> > > > > > > > > Cc: Geert Uytterhoeven <geert@...ux-m68k.org>
> > > > > > > > > Signed-off-by: Saravana Kannan <saravanak@...gle.com>
> > > > > > > > > ---
> > > > > > > > >  drivers/base/dd.c | 16 ++++++++++++++++
> > > > > > > > >  1 file changed, 16 insertions(+)
> > > > > > > > >
> > > > > > > > > diff --git a/drivers/base/dd.c b/drivers/base/dd.c
> > > > > > > > > index 9a1d940342ac..52b2148c7983 100644
> > > > > > > > > --- a/drivers/base/dd.c
> > > > > > > > > +++ b/drivers/base/dd.c
> > > > > > > > > @@ -109,6 +109,8 @@ static void deferred_probe_work_func(struct work_struct *work)
> > > > > > > > >                  * probe makes that very unsafe.
> > > > > > > > >                  */
> > > > > > > > >                 device_pm_move_to_tail(dev);
> > > > > > > > > +               /* Greg/Rafael: SHOULD I DELETE THIS? ^^ I think I should, but
> > > > > > > > > +                * I'm worried if it'll have some unintended consequeneces. */
> > > > > > > >
> > > > > > > > Yes, this needs to go away if you make the other change.
> > > > > > > >
> > > > > > > > >
> > > > > > > > >                 dev_dbg(dev, "Retrying from deferred list\n");
> > > > > > > > >                 bus_probe_device(dev);
> > > > > > > > > @@ -557,6 +559,20 @@ static int really_probe(struct device *dev, struct device_driver *drv)
> > > > > > > > >                 goto re_probe;
> > > > > > > > >         }
> > > > > > > > >
> > > > > > > > > +       /*
> > > > > > > > > +        * The devices are added to the dpm_list (resume/suspend (reverse
> > > > > > > > > +        * order) list) as they are registered with the driver core. But the
> > > > > > > > > +        * order the devices are added doesn't necessarily match the real
> > > > > > > > > +        * dependency order.
> > > > > > > > > +        *
> > > > > > > > > +        * The successful probe order is a much better signal. If a device just
> > > > > > > > > +        * probed successfully, then we know for sure that all the devices that
> > > > > > > > > +        * probed before it don't depend on the device. So, we can safely move
> > > > > > > > > +        * the device to the end of the dpm_list. As more devices probe,
> > > > > > > > > +        * they'll automatically get ordered correctly.
> > > > > > > > > +        */
> > > > > > > > > +       device_pm_move_to_tail(dev);
> > > > > > > >
> > > > > > > > But it would be good to somehow limit this to the devices affected by
> > > > > > > > deferred probing or we'll end up reordering dpm_list unnecessarily for
> > > > > > > > many times in the actual majority of cases.
> > > > > > >
> > > > > > > Yes, lots of unnecessary reordering, but doing it only for deferred
> > > > > > > probes IS the problem. In the example I gave, the consumer is never
> > > > > > > deferred probe because the supplier happens to finish probing before
> > > > > > > the consumer probe is even attempted.
> > > > > >
> > > > > > But why would the supplier be moved to the end of dpm_list without
> > > > > > moving the consumer along with it?
> > > > >
> > > > > There is no device link between the supplier/consumer in this case.
> > > >
> > > > So this is the real problem, isn't it?
> > > >
> > > > > Sadly there are plenty of cases where device links aren't present to
> > > > > capture supplier/consumer dependencies.
> > > >
> > > > And so that's why you want to add a ton of overhead to driver probing
> > > > in all of the cases in which that is not an issue?
> > >
> > > Well, until all/most of the frameworks add device links or
> > > fw_devlink=on by default, it doesn't hurt to have suspend/resume work
> > > in more platforms.
> >
> > In the presence of deferred probing, that is.
> >
> > Note that deferred probing gets in the way here and so the problem is
> > related to it.
>
> I mean, we officially support deferred probing. Shouldn't we fix it so
> that it doesn't break suspend/resume?

Yes, we should fix deferred probing.

> Also, it's pretty easy to have
> cases where one module probes multiple device instances and loading it
> in one order would break dpm_list order for one device and loading it
> in another order would break it for another device. And there would be
> no "proper" order to load modules (because module order != device
> order).

I'm not saying that the current code is perfect.  I'm saying that the
fix as proposed adds too much cost for everybody who may not care IMO.

>
> > > What about the option of not adding to dpm_list until a device is
> > > probed? Is it DOA?
> >
> > Yes, it is, I'm afraid.  There are devices without drivers. :-)
>
> Devices that still suspend/resume without drivers?! I didn't know that
> was possible.

There are "class" devices, "type" devices and devices that simply have
no drivers, but the bus type code may still want to touch them during
system-wide suspend/resume.

Modules may still not be loaded when the system is suspended for the
first time etc.

Powered by blists - more mailing lists