linux-kernel - Re: [PATCH v1] driver core: Extend deferred probe timeout on driver registration

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives
Hash Suite: Windows password security audit tool. GUI, reports in PDF.
[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]
Message-ID: <Youleo3Ganxbc1sq@dev-arch.thelio-3990X>
Date:   Mon, 23 May 2022 08:17:14 -0700
From:   Nathan Chancellor <nathan@...nel.org>
To:     Saravana Kannan <saravanak@...gle.com>
Cc:     Jonathan Corbet <corbet@....net>,
        Greg Kroah-Hartman <gregkh@...uxfoundation.org>,
        "Rafael J. Wysocki" <rafael@...nel.org>,
        "Rafael J. Wysocki" <rjw@...ysocki.net>,
        Rob Herring <robh@...nel.org>,
        Linus Walleij <linus.walleij@...aro.org>,
        Will Deacon <will@...nel.org>,
        Ulf Hansson <ulf.hansson@...aro.org>,
        Kevin Hilman <khilman@...nel.org>,
        Thierry Reding <treding@...dia.com>,
        Mark Brown <broonie@...nel.org>, Pavel Machek <pavel@....cz>,
        Geert Uytterhoeven <geert@...ux-m68k.org>,
        Yoshihiro Shimoda <yoshihiro.shimoda.uh@...esas.com>,
        Paul Kocialkowski <paul.kocialkowski@...tlin.com>,
        linux-gpio@...r.kernel.org, linux-pm@...r.kernel.org,
        iommu@...ts.linux-foundation.org, kernel-team@...roid.com,
        linux-doc@...r.kernel.org, linux-kernel@...r.kernel.org,
        linux-s390@...r.kernel.org
Subject: Re: [PATCH v1] driver core: Extend deferred probe timeout on driver
 registration

On Fri, May 20, 2022 at 05:15:55PM -0700, Saravana Kannan wrote:
> On Fri, May 20, 2022 at 5:04 PM Nathan Chancellor <nathan@...nel.org> wrote:
> >
> > On Fri, May 20, 2022 at 04:49:48PM -0700, Saravana Kannan wrote:
> > > On Fri, May 20, 2022 at 4:30 PM Nathan Chancellor <nathan@...nel.org> wrote:
> > > >
> > > > Hi Saravana,
> > > >
> > > > On Fri, Apr 29, 2022 at 03:09:32PM -0700, Saravana Kannan wrote:
> > > > > The deferred probe timer that's used for this currently starts at
> > > > > late_initcall and runs for driver_deferred_probe_timeout seconds. The
> > > > > assumption being that all available drivers would be loaded and
> > > > > registered before the timer expires. This means, the
> > > > > driver_deferred_probe_timeout has to be pretty large for it to cover the
> > > > > worst case. But if we set the default value for it to cover the worst
> > > > > case, it would significantly slow down the average case. For this
> > > > > reason, the default value is set to 0.
> > > > >
> > > > > Also, with CONFIG_MODULES=y and the current default values of
> > > > > driver_deferred_probe_timeout=0 and fw_devlink=on, devices with missing
> > > > > drivers will cause their consumer devices to always defer their probes.
> > > > > This is because device links created by fw_devlink defer the probe even
> > > > > before the consumer driver's probe() is called.
> > > > >
> > > > > Instead of a fixed timeout, if we extend an unexpired deferred probe
> > > > > timer on every successful driver registration, with the expectation more
> > > > > modules would be loaded in the near future, then the default value of
> > > > > driver_deferred_probe_timeout only needs to be as long as the worst case
> > > > > time difference between two consecutive module loads.
> > > > >
> > > > > So let's implement that and set the default value to 10 seconds when
> > > > > CONFIG_MODULES=y.
> > > > >
> > > > > Cc: Greg Kroah-Hartman <gregkh@...uxfoundation.org>
> > > > > Cc: "Rafael J. Wysocki" <rjw@...ysocki.net>
> > > > > Cc: Rob Herring <robh@...nel.org>
> > > > > Cc: Linus Walleij <linus.walleij@...aro.org>
> > > > > Cc: Will Deacon <will@...nel.org>
> > > > > Cc: Ulf Hansson <ulf.hansson@...aro.org>
> > > > > Cc: Kevin Hilman <khilman@...nel.org>
> > > > > Cc: Thierry Reding <treding@...dia.com>
> > > > > Cc: Mark Brown <broonie@...nel.org>
> > > > > Cc: Pavel Machek <pavel@....cz>
> > > > > Cc: Geert Uytterhoeven <geert@...ux-m68k.org>
> > > > > Cc: Yoshihiro Shimoda <yoshihiro.shimoda.uh@...esas.com>
> > > > > Cc: Paul Kocialkowski <paul.kocialkowski@...tlin.com>
> > > > > Cc: linux-gpio@...r.kernel.org
> > > > > Cc: linux-pm@...r.kernel.org
> > > > > Cc: iommu@...ts.linux-foundation.org
> > > > > Signed-off-by: Saravana Kannan <saravanak@...gle.com>
> > > >
> > > > I bisected a boot hang with ARCH=s390 defconfig in QEMU down to this
> > > > change as commit 2b28a1a84a0e ("driver core: Extend deferred probe
> > > > timeout on driver registration") in next-20220520 (bisect log below).
> > > >
> > > > $ make -skj"$(nproc)" ARCH=s390 CROSS_COMPILE=s390x-linux-gnu- defconfig bzImage
> > > >
> > > > $ timeout --foreground 15m stdbuf -oL -eL \
> > > > qemu-system-s390x \
> > > > -initrd ... \
> > > > -M s390-ccw-virtio \
> > > > -display none \
> > > > -kernel arch/s390/boot/bzImage \
> > > > -m 512m \
> > > > -nodefaults \
> > > > -serial mon:stdio
> > > > ...
> > > > [    2.077303] In-situ OAM (IOAM) with IPv6
> > > > [    2.077639] NET: Registered PF_PACKET protocol family
> > > > [    2.078063] bridge: filtering via arp/ip/ip6tables is no longer available by default. Update your scripts to load br_netfilter if you need this.
> > > > [    2.078795] Key type dns_resolver registered
> > > > [    2.079317] cio: Channel measurement facility initialized using format extended (mode autodetected)
> > > > [    2.081494] Discipline DIAG cannot be used without z/VM
> > > > [  260.626363] random: crng init done
> > > > qemu-system-s390x: terminating on signal 15 from pid 3815762 (timeout)
> > > >
> > > > We have a simple rootfs available if necessary:
> > > >
> > > > https://github.com/ClangBuiltLinux/boot-utils/raw/bc0d17785eb67f1edd0ee0a134970a807895f741/images/s390/rootfs.cpio.zst
> > > >
> > > > If there is any other information I can provide, please let me know!
> > >
> > > Hmm... strange. Can you please try the following command line options
> > > and tell me which of these has the issue and which don't?
> >
> > Sure thing!
> >
> > > 1) deferred_probe_timeout=0
> >
> > No issue.
> >
> > > 2) deferred_probe_timeout=1
> > > 3) deferred_probe_timeout=300
> >
> > Both of these appear to hang in the same way, I let each sit for five
> > minutes.
> 
> Strange that a sufficiently large timeout isn't helping. Is it trying
> to boot off a network mount? I'll continue looking into this next
> week.

I don't think so, it seems like doing that requires some extra flags
that we do not have:

https://wiki.qemu.org/Features/S390xNetworkBoot

If you need any additional information or want something tested, please
let me know!

Cheers,
Nathan