linux-kernel - Re: autofs mount/umount hangs with recent kernel?

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [day] [month] [year] [list]

Message-ID: <CAPt2mGNYQwd6S90NXU8p+SDufCUmTLqKQb8MuB+s-hr=1g3uJw@mail.gmail.com>
Date:   Mon, 6 Nov 2023 11:05:25 +0000
From:   Daire Byrne <daire@...g.com>
To:     Bagas Sanjaya <bagasdotme@...il.com>
Cc:     Linux NFS <linux-nfs@...r.kernel.org>,
        Linux Kernel Mailing List <linux-kernel@...r.kernel.org>,
        Linux Regressions <regressions@...ts.linux.dev>,
        Chuck Lever <chuck.lever@...cle.com>,
        Jeff Layton <jlayton@...nel.org>,
        Trond Myklebust <trond.myklebust@...merspace.com>,
        Anna Schumaker <anna@...nel.org>
Subject: Re: autofs mount/umount hangs with recent kernel?

On Fri, 3 Nov 2023 at 14:10, Bagas Sanjaya <bagasdotme@...il.com> wrote:
>
> On Fri, Nov 03, 2023 at 09:40:44AM +0000, Daire Byrne wrote:
> > Hi,
> >
> > We have large compute clusters that, amongst other things, spend their
> > day mounting & unounting lots of Linux NFS servers via autofs. This
> > has worked fine for many years and client kernel versions and was
> > working without incident even with our current v6.3.x production
> > kernels.
> >
> > During the v6.6-rc cycle while testing that kernel, I noticed that
> > every now and then, the umount/mount would hang randomly and the
> > compute host would get stuck and not complete it's work until a
> > reboot. I thought I'd wait until v6.6 was released and check again -
> > the issue persists.
> >
> > I have not had the opportunity to test the v6.4 & v6.5 kernels in
> > between yet. The stack traces look something like this:
>
> Please do bisection to find the exact commit that introduces your
> regression. See Documentation/admin-guide/bug-bisect.rst in the kernel
> sources for more information.

I tested the v6.5.10 kernel and I can't reproduce with that, so I
guess I'll start the process of bisecting between v6.5.10 & v6.6.

It takes around 24 hours and 72 client hosts until we see it happen on
at least one, so it'll be slow going.

Weirdly, I also have the v6.6 kernel running on a different set of
hosts and they have not shown the problem at all. The only difference
being AMD clients with higher process concurrency = bad, and Intel
clients with less process concurrency = okay.

> >
> > [202752.264187] INFO: task umount.nfs:58118 blocked for more than 245 seconds.
> > [202752.264237]       Tainted: G            E      6.6.0-1.dneg.x86_64 #1
>
> Can you reproduce on untainted (vanilla) kernel?

Yea, I'm not sure what is going on there. Some compile issue as it's
the "deflate" module complaining about a missing key.

> > And like I said, the mount/umount against the server hangs
> > indefinitely on the client. It is somewhat interesting that autofs
> > still tries to trigger a subsequent mount even though the umount has
> > not completed.
> >
> > The NFS servers are running RHEL8.5 and we are using NFSv3. I also
> > reproduced it with a fairly recent nfs-utils-2.6.2 on the client
> > compute hosts.
>
> What distro on the client?

It's EL79 on the client but I have updated the nfs-utils to something
more modern. I have not updated the autofs package though.

I'll report back in a week or so...

Cheers,

Daire