linux-kernel - Re: Regression in 5.1.20: Reading long directory fails

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Date:   Wed, 28 Aug 2019 13:46:09 -0400
From:   bfields@...ldses.org (J. Bruce Fields)
To:     Jason L Tibbitts III <tibbs@...h.uh.edu>
Cc:     linux-nfs@...r.kernel.org, km@...all.com,
        linux-kernel@...r.kernel.org
Subject: Re: Regression in 5.1.20: Reading long directory fails

On Thu, Aug 22, 2019 at 02:39:26PM -0500, Jason L Tibbitts III wrote:
> I now have another user reporting the same failure of readdir on a long
> directory which showed up in 5.1.20 and was traced to
> 3536b79ba75ba44b9ac1a9f1634f2e833bbb735c.  I'm not sure what to do to
> get more traction besides reposting and adding some addresses to the CC
> list.  If there is any information I can provide which might help to get
> to the bottom of this, please let me know.
> 
> To recap:
> 
> 5.1.20 introduced a regression reading some large directories.  In this
> case, the directory should have 7800 files or so in it:
> 
> [root@...0 ~]# ls -l ~dblecher|wc -l
> ls: reading directory '/home/dblecher': Input/output error
> 1844
> [root@...0 ~]# cat /proc/version Linux version 5.1.20-300.fc30.x86_64 (mockbuild@...rnel04.phx2.fedoraproject.org) (gcc version 9.1.1 20190503 (Red Hat 9.1.1-1) (GCC)) #1 SMP Fri Jul 26 15:03:11 UTC 2019
> 
> (The server is a Centos 7 machine running kernel 3.10.0-957.12.2.el7.x86_64.)
> 
> Building a kernel which reverts commit 3536b79ba75ba44b9ac1a9f1634f2e833bbb735c:
>   Revert "NFS: readdirplus optimization by cache mechanism" (memleak)

Looks like that's db531db951f950b8 upstream.  (Do you know if it's
reproduceable upstream as well?)

> fixes the issue, but of course that revert was fixing a real issue so
> I'm not sure what to do.
> 
> I can trivially reproduce this by simply trying to list the problematic
> directories but I'm not sure how to construct such a directory; simply
> creating 10000 files doesn't cause the problem for me.

Maybe it depends on having names of the right length to place some bit
of xdr on a boundary.  I wonder if it'd be possible to reproduce just by
varying the name lengths randomly till you hit it.

The fact that the problematic patch fixed a memory leak also makes me
wonder if it might have gone to far and freed something out from under
the readdir code.

> I am willing to
> test patches and can build my own kernels, and I'm happy to provide any
> debugging information you might require.  Unfortunately I don't know
> enough to dig in and figure out for myself what's going wrong.
> 
> I did file https://bugzilla.redhat.com/show_bug.cgi?id=1740954 just to
> have this in a bug tracker somewhere.  I'm happy to file one somewhere
> else if that would help.

No clever debugging ideas off the top of my head, I'm afraid.  I might
start by patching the kernel or doing some tracing to figure out exactly
where that EIO is being generated?

--b.