netdev - [bisected] Re: odd NFS v4.2 regression 4.4..5.15

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives
Hash Suite: Windows password security audit tool. GUI, reports in PDF.
[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]
Message-ID: <0100fc6eb38b745f45cc614cee30f332c1b6456d.camel@gmx.de>
Date:   Wed, 13 Oct 2021 08:35:29 +0200
From:   Mike Galbraith <efault@....de>
To:     netdev <netdev@...r.kernel.org>
Cc:     Jeff Layton <jeff.layton@...marydata.com>,
        Trond Myklebust <trond.myklebust@...marydata.com>
Subject: [bisected] Re: odd NFS v4.2 regression 4.4..5.15

On Tue, 2021-10-12 at 15:54 +0200, Mike Galbraith wrote:
> Greetings network wizards,
>
> I seem to have stumbled across a kind of odd looking NFS regression
> while looking at something unrelated.
>
> I have a few VMs that are mirrors of my desktop box, including some
> data, but they usually over-mount directories with tons of data via
> NFS, thus hang off the host like a mini-me Siamese twin for testing.
>
> In the data below, I run a script in git-daemon's home to extract RT
> patch sets with git-format-patch, on the host for reference, on the
> host in an NFS mount of itself, and the mini-me VM, where I started.
>
> Note: bonnie throughput looked fine for v4.2.
>
> host
> time su - git -c ./format-rt-patches.sh
> real    1m5.923s
> user    0m54.692s
> sys     0m11.550s
>
> NFS mount my own box, and do the same from the same spot
> time su git -c 'cd /homer/home/git;./format-rt-patches.sh'
>
> 4.4.231     v4.2        v3
> 1   real    2m27.046s   2m2.059s
>     user    0m59.190s   0m58.701s
>     sys     0m41.448s   0m32.541s
>
> 5.15-rc5    v4.2        v3
>    real     3m14.954s   2m8.366s
>    user     0m59.901s   0m58.317s
>    sys      0m52.708s   0m32.313s
>         vs 1   0.754       0.951
>
> repeats     v4.2        v3
>    real     3m16.313s   2m7.940s
>    real     3m10.905s   2m8.029s

Found the v4.2 regression (for this load anyway) to have landed in 5.4.
It then bisected cleanly to either the build bug or its neighbor.

There are only 'skip'ped commits left to test.
The first bad commit could be any of:
fd4f83fd7dfb1bce2f1af51fcbaf6575f4b9d189
eb82dd393744107ebc365a53e7813c7c67cb203b
We cannot bisect more!

git bisect start
# good: [4d856f72c10ecb060868ed10ff1b1453943fc6c8] Linux 5.3
git bisect good 4d856f72c10ecb060868ed10ff1b1453943fc6c8
# bad: [219d54332a09e8d8741c1e1982f5eae56099de85] Linux 5.4
git bisect bad 219d54332a09e8d8741c1e1982f5eae56099de85
# good: [a9f8b38a071b468276a243ea3ea5a0636e848cf2] Merge tag 'for-linus-5.4-1' of git://github.com/cminyard/linux-ipmi
git bisect good a9f8b38a071b468276a243ea3ea5a0636e848cf2
# good: [5825a95fe92566ada2292a65de030850b5cff1da] Merge tag 'selinux-pr-20190917' of git://git.kernel.org/pub/scm/linux/kernel/git/pcmoore/selinux
git bisect good 5825a95fe92566ada2292a65de030850b5cff1da
# bad: [48acba989ed5d8707500193048d6c4c5945d5f43] Merge tag 'riscv/for-v5.4-rc3' of git://git.kernel.org/pub/scm/linux/kernel/git/riscv/linux
git bisect bad 48acba989ed5d8707500193048d6c4c5945d5f43
# good: [7bccb9f10c8f36ee791769b531ed4d28f6379aae] Merge tag 'linux-watchdog-5.4-rc1' of git://www.linux-watchdog.org/linux-watchdog
git bisect good 7bccb9f10c8f36ee791769b531ed4d28f6379aae
# bad: [ef129d34149ea23d0d442844fc25ae26a85589fc] selftests/net: add nettest to .gitignore
git bisect bad ef129d34149ea23d0d442844fc25ae26a85589fc
# bad: [9c5efe9ae7df78600c0ee7bcce27516eb687fa6e] Merge branch 'sched-urgent-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip
git bisect bad 9c5efe9ae7df78600c0ee7bcce27516eb687fa6e
# good: [8bbe0dec38e147a50e9dd5f585295f7e68e0f2d0] Merge tag 'for-linus' of git://git.kernel.org/pub/scm/virt/kvm/kvm
git bisect good 8bbe0dec38e147a50e9dd5f585295f7e68e0f2d0
# bad: [298fb76a5583900a155d387efaf37a8b39e5dea2] Merge tag 'nfsd-5.4' of git://linux-nfs.org/~bfields/linux
git bisect bad 298fb76a5583900a155d387efaf37a8b39e5dea2
# bad: [bbf2f098838aa86cee240a7a11486e0601d56a3f] nfsd: Reset the boot verifier on all write I/O errors
git bisect bad bbf2f098838aa86cee240a7a11486e0601d56a3f
# skip: [fd4f83fd7dfb1bce2f1af51fcbaf6575f4b9d189] nfsd: convert nfs4_file->fi_fds array to use nfsd_files
git bisect skip fd4f83fd7dfb1bce2f1af51fcbaf6575f4b9d189
# bad: [5e113224c17e2fb156b785ddbbc48a0209fddb0c] nfsd: nfsd_file cache entries should be per net namespace
git bisect bad 5e113224c17e2fb156b785ddbbc48a0209fddb0c
# good: [48cd7b51258c1a158293cefb97dda988080f5e13] nfsd: hook up nfsd_read to the nfsd_file cache
git bisect good 48cd7b51258c1a158293cefb97dda988080f5e13
# bad: [501cb1849f865960501d19d54e6a5af306f9b6fd] nfsd: rip out the raparms cache
git bisect bad 501cb1849f865960501d19d54e6a5af306f9b6fd
# bad: [eb82dd393744107ebc365a53e7813c7c67cb203b] nfsd: convert fi_deleg_file and ls_file fields to nfsd_file
git bisect bad eb82dd393744107ebc365a53e7813c7c67cb203b
# good: [5920afa3c85ff38642f652b6e3880e79392fcc89] nfsd: hook nfsd_commit up to the nfsd_file cache
git bisect good 5920afa3c85ff38642f652b6e3880e79392fcc89
# only skipped commits left to test
# possible first bad commit: [eb82dd393744107ebc365a53e7813c7c67cb203b] nfsd: convert fi_deleg_file and ls_file fields to nfsd_file
# possible first bad commit: [fd4f83fd7dfb1bce2f1af51fcbaf6575f4b9d189] nfsd: convert nfs4_file->fi_fds array to use nfsd_files
# good: [5920afa3c85ff38642f652b6e3880e79392fcc89] nfsd: hook nfsd_commit up to the nfsd_file cache
git bisect good 5920afa3c85ff38642f652b6e3880e79392fcc89
# only skipped commits left to test
# possible first bad commit: [eb82dd393744107ebc365a53e7813c7c67cb203b] nfsd: convert fi_deleg_file and ls_file fields to nfsd_file
# possible first bad commit: [fd4f83fd7dfb1bce2f1af51fcbaf6575f4b9d189] nfsd: convert nfs4_file->fi_fds array to use nfsd_files