Date:   Wed, 13 Oct 2021 08:39:59 +0200
From:   Mike Galbraith <efault@....de>
To:     netdev <netdev@...r.kernel.org>
Cc:     Jeff Layton <jlayton@...nel.org>,
        Trond Myklebust <trond.myklebust@...marydata.com>
Subject: Re: [bisected] Re: odd NFS v4.2 regression 4.4..5.15

[replaces bouncy address]

On Wed, 2021-10-13 at 08:35 +0200, Mike Galbraith wrote:
> On Tue, 2021-10-12 at 15:54 +0200, Mike Galbraith wrote:
> > Greetings network wizards,
> >
> > I seem to have stumbled across a kind of odd looking NFS regression
> > while looking at something unrelated.
> >
> > I have a few VMs that are mirrors of my desktop box, including some
> > data, but they usually over-mount directories with tons of data via
> > NFS, thus hang off the host like a mini-me Siamese twin for testing.
> >
> > In the data below, I run a script in git-daemon's home to extract RT
> > patch sets with git-format-patch, on the host for reference, on the
> > host in an NFS mount of itself, and the mini-me VM, where I started.
> >
> > Note: bonnie throughput looked fine for v4.2.
> >
> > host
> > time su - git -c ./format-rt-patches.sh
> > real    1m5.923s
> > user    0m54.692s
> > sys     0m11.550s
> >
> > NFS mount my own box, and do the same from the same spot
> > time su git -c 'cd /homer/home/git;./format-rt-patches.sh'
> >
> > 4.4.231     v4.2        v3
> > 1   real    2m27.046s   2m2.059s
> >     user    0m59.190s   0m58.701s
> >     sys     0m41.448s   0m32.541s
> >
> > 5.15-rc5    v4.2        v3
> >    real     3m14.954s   2m8.366s
> >    user     0m59.901s   0m58.317s
> >    sys      0m52.708s   0m32.313s
> >         vs 1   0.754       0.951
> >
> > repeats     v4.2        v3
> >    real     3m16.313s   2m7.940s
> >    real     3m10.905s   2m8.029s
>
> Found the v4.2 regression (for this load anyway) to have landed in 5.4.
> It then bisected cleanly to either the build bug or its neighbor.
>
> There are only 'skip'ped commits left to test.
> The first bad commit could be any of:
> fd4f83fd7dfb1bce2f1af51fcbaf6575f4b9d189
> eb82dd393744107ebc365a53e7813c7c67cb203b
> We cannot bisect more!
>
> git bisect start
> # good: [4d856f72c10ecb060868ed10ff1b1453943fc6c8] Linux 5.3
> git bisect good 4d856f72c10ecb060868ed10ff1b1453943fc6c8
> # bad: [219d54332a09e8d8741c1e1982f5eae56099de85] Linux 5.4
> git bisect bad 219d54332a09e8d8741c1e1982f5eae56099de85
> # good: [a9f8b38a071b468276a243ea3ea5a0636e848cf2] Merge tag 'for-linus-5.4-1' of git://github.com/cminyard/linux-ipmi
> git bisect good a9f8b38a071b468276a243ea3ea5a0636e848cf2
> # good: [5825a95fe92566ada2292a65de030850b5cff1da] Merge tag 'selinux-pr-20190917' of git://git.kernel.org/pub/scm/linux/kernel/git/pcmoore/selinux
> git bisect good 5825a95fe92566ada2292a65de030850b5cff1da
> # bad: [48acba989ed5d8707500193048d6c4c5945d5f43] Merge tag 'riscv/for-v5.4-rc3' of git://git.kernel.org/pub/scm/linux/kernel/git/riscv/linux
> git bisect bad 48acba989ed5d8707500193048d6c4c5945d5f43
> # good: [7bccb9f10c8f36ee791769b531ed4d28f6379aae] Merge tag 'linux-watchdog-5.4-rc1' of git://www.linux-watchdog.org/linux-watchdog
> git bisect good 7bccb9f10c8f36ee791769b531ed4d28f6379aae
> # bad: [ef129d34149ea23d0d442844fc25ae26a85589fc] selftests/net: add nettest to .gitignore
> git bisect bad ef129d34149ea23d0d442844fc25ae26a85589fc
> # bad: [9c5efe9ae7df78600c0ee7bcce27516eb687fa6e] Merge branch 'sched-urgent-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip
> git bisect bad 9c5efe9ae7df78600c0ee7bcce27516eb687fa6e
> # good: [8bbe0dec38e147a50e9dd5f585295f7e68e0f2d0] Merge tag 'for-linus' of git://git.kernel.org/pub/scm/virt/kvm/kvm
> git bisect good 8bbe0dec38e147a50e9dd5f585295f7e68e0f2d0
> # bad: [298fb76a5583900a155d387efaf37a8b39e5dea2] Merge tag 'nfsd-5.4' of git://linux-nfs.org/~bfields/linux
> git bisect bad 298fb76a5583900a155d387efaf37a8b39e5dea2
> # bad: [bbf2f098838aa86cee240a7a11486e0601d56a3f] nfsd: Reset the boot verifier on all write I/O errors
> git bisect bad bbf2f098838aa86cee240a7a11486e0601d56a3f
> # skip: [fd4f83fd7dfb1bce2f1af51fcbaf6575f4b9d189] nfsd: convert nfs4_file->fi_fds array to use nfsd_files
> git bisect skip fd4f83fd7dfb1bce2f1af51fcbaf6575f4b9d189
> # bad: [5e113224c17e2fb156b785ddbbc48a0209fddb0c] nfsd: nfsd_file cache entries should be per net namespace
> git bisect bad 5e113224c17e2fb156b785ddbbc48a0209fddb0c
> # good: [48cd7b51258c1a158293cefb97dda988080f5e13] nfsd: hook up nfsd_read to the nfsd_file cache
> git bisect good 48cd7b51258c1a158293cefb97dda988080f5e13
> # bad: [501cb1849f865960501d19d54e6a5af306f9b6fd] nfsd: rip out the raparms cache
> git bisect bad 501cb1849f865960501d19d54e6a5af306f9b6fd
> # bad: [eb82dd393744107ebc365a53e7813c7c67cb203b] nfsd: convert fi_deleg_file and ls_file fields to nfsd_file
> git bisect bad eb82dd393744107ebc365a53e7813c7c67cb203b
> # good: [5920afa3c85ff38642f652b6e3880e79392fcc89] nfsd: hook nfsd_commit up to the nfsd_file cache
> git bisect good 5920afa3c85ff38642f652b6e3880e79392fcc89
> # only skipped commits left to test
> # possible first bad commit: [eb82dd393744107ebc365a53e7813c7c67cb203b] nfsd: convert fi_deleg_file and ls_file fields to nfsd_file
> # possible first bad commit: [fd4f83fd7dfb1bce2f1af51fcbaf6575f4b9d189] nfsd: convert nfs4_file->fi_fds array to use nfsd_files
> # good: [5920afa3c85ff38642f652b6e3880e79392fcc89] nfsd: hook nfsd_commit up to the nfsd_file cache
> git bisect good 5920afa3c85ff38642f652b6e3880e79392fcc89
> # only skipped commits left to test
> # possible first bad commit: [eb82dd393744107ebc365a53e7813c7c67cb203b] nfsd: convert fi_deleg_file and ls_file fields to nfsd_file
> # possible first bad commit: [fd4f83fd7dfb1bce2f1af51fcbaf6575f4b9d189] nfsd: convert nfs4_file->fi_fds array to use nfsd_files
>
>

Powered by blists - more mailing lists