linux-kernel - Re: [PATCH v5 00/14] dm-raid/md/raid: fix v6.7 regressions

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <ZcVhA_IqXH2Pg79t@bmarzins-01.fast.eng.rdu2.dc.redhat.com>
Date: Thu, 8 Feb 2024 18:17:23 -0500
From: Benjamin Marzinski <bmarzins@...hat.com>
To: Song Liu <song@...nel.org>
Cc: Yu Kuai <yukuai1@...weicloud.com>, mpatocka@...hat.com, heinzm@...hat.com,
        xni@...hat.com, blazej.kucman@...ux.intel.com, agk@...hat.com,
        snitzer@...nel.org, dm-devel@...ts.linux.dev, jbrassow@....redhat.com,
        neilb@...e.de, shli@...com, akpm@...l.org,
        linux-kernel@...r.kernel.org, linux-raid@...r.kernel.org,
        yi.zhang@...wei.com, yangerkun@...wei.com,
        "yukuai (C)" <yukuai3@...wei.com>
Subject: Re: [PATCH v5 00/14] dm-raid/md/raid: fix v6.7 regressions

On Thu, Feb 08, 2024 at 12:04:45AM -0800, Song Liu wrote:
> Hi Benjamin,
> 
> On Mon, Feb 5, 2024 at 7:58 PM Benjamin Marzinski <bmarzins@...hat.com> wrote:
> >
> > On Tue, Feb 06, 2024 at 09:36:18AM +0800, Yu Kuai wrote:
> > > Hi!
> > >
> > > 在 2024/02/06 3:35, Benjamin Marzinski 写道:
> > > > Could you run the test with something like
> > > >
> > > > # make check_local T=lvconvert-repair-raid.sh VERBOSE=1 > out 2>&1
> > > >
> > > > and post the output.
> > >
> > > Attached is the output from my VM.
> >
> > Instead of running the tests from the lvm2 git repo, if you run
> >
> > # make -C test install
> >
> > to install the tests, and then create a results directory and run the
> > test from there, do you still see the error in the 6.6 kernel?
> >
> > # make ~/results
> > # cd ~/results
> > # lvm2-testsuite --only lvconvert-repair-raid.sh
> >
> > Running the tests this way will test the installed lvm2 binaries on your
> > system, instead of the ones in the lvm2 git repo. They may be compiled
> > differently.
> 
> I am not able to get reliable results from shell/lvconvert-repair-raid.sh
> either. For 6.6.0 kernel, the test fails. On 6.8-rc1 kernel, the test fails
> sometimes.
> 
> Could you please share more information about your test setup?
> Specifically:
> 1. Which tree/branch/tag are you testing?
> 2. What's the .config used in the tests?
> 3. How do you run the test suite? One test at a time, or all of them
> together?
> 4. How do you handle "test passes sometimes" cases?

So, I have been able to recreate the case where lvconvert-repair-raid.sh
keeps failing. It happens when I tried running the reproducer on a virtual
machine made using a cloud image, instead of one that I manually
installed. I'm not sure why there is a difference. But I can show you
how I can reliably recreate the errors I'm seeing.


Create a new Fedora 39 virtual machine with the following commands (I'm
not sure if it is possible to reproduce this on a machine using less
memory and cpus, but I can try that if you need me to. You probably also
want to pick a faster Fedora Mirror for the image location):
# virt-install --name repair-test --memory 8192 --vcpus 8 --disk size=40 --graphics none --extra-args "console=ttyS0" --osinfo detect=on,name=fedora-unknown --location https://download.fedoraproject.org/pub/fedora/linux/releases/39/Server/x86_64/os/

Install to the whole virtual drive, using the default LVM partitioning.
Then ssh into the VM and run the following commands to setup the
lvm2-testsuite and 6.6.0 kernel:

# dnf upgrade grub2
# dnf install -y git gcc bc flex make bison openssl openssl-devel dwarves zstd elfutils-libelf-devel libaio-devel lvm2 g++
# git clone git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
# git clone git://sourceware.org/git/lvm2.git
# cd ~/lvm2
# ./configure
# make
# cd ~/linux
# git checkout -b ver6.6 v6.6
# cp /boot/config-`uname -r` .config
# make olddefconfig
# modprobe -a dm_raid dm_delay ext4 raid1 raid10 brd
# yes "" | make localmodconfig
# make -j8
# make modules_install
# make install
# reboot

ssh back into the VM, and run the following commands to run
lvm2-testsuite:

# mount -o remount,dev /tmp
# cd ~/lvm2
# make check T=lvconvert-repair-raid.sh

This should always pass. I ran it 100 times without failure.

To test the patched kernel, run:

# cd ~/linux
# git checkout -b dmraid-fix-v5 v6.8-rc3
# git am ~/dmraid-fix-v5.mbox  ***Apply the v5 patches***
# make olddefconfig
# make -j8
# make modules_install
# make install
# reboot

Rerun the lvm2-testsuite with the same commands as before:

# mount -o remount,dev /tmp
# cd ~/lvm2
# make check T=lvconvert-repair-raid.sh

This fails about 20% of the time, usually at either line 146 or 164. You
can check by running the following command when the test fails.

# grep "STACKTRACE()" ~/lvm2/test/results/ndev-vanilla\:shell_lvconvert-repair-raid.sh.txt
[ 0:13.152] ## 1 STACKTRACE() called from /root/lvm2/test/shell/lvconvert-repair-raid.sh:146

Let me know if you have any questions, or if this doesn't work for you.

-Ben

> Thanks,
> Song