linux-kernel - Re: Infiniate systemd loop when power off the machine with multiple MD RAIDs

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <CAMz9Wg9y52iuxJRSQFC2N5Katt72v-o=JvEjegJt-MwORmw9tQ@mail.gmail.com>
Date:   Sat, 26 Aug 2023 12:31:15 +0800
From:   AceLan Kao <acelan@...il.com>
To:     Song Liu <song@...nel.org>
Cc:     Guoqing Jiang <guoqing.jiang@...ux.dev>,
        Mariusz Tkaczyk <mariusz.tkaczyk@...ux.intel.com>,
        Bagas Sanjaya <bagasdotme@...il.com>,
        Christoph Hellwig <hch@....de>,
        Linux Kernel Mailing List <linux-kernel@...r.kernel.org>,
        Linux Regressions <regressions@...ts.linux.dev>,
        Linux RAID <linux-raid@...r.kernel.org>
Subject: Re: Infiniate systemd loop when power off the machine with multiple
 MD RAIDs

Hi Song,

Song Liu <song@...nel.org> 於 2023年8月23日 週三 下午9:25寫道：
>
> Hi AceLan,
>
> On Wed, Aug 23, 2023 at 1:03 AM AceLan Kao <acelan@...il.com> wrote:
> >
> [...]
> > >
> > > My suggestion is delete the list node under this scenario, did you try
> > > above?
> > Still no luck, the patch doesn't work, the sympton is the same.
> [...]
> > >
> > > Is the issue still reproducible with remove below from cmd?
> > >
> > > echo "repair" | sudo tee /sys/class/block/md12?/md/sync_action
> > >
> > > Just want to know if resync thread is related with the issue or not.
> > Probably not, we can reproduce the issue without resync, just feel
> > it's easier to reproduce the issue, so I put the command in my script.
> >
>
> Could you please run the follow two experiments?
>
> 1. Confirm 12a6caf273240a triggers this. Specifically:
>    git checkout 12a6caf273240a => repros
>    git checkout 12a6caf273240a~1 => cannot repro
Yes, I'm pretty sure about this, that's my bisect result and I just
confirmed it again.
I also tried reverting 12a6caf273240a and the issue is gone.


>
> 2. Try with the following change (add debug messages), which hopefully
>    shows which command is holding a reference on mddev->openers.
>
> Thanks,
> Song
>
> diff --git i/drivers/md/md.c w/drivers/md/md.c
> index 78be7811a89f..3e9b718b32c1 100644
> --- i/drivers/md/md.c
> +++ w/drivers/md/md.c
> @@ -7574,11 +7574,15 @@ static int md_ioctl(struct block_device *bdev,
> blk_mode_t mode,
>                 if (mddev->pers && atomic_read(&mddev->openers) > 1) {
>                         mutex_unlock(&mddev->open_mutex);
>                         err = -EBUSY;
> +                       pr_warn("%s return -EBUSY for %s with
> mddev->openers = %d\n",
> +                               __func__, mdname(mddev),
> atomic_read(&mddev->openers));
>                         goto out;
>                 }
>                 if (test_and_set_bit(MD_CLOSING, &mddev->flags)) {
>                         mutex_unlock(&mddev->open_mutex);
>                         err = -EBUSY;
> +                       pr_warn("%s return -EBUSY for %s with
> MD_CLOSING bit set\n",
> +                               __func__, mdname(mddev));
>                         goto out;
>                 }
>                 did_set_md_closing = true;
> @@ -7789,6 +7793,8 @@ static int md_open(struct gendisk *disk, blk_mode_t mode)
>                 goto out_unlock;
>
>         atomic_inc(&mddev->openers);
> +       pr_info("%s:%s openers++ = %d by %s\n", __func__, mdname(mddev),
> +               atomic_read(&mddev->openers), current->comm);
>         mutex_unlock(&mddev->open_mutex);
>
>         disk_check_media_change(disk);
> @@ -7807,6 +7813,8 @@ static void md_release(struct gendisk *disk)
>
>         BUG_ON(!mddev);
>         atomic_dec(&mddev->openers);
> +       pr_info("%s:%s openers-- = %d by %s\n", __func__, mdname(mddev),
> +               atomic_read(&mddev->openers), current->comm);
>         mddev_put(mddev);
>  }
It's pretty strange that I can't reproduce the issue after applied the patch.

I tried to figure out which part affect the issue and found when I
comment out the pr_info() In md_release(), the issue could be
reproduced.

--
Chia-Lin Kao(AceLan)
http://blog.acelan.idv.tw/
E-Mail: acelan.kaoATcanonical.com (s/AT/@/)

View attachment "minicom.1.log" of type "text/x-log" (797872 bytes)