| lists.openwall.net | lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC | |
|
Open Source and information security mailing list archives
| ||
|
Message-ID: <20230822083923.00007fb6@linux.intel.com>
Date: Tue, 22 Aug 2023 08:39:23 +0200
From: Mariusz Tkaczyk <mariusz.tkaczyk@...ux.intel.com>
To: Song Liu <song@...nel.org>
Cc: Guoqing Jiang <guoqing.jiang@...ux.dev>,
Bagas Sanjaya <bagasdotme@...il.com>,
Christoph Hellwig <hch@....de>, AceLan Kao <acelan@...il.com>,
Linux Kernel Mailing List <linux-kernel@...r.kernel.org>,
Linux Regressions <regressions@...ts.linux.dev>,
Linux RAID <linux-raid@...r.kernel.org>
Subject: Re: Infiniate systemd loop when power off the machine with multiple
MD RAIDs
On Mon, 21 Aug 2023 23:17:54 -0700
Song Liu <song@...nel.org> wrote:
> On Mon, Aug 21, 2023 at 8:51 PM Guoqing Jiang <guoqing.jiang@...ux.dev> wrote:
> >
> >
> >
> > On 8/18/23 16:16, Mariusz Tkaczyk wrote:
> > > On Wed, 16 Aug 2023 16:37:26 +0700
> > > Bagas Sanjaya<bagasdotme@...il.com> wrote:
> > >
> > >> Hi,
> > >>
> > >> I notice a regression report on Bugzilla [1]. Quoting from it:
> > >>
> > >>> It needs to build at least 2 different RAIDs(eg. RAID0 and RAID10, RAID5
> > >>> and RAID10) and then you will see below error repeatly(need to use
> > >>> serial console to see it)
> > >>>
> > >>> [ 205.360738] systemd-shutdown[1]: Stopping MD devices.
> > >>> [ 205.366384] systemd-shutdown[1]: sd-device-enumerator: Scan all dirs
> > >>> [ 205.373327] systemd-shutdown[1]: sd-device-enumerator: Scanning
> > >>> /sys/bus [ 205.380427] systemd-shutdown[1]: sd-device-enumerator:
> > >>> Scanning /sys/class [ 205.388257] systemd-shutdown[1]: Stopping MD
> > >>> /dev/md127 (9:127). [ 205.394880] systemd-shutdown[1]: Failed to sync
> > >>> MD block device /dev/md127, ignoring: Input/output error [ 205.404975]
> > >>> md: md127 stopped. [ 205.470491] systemd-shutdown[1]: Stopping MD
> > >>> /dev/md126 (9:126). [ 205.770179] md: md126: resync interrupted.
> > >>> [ 205.776258] md126: detected capacity change from 1900396544 to 0
> > >>> [ 205.783349] md: md126 stopped.
> > >>> [ 205.862258] systemd-shutdown[1]: Stopping MD /dev/md125 (9:125).
> > >>> [ 205.862435] md: md126 stopped.
> > >>> [ 205.868376] systemd-shutdown[1]: Failed to sync MD block device
> > >>> /dev/md125, ignoring: Input/output error [ 205.872845] block device
> > >>> autoloading is deprecated and will be removed. [ 205.880955] md: md125
> > >>> stopped. [ 205.934349] systemd-shutdown[1]: Stopping MD /dev/md124p2
> > >>> (259:7). [ 205.947707] systemd-shutdown[1]: Could not stop MD
> > >>> /dev/md124p2: Device or resource busy [ 205.957004]
> > >>> systemd-shutdown[1]: Stopping MD /dev/md124p1 (259:6). [ 205.964177]
> > >>> systemd-shutdown[1]: Could not stop MD /dev/md124p1: Device or resource
> > >>> busy [ 205.973155] systemd-shutdown[1]: Stopping MD /dev/md124 (9:124).
> > >>> [ 205.979789] systemd-shutdown[1]: Could not stop MD /dev/md124: Device
> > >>> or resource busy [ 205.988475] systemd-shutdown[1]: Not all MD devices
> > >>> stopped, 4 left.
> > >> See Bugzilla for the full thread and attached full journalctl log.
> > >>
> > >> Anyway, I'm adding this regression to be tracked by regzbot:
> > >>
> > >> #regzbot introduced: 12a6caf273240a
> > >> https://bugzilla.kernel.org/show_bug.cgi?id=217798 #regzbot title:
> > >> systemd shutdown hang on machine with different RAID levels
> > >>
> > >> Thanks.
> > >>
> > >> [1]:https://bugzilla.kernel.org/show_bug.cgi?id=217798
> > >>
> > > Hello,
> > > The issue is reproducible with IMSM metadata too, around 20% of reboot
> > > hangs. I will try to raise the priority in the bug because it is valid
> > > high- the base functionality of the system is affected.
> >
> > Since it it reproducible from your side, is it possible to turn the
> > reproduce steps into a test case
> > given the importance?
I didn't try to reproduce it locally yet because customer was able to
bisect the regression and it pointed them to the same patch so I connected it
and asked author to take a look first. At a first glance, I wanted to get
community voice to see if it is not something obvious.
So far I know, customer is creating 3 IMSM raid arrays, one is the system
volume and do a reboot and it sporadically fails (around 20%). That is all.
> >
> > I guess If all arrays are set with MD_DELETED flag, then reboot might
> > hang, not sure whether
> > below (maybe need to flush wq as well before list_del) helps or not,
> > just FYI.
> >
> > @@ -9566,8 +9566,10 @@ static int md_notify_reboot(struct notifier_block
> > *this,
> >
> > spin_lock(&all_mddevs_lock);
> > list_for_each_entry_safe(mddev, n, &all_mddevs, all_mddevs) {
> > - if (!mddev_get(mddev))
> > + if (!mddev_get(mddev)) {
> > + list_del(&mddev->all_mddevs);
> > continue;
> > + }
>
> I am still not able to reproduce this, probably due to differences in the
> timing. Maybe we only need something like:
>
> diff --git i/drivers/md/md.c w/drivers/md/md.c
> index 5c3c19b8d509..ebb529b0faf8 100644
> --- i/drivers/md/md.c
> +++ w/drivers/md/md.c
> @@ -9619,8 +9619,10 @@ static int md_notify_reboot(struct notifier_block
> *this,
>
> spin_lock(&all_mddevs_lock);
> list_for_each_entry_safe(mddev, n, &all_mddevs, all_mddevs) {
> - if (!mddev_get(mddev))
> + if (!mddev_get(mddev)) {
> + need_delay = 1;
> continue;
> + }
> spin_unlock(&all_mddevs_lock);
> if (mddev_trylock(mddev)) {
> if (mddev->pers)
>
>
> Thanks,
> Song
I will try to reproduce issue at Intel lab to check this.
Thanks,
Mariusz
Powered by blists - more mailing lists