linux-kernel - Re: Infiniate systemd loop when power off the machine with multiple MD RAIDs

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <c7454021-5559-6556-e9ec-de856e00f1fb@suse.de>
Date:   Wed, 23 Aug 2023 08:04:10 +0200
From:   Hannes Reinecke <hare@...e.de>
To:     Yu Kuai <yukuai1@...weicloud.com>,
        Carlos Carvalho <carlos@...ica.ufpr.br>,
        Song Liu <song@...nel.org>
Cc:     Bagas Sanjaya <bagasdotme@...il.com>,
        Christoph Hellwig <hch@....de>, AceLan Kao <acelan@...il.com>,
        Linux Kernel Mailing List <linux-kernel@...r.kernel.org>,
        Linux Regressions <regressions@...ts.linux.dev>,
        Linux RAID <linux-raid@...r.kernel.org>,
        "yukuai (C)" <yukuai3@...wei.com>,
        "yangerkun@...wei.com" <yangerkun@...wei.com>
Subject: Re: Infiniate systemd loop when power off the machine with multiple
 MD RAIDs

On 8/23/23 03:28, Yu Kuai wrote:
> Hi,
> 
> 在 2023/08/23 3:13, Carlos Carvalho 写道:
>> Song Liu (song@...nel.org) wrote on Tue, Aug 22, 2023 at 03:56:04PM -03:
>>> >From systemd code, i.e. function delete_md(), this error:
>>>
>>> [ 205.957004] systemd-shutdown[1]: Stopping MD /dev/md124p1 (259:6).
>>> [ 205.964177] systemd-shutdown[1]: Could not stop MD /dev/md124p1:
>>> Device or resource busy
>>>
>>> is most likely triggered by ioctl(STOP_ARRAY).
>>>
>>> And based on the code, I think the ioctl fails here:
>>>
>>>          if (cmd == STOP_ARRAY || cmd == STOP_ARRAY_RO) {
>>>                  /* Need to flush page cache, and ensure no-one else 
>>> opens
>>>                   * and writes
>>>                   */
>>>                  mutex_lock(&mddev->open_mutex);
>>>                  if (mddev->pers && atomic_read(&mddev->openers) > 1) {
>>>                          mutex_unlock(&mddev->open_mutex);
>>>                          err = -EBUSY;
>>>                          goto out;        ////////////////////// HERE
> 
> Yes, I suspect here as well, but I do some test with error injection to
> gurantee that ioctl(STOP_ARRAY) always return -EBUSY, but I found that
> system reboot didn't hang, it'll try a few times but eventually reboot
> finished.
> 
>>>                  }
>>>                  if (test_and_set_bit(MD_CLOSING, &mddev->flags)) {
>>>                          mutex_unlock(&mddev->open_mutex);
>>>                          err = -EBUSY;
>>>                          goto out;
>>>                  }
>>>                  did_set_md_closing = true;
>>>                  mutex_unlock(&mddev->open_mutex);
>>>                  sync_blockdev(bdev);
>>>          }
>>
>> Probably. The problem is why doesn't it manage to flush the page 
>> cache? I find
>> strange that the problem appears only when trying to stop the array, I 
>> get it
>> when trying to umount the filesystem, where it also hangs because of 
>> the same
>> reason. The kworker thread runs continuously using 100% cpu of only 1 
>> core.
> 
> The kworker do you mean the daemon thread or the sync thread?  runs
> continuously using 100% cpu doen't sounds correct to me.
> 
That sounds more like kworker waiting on a mutex to be released.
Can you trigger a kdump while in that state?
Or do a sysrq-T to get a stack trace of all running processes?
That should tell you where exactly the 100% cpu time is being
wasted.

Cheers,

Hannes
-- 
Dr. Hannes Reinecke                Kernel Storage Architect
hare@...e.de                              +49 911 74053 688
SUSE Software Solutions GmbH, Maxfeldstr. 5, 90409 Nürnberg
HRB 36809 (AG Nürnberg), Geschäftsführer: Ivo Totev, Andrew
Myers, Andrew McDonald, Martje Boudien Moerman