linux-kernel - Re: [PATCH -next 1/6] Revert "md: unlock mddev before reap sync_thread in action

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <CAPhsuW7c2b4yYbwNcqKW+TBL=QYEzchnVQ4pDLBT-xoBoTvQmg@mail.gmail.com>
Date:   Tue, 28 Mar 2023 16:58:59 -0700
From:   Song Liu <song@...nel.org>
To:     Yu Kuai <yukuai1@...weicloud.com>
Cc:     Guoqing Jiang <guoqing.jiang@...ux.dev>, logang@...tatee.com,
        pmenzel@...gen.mpg.de, agk@...hat.com, snitzer@...nel.org,
        linux-kernel@...r.kernel.org, linux-raid@...r.kernel.org,
        yi.zhang@...wei.com, yangerkun@...wei.com,
        Marc Smith <msmith626@...il.com>,
        "yukuai (C)" <yukuai3@...wei.com>
Subject: Re: [PATCH -next 1/6] Revert "md: unlock mddev before reap
 sync_thread in action_store"

On Wed, Mar 22, 2023 at 11:32 PM Yu Kuai <yukuai1@...weicloud.com> wrote:
>
> Hi,
>
> 在 2023/03/23 11:50, Guoqing Jiang 写道:
>
> > Combined your debug patch with above steps. Seems you are
> >
> > 1. add delay to action_store, so it can't get lock in time.
> > 2. echo "want_replacement"**triggers md_check_recovery which can grab lock
> >      to start sync thread.
> > 3. action_store finally hold lock to clear RECOVERY_RUNNING in reap sync
> > thread.
> > 4. Then the new added BUG_ON is invoked since RECOVERY_RUNNING is cleared
> >      in step 3.
>
> Yes, this is exactly what I did.
>
> > sync_thread can be interrupted once MD_RECOVERY_INTR is set which means
> > the RUNNING
> > can be cleared, so I am not sure the added BUG_ON is reasonable. And
> > change BUG_ON
>
> I think BUG_ON() is reasonable because only md_reap_sync_thread can
> clear it, md_do_sync will exit quictly if MD_RECOVERY_INTR is set, but
> md_do_sync should not see that MD_RECOVERY_RUNNING is cleared, otherwise
> there is no gurantee that only one sync_thread can be in progress.
>
> > like this makes more sense to me.
> >
> > +BUG_ON(!test_bit(MD_RECOVERY_RUNNING, &mddev->recovery) &&
> > +!test_bit(MD_RECOVERY_INTR, &mddev->recovery));
>
> I think this can be reporduced likewise, md_check_recovery clear
> MD_RECOVERY_INTR, and new sync_thread triggered by echo
> "want_replacement" won't set this bit.
>
> >
> > I think there might be racy window like you described but it should be
> > really small, I prefer
> > to just add a few lines like this instead of revert and introduce new
> > lock to resolve the same
> > issue (if it is).
>
> The new lock that I add in this patchset is just try to synchronize idle
> and forzen from action_store（patch 3), I can drop it if you think this
> is not necessary.
>
> The main changes is patch 4, new lines is not much and I really don't
> like to add new flags unless we have to, current code is already hard
> to understand...
>
> By the way, I'm concerned that drop the mutex to unregister sync_thread
> might not be safe, since the mutex protects lots of stuff, and there
> might exist other implicit dependencies.
>
> >
> > TBH, I am reluctant to see the changes in the series, it can only be
> > considered
> > acceptable with conditions:
> >
> > 1. the previous raid456 bug can be fixed in this way too, hopefully Marc
> > or others
> >      can verify it.
> > 2. pass all the tests in mdadm

AFAICT, this set looks like a better solution for this problem. But I agree
that we need to make sure it fixes the original bug. mdadm tests are not
in a very good shape at the moment. I will spend more time to look into
these tests.

Thanks,
Song