Message-ID: <CAOPYjvYhEzeF3vdd9GXCX+k_-OmsE1yP7VNozcMt4vOyFLDAfw@mail.gmail.com>
Date: Fri, 22 Dec 2023 10:34:58 +0800
From: 20 39 <2045gemini@...il.com>
To: Yu Kuai <yukuai1@...weicloud.com>
Cc: song@...nel.org, linux-raid@...r.kernel.org, linux-kernel@...r.kernel.org, 
	baijiaju1990@...look.com, BassCheck <bass@...a.edu.cn>, 
	"yukuai (C)" <yukuai3@...wei.com>
Subject: Re: [PATCH] md/raid5: fix atomicity violation in raid5_cache_count

Hi Kuai,

Thank you for your patience. This email is essentially the same as my
previous one, only now adjusted to plain text format. I apologize for
any inconvenience caused earlier.

Thanks for your email and the insightful points you've raised. Let me
clarify a few aspects regarding the raid5_cache_count() and
raid5_set_cache_size() functions.

1. Callback Function in setup_conf(): You mentioned that
raid5_cache_count() is called from setup_conf() where reconfig_mutex
is held. While this is true, it's important to note that
raid5_cache_count() is actually initialized as a callback function in
setup_conf(), as described in /include/linux/shrinker.h. This means it
could be invoked later in a context where the reconfig_mutex isn't
necessarily held. The documentation in shrinker.h indicates potential
invocation scenarios beyond the initial setup context.

2. Comment on Unlikely Scenario: Regarding the comment in the code /*
unlikely, but not impossible */, it seems to acknowledge the
possibility of a race condition where conf->max_nr_stripes is less
than conf->min_nr_stripes. This comment implies that previous
developers might have noticed potential issues with race conditions,
even if they are considered unlikely.

3. Use of Local Variables Instead of Locks: In addressing this issue,
my approach doesn't involve introducing additional locks, which could
potentially lead to deadlocks or other concurrency problems. Instead,
I've opted to store the values in local variables to ensure that the
check remains effective and consistent throughout its execution. This
modification aims to prevent reading intermediate values that might
lead to incorrect calculations, without introducing new locking
complexities.

I'm looking forward to any further insights or suggestions you might
have on this matter.

Best regards,
Han


20 39 <2045gemini@...il.com> 于2023年12月22日周五 10:21写道：
>
> Hi Kuai,
>
> Thanks for your email and the insightful points you've raised. Let me clarify a few aspects regarding the raid5_cache_count() and raid5_set_cache_size() functions.
>
> 1. Callback Function in setup_conf(): You mentioned that raid5_cache_count() is called from setup_conf() where reconfig_mutex is held. While this is true, it's important to note that raid5_cache_count() is actually initialized as a callback function in setup_conf(), as described in /include/linux/shrinker.h. This means it could be invoked later in a context where the reconfig_mutex isn't necessarily held. The documentation in shrinker.h indicates potential invocation scenarios beyond the initial setup context.
>
> 2. Comment on Unlikely Scenario: Regarding the comment in the code /* unlikely, but not impossible */, it seems to acknowledge the possibility of a race condition where conf->max_nr_stripes is less than conf->min_nr_stripes. This comment implies that previous developers might have noticed potential issues with race conditions, even if they are considered unlikely.
>
> 3. Use of Local Variables Instead of Locks: In addressing this issue, my approach doesn't involve introducing additional locks, which could potentially lead to deadlocks or other concurrency problems. Instead, I've opted to store the values in local variables to ensure that the check remains effective and consistent throughout its execution. This modification aims to prevent reading intermediate values that might lead to incorrect calculations, without introducing new locking complexities.
>
> I'm looking forward to any further insights or suggestions you might have on this matter.
>
> Best regards,
> Han
>
> Yu Kuai <yukuai1@...weicloud.com> 于2023年12月22日周五 09:37写道：
>>
>> Hi,
>>
>> 在 2023/12/21 18:43, Gui-Dong Han 写道:
>> > In raid5_cache_count():
>> >       if (conf->max_nr_stripes < conf->min_nr_stripes)
>> >               return 0;
>> >       return conf->max_nr_stripes - conf->min_nr_stripes;
>> > The current check is ineffective, as the values could change immediately
>> > after being checked.
>> >
>> > In raid5_set_cache_size():
>> >       ...
>> >       conf->min_nr_stripes = size;
>> >       ...
>> >       while (size > conf->max_nr_stripes)
>> >               conf->min_nr_stripes = conf->max_nr_stripes;
>> >       ...
>> >
>>
>> raid5_cache_count() is called from setup_conf() where reconfig_mtuex is
>> held.
>>
>> raid5_set_cache_size() is called from:
>> 1) raid5_store_stripe_cache_size(), reconfig_mutex is held
>> 2) r5l_start() from raid5_add_disk(), reconfig_mutex is held
>> 3) raid_ctr(), reconfig_mutex is held
>>
>> So, how can they concurrent in the first place?
>>
>> Thanks,
>> Kuai
>>
>> > Due to intermediate value updates in raid5_set_cache_size(), concurrent
>> > execution of raid5_cache_count() and raid5_set_cache_size() may lead to
>> > inconsistent reads of conf->max_nr_stripes and conf->min_nr_stripes.
>> > The current checks are ineffective as values could change immediately
>> > after being checked, raising the risk of conf->min_nr_stripes exceeding
>> > conf->max_nr_stripes and potentially causing an integer overflow.
>> >
>> > This possible bug is found by an experimental static analysis tool
>> > developed by our team. This tool analyzes the locking APIs to extract
>> > function pairs that can be concurrently executed, and then analyzes the
>> > instructions in the paired functions to identify possible concurrency bugs
>> > including data races and atomicity violations. The above possible bug is
>> > reported when our tool analyzes the source code of Linux 6.2.
>> >
>> > To resolve this issue, it is suggested to introduce local variables
>> > 'min_stripes' and 'max_stripes' in raid5_cache_count() to ensure the
>> > values remain stable throughout the check. Adding locks in
>> > raid5_cache_count() fails to resolve atomicity violations, as
>> > raid5_set_cache_size() may hold intermediate values of
>> > conf->min_nr_stripes while unlocked. With this patch applied, our tool no
>> > longer reports the bug, with the kernel configuration allyesconfig for
>> > x86_64. Due to the lack of associated hardware, we cannot test the patch
>> > in runtime testing, and just verify it according to the code logic.
>> >
>> > Fixes: edbe83ab4c27e ("md/raid5: allow the stripe_cache to grow and ...")
>> > Reported-by: BassCheck <bass@...a.edu.cn>
>> > Signed-off-by: Gui-Dong Han <2045gemini@...il.com>
>> > ---
>> >   drivers/md/raid5.c | 7 ++++---
>> >   1 file changed, 4 insertions(+), 3 deletions(-)
>> >
>> > diff --git a/drivers/md/raid5.c b/drivers/md/raid5.c
>> > index 8497880135ee..62ebf33402cc 100644
>> > --- a/drivers/md/raid5.c
>> > +++ b/drivers/md/raid5.c
>> > @@ -7390,11 +7390,12 @@ static unsigned long raid5_cache_count(struct shrinker *shrink,
>> >                                      struct shrink_control *sc)
>> >   {
>> >       struct r5conf *conf = shrink->private_data;
>> > -
>> > -     if (conf->max_nr_stripes < conf->min_nr_stripes)
>> > +     int max_stripes = conf->max_nr_stripes;
>> > +     int min_stripes = conf->min_nr_stripes;
>> > +     if (max_stripes < min_stripes)
>> >               /* unlikely, but not impossible */
>> >               return 0;
>> > -     return conf->max_nr_stripes - conf->min_nr_stripes;
>> > +     return max_stripes - min_stripes;
>> >   }
>> >
>> >   static struct r5conf *setup_conf(struct mddev *mddev)
>> >
>>

Powered by blists - more mailing lists