linux-kernel - Re: [PATCH] srcu: Reduce blocking agressiveness of expedited grace periods further

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <db30cf38-7ee7-30b0-8dff-b2cc504c3588@quicinc.com>
Date:   Tue, 28 Jun 2022 15:32:46 +0530
From:   Neeraj Upadhyay <quic_neeraju@...cinc.com>
To:     Marc Zyngier <maz@...nel.org>
CC:     <paulmck@...nel.org>, <frederic@...nel.org>,
        <josh@...htriplett.org>, <rostedt@...dmis.org>,
        <mathieu.desnoyers@...icios.com>, <jiangshanlai@...il.com>,
        <joel@...lfernandes.org>, <linux-kernel@...r.kernel.org>,
        <zhangfei.gao@...mail.com>, <boqun.feng@...il.com>,
        <urezki@...il.com>, <shameerali.kolothum.thodi@...wei.com>,
        <pbonzini@...hat.com>, <mtosatti@...hat.com>,
        <eric.auger@...hat.com>, <chenxiang66@...ilicon.com>
Subject: Re: [PATCH] srcu: Reduce blocking agressiveness of expedited grace
 periods further



On 6/28/2022 3:01 PM, Marc Zyngier wrote:
> On Tue, 28 Jun 2022 10:17:24 +0100,
> Neeraj Upadhyay <quic_neeraju@...cinc.com> wrote:
>>
>>
>>
>> On 6/28/2022 2:32 PM, Marc Zyngier wrote:
>>> On Mon, 27 Jun 2022 13:37:06 +0100,
>>> Neeraj Upadhyay <quic_neeraju@...cinc.com> wrote:
>>>>
>>>> Commit 640a7d37c3f4 ("srcu: Block less aggressively for expedited
>>>> grace periods") highlights a problem where aggressively blocking
>>>> SRCU expedited grace periods, as was introduced in commit
>>>> 282d8998e997 ("srcu: Prevent expedited GPs and blocking readers
>>>> from consuming CPU"), introduces ~2 minutes delay to the overall
>>>> ~3.5 minutes boot time, when starting VMs with "-bios QEMU_EFI.fd"
>>>> cmdline on qemu, which results in very high rate of memslots
>>>> add/remove, which causes > ~6000 synchronize_srcu() calls for
>>>> kvm->srcu SRCU instance.
>>>>
>>>> Below table captures the experiments done by Zhangfei Gao, Shameer,
>>>> to measure the boottime impact with various values of non-sleeping
>>>> per phase counts, with HZ_250 and preemption enabled:
>>>>
>>>> +──────────────────────────+────────────────+
>>>> | SRCU_MAX_NODELAY_PHASE   | Boot time (s)  |
>>>> +──────────────────────────+────────────────+
>>>> | 100                      | 30.053         |
>>>> | 150                      | 25.151         |
>>>> | 200                      | 20.704         |
>>>> | 250                      | 15.748         |
>>>> | 500                      | 11.401         |
>>>> | 1000                     | 11.443         |
>>>> | 10000                    | 11.258         |
>>>> | 1000000                  | 11.154         |
>>>> +──────────────────────────+────────────────+
>>>>
>>>> Analysis on the experiment results showed improved boot time
>>>> with non blocking delays close to one jiffy duration. This
>>>> was also seen when number of per-phase iterations were scaled
>>>> to one jiffy.
>>>>
>>>> So, this change scales per-grace-period phase number of non-sleeping
>>>> polls, soiuch that, non-sleeping polls are done for one jiffy. In addition
>>>> to this, srcu_get_delay() call in srcu_gp_end(), which is used to calculate
>>>> the delay used for scheduling callbacks, is replaced with the check for
>>>> expedited grace period. This is done, to schedule cbs for completed expedited
>>>> grace periods immediately, which results in improved boot time seen in
>>>> experiments.
>>>>
>>>> In addition to the changes to default per phase delays, this change
>>>> adds 3 new kernel parameters - srcutree.srcu_max_nodelay,
>>>> srcutree.srcu_max_nodelay_phase, srcutree.srcu_retry_check_delay.
>>>> This allows users to configure the srcu grace period scanning delays,
>>>> depending on their system configuration requirements.
>>>>
>>>> Signed-off-by: Neeraj Upadhyay <quic_neeraju@...cinc.com>
>>>
>>> I've given this a go on one of my test platforms (the one I noticed
>>> the issue on the first place), and found that the initial part of the
>>> EFI boot under KVM (pointlessly wiping the emulated flash) went down
>>> to 1m7s from 3m50s (HZ=250).
>>>
>>> Clearly a massive improvement, but still a far cry from the original
>>> ~40s (yes, this box is utter crap -- which is why I use it).
>>
>> Do you see any improvement by using "srcutree.srcu_max_nodelay=1000"
>> bootarg, on top of this patch?
> 
> Yup, this brings it back to 43s on a quick test run, which is close
> enough to what I had before.
> 

Cool, thanks!

> How does a random user come up with such a value though?
>

It need to be tuned :) The patch actually adds 2 jiffies (vs the one 
jiffy mentioned in the commit log) of non-sleep per phase delay. Each 
phase iteration uses a delay of 10 us. So, for CONFIG_HZ_250 its around 
800 iterations.


Thanks
Neeraj

> Thanks,
> 
> 	M.
>