linux-kernel - [PATCH] rcu: Reduce the frequency of triggering irq-work for strict grace period

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [thread-next>] [day] [month] [year] [list]

Message-Id: <20220808022626.12825-1-qiang1.zhang@intel.com>
Date:   Mon,  8 Aug 2022 10:26:26 +0800
From:   Zqiang <qiang1.zhang@...el.com>
To:     paulmck@...nel.org, frederic@...nel.org, quic_neeraju@...cinc.com
Cc:     rcu@...r.kernel.org, linux-kernel@...r.kernel.org
Subject: [PATCH] rcu: Reduce the frequency of triggering irq-work for strict grace period

For kernel built with PREEMPT_RCU=y and RCU_STRICT_GRACE_PERIOD=y,
currently, the irq-work will be triggered by high frequency to make
the scheduler re-evaluate and call hooks to check whether there are
qs need to report when exit RCU read critical section in irq-disbaled
context. however if there is no gp in progress, the irq-work trigger
is meaningless.

This commit reduced irq-work trigger frequency by check whether the
current CPU is not experiencing qs or there is task that blocks
the current grace period for RCU_STRICT_GRACE_PERIOD=y kernel.

The test results are as follows:

echo rcu_preempt_deferred_qs_handler > /sys/kernel/debug/tracing/set_ftrace_filter
echo 1 > /sys/kernel/debug/tracing/function_profile_enabled
insmod rcutorture.ko
sleep 20
rmmod rcutorture.ko
echo 0 > /sys/kernel/debug/tracing/function_profile_enabled
echo > /sys/kernel/debug/tracing/set_ftrace_filter

head /sys/kernel/debug/tracing/trace_stat/function*

original:

==> /sys/kernel/debug/tracing/trace_stat/function0 <==
  Function                               Hit    Time            Avg             s^2
  --------                               ---    ----            ---             ---
  rcu_preempt_deferred_qs_handle      838746    182650.3 us     0.217 us        0.004 us

==> /sys/kernel/debug/tracing/trace_stat/function1 <==
  Function                               Hit    Time            Avg             s^2
  --------                               ---    ----            ---             ---
  rcu_preempt_deferred_qs_handle      841768    191138.6 us     0.227 us        0.024 us

==> /sys/kernel/debug/tracing/trace_stat/function2 <==
  Function                               Hit    Time            Avg             s^2
  --------                               ---    ----            ---             ---
  rcu_preempt_deferred_qs_handle      828243    180455.6 us     0.217 us        0.004 us

==> /sys/kernel/debug/tracing/trace_stat/function3 <==
  Function                               Hit    Time            Avg             s^2
  --------                               ---    ----            ---             ---
  rcu_preempt_deferred_qs_handle      810258    189158.1 us     0.233 us        0.023 us

apply patch:

==> /sys/kernel/debug/tracing/trace_stat/function0 <==
  Function                               Hit    Time            Avg             s^2
  --------                               ---    ----            ---             ---
  rcu_preempt_deferred_qs_handle      302373    67434.95 us     0.223 us        0.001 us

==> /sys/kernel/debug/tracing/trace_stat/function1 <==
  Function                               Hit    Time            Avg             s^2
  --------                               ---    ----            ---             ---
  rcu_preempt_deferred_qs_handle      307174    68398.34 us     0.222 us        0.002 us

==> /sys/kernel/debug/tracing/trace_stat/function2 <==
  Function                               Hit    Time            Avg             s^2
  --------                               ---    ----            ---             ---
  rcu_preempt_deferred_qs_handle      250910    56157.42 us     0.223 us        0.002 us

==> /sys/kernel/debug/tracing/trace_stat/function3 <==
  Function                               Hit    Time            Avg             s^2
  --------                               ---    ----            ---             ---
  rcu_preempt_deferred_qs_handle      279902    62644.64 us     0.223 us        0.003 us

Signed-off-by: Zqiang <qiang1.zhang@...el.com>
---
 kernel/rcu/tree_plugin.h | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/kernel/rcu/tree_plugin.h b/kernel/rcu/tree_plugin.h
index b76076014e12..1514909cf905 100644
--- a/kernel/rcu/tree_plugin.h
+++ b/kernel/rcu/tree_plugin.h
@@ -641,7 +641,8 @@ static void rcu_read_unlock_special(struct task_struct *t)
 
 		expboost = (t->rcu_blocked_node && READ_ONCE(t->rcu_blocked_node->exp_tasks)) ||
 			   (rdp->grpmask & READ_ONCE(rnp->expmask)) ||
-			   IS_ENABLED(CONFIG_RCU_STRICT_GRACE_PERIOD) ||
+			   (IS_ENABLED(CONFIG_RCU_STRICT_GRACE_PERIOD) &&
+			   ((rdp->grpmask & READ_ONCE(rnp->qsmask)) || t->rcu_blocked_node)) ||
 			   (IS_ENABLED(CONFIG_RCU_BOOST) && irqs_were_disabled &&
 			    t->rcu_blocked_node);
 		// Need to defer quiescent state until everything is enabled.
-- 
2.25.1