linux-kernel - Re: Question on task_blocks_on_rt

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Date:   Thu, 3 Sep 2020 13:06:39 -0700
From:   "Paul E. McKenney" <paulmck@...nel.org>
To:     Davidlohr Bueso <dave@...olabs.net>
Cc:     peterz@...radead.org, mingo@...hat.com, will@...nel.org,
        linux-kernel@...r.kernel.org
Subject: Re: Question on task_blocks_on_rt_mutex()

On Wed, Sep 02, 2020 at 08:54:10AM -0700, Paul E. McKenney wrote:
> On Tue, Sep 01, 2020 at 06:51:28PM -0700, Davidlohr Bueso wrote:
> > On Tue, 01 Sep 2020, Paul E. McKenney wrote:
> > 
> > > And it appears that a default-niced CPU-bound SCHED_OTHER process is
> > > not preempted by a newly awakened MAX_NICE SCHED_OTHER process.  OK,
> > > OK, I never waited for more than 10 minutes, but on my 2.2GHz that is
> > > close enough to a hang for most people.
> > > 
> > > Which means that the patch below prevents the hangs.  And maybe does
> > > other things as well, firing rcutorture up on it to check.
> > > 
> > > But is this indefinite delay expected behavior?
> > > 
> > > This reproduces for me on current mainline as follows:
> > > 
> > > tools/testing/selftests/rcutorture/bin/kvm.sh --allcpus --torture lock --duration 3 --configs LOCK05
> > > 
> > > This hangs within a minute of boot on my setup.  Here "hangs" is defined
> > > as stopping the per-15-second console output of:
> > > 	Writes:  Total: 569906696 Max/Min: 81495031/63736508   Fail: 0
> > 
> > Ok this doesn't seem to be related to lockless wake_qs then. fyi there have
> > been missed wakeups in the past where wake_q_add() fails the cmpxchg because
> > the task is already pending a wakeup leading to the actual wakeup ocurring
> > before its corresponding wake_up_q(). This is why we have wake_q_add_safe().
> > But for rtmutexes, because there is no lock stealing only top-waiter is awoken
> > as well as try_to_take_rt_mutex() is done under the lock->wait_lock I was not
> > seeing an actual race here.
> 
> This problem is avoided if stutter_wait() does the occasional sleep.
> I would have expected preemption to take effect, but even setting the
> kthreads in stutter_wait() to MAX_NICE doesn't help.  The current fix
> destroys intended instant-on nature of stutter_wait(), so the eventual
> fix will need to use hrtimer-based sleeps or some such.

And here is my current best shot at a workaround/fix/whatever.  Thoughts?

							Thanx, Paul

------------------------------------------------------------------------

commit d93a64389f4d544ded241d0ba30b2586497f5dc0
Author: Paul E. McKenney <paulmck@...nel.org>
Date:   Tue Sep 1 16:58:41 2020 -0700

    torture: Periodically pause in stutter_wait()
    
    Running locktorture scenario LOCK05 results in hangs:
    
    tools/testing/selftests/rcutorture/bin/kvm.sh --allcpus --torture lock --duration 3 --configs LOCK05
    
    The lock_torture_writer() kthreads set themselves to MAX_NICE while
    running SCHED_OTHER.  Other locktorture kthreads run at default niceness,
    also SCHED_OTHER.  This results in these other locktorture kthreads
    indefinitely preempting the lock_torture_writer() kthreads.  Note that
    the cond_resched() in the stutter_wait() function's loop is ineffective
    because this scenario is built with CONFIG_PREEMPT=y.
    
    It is not clear that such indefinite preemption is supposed to happen, but
    in the meantime this commit prevents kthreads running in stutter_wait()
    from being completely CPU-bound, thus allowing the other threads to get
    some CPU in a timely fashion.  This commit also uses hrtimers to provide
    very short sleeps to avoid degrading the sudden-on testing that stutter
    is supposed to provide.
    
    Signed-off-by: Paul E. McKenney <paulmck@...nel.org>

diff --git a/kernel/torture.c b/kernel/torture.c
index 1061492..5488ad2 100644
--- a/kernel/torture.c
+++ b/kernel/torture.c
@@ -602,8 +602,11 @@ static int stutter_gap;
  */
 bool stutter_wait(const char *title)
 {
-	int spt;
+	ktime_t delay;
+	unsigned i = 0;
+	int oldnice;
 	bool ret = false;
+	int spt;
 
 	cond_resched_tasks_rcu_qs();
 	spt = READ_ONCE(stutter_pause_test);
@@ -612,8 +615,17 @@ bool stutter_wait(const char *title)
 		if (spt == 1) {
 			schedule_timeout_interruptible(1);
 		} else if (spt == 2) {
-			while (READ_ONCE(stutter_pause_test))
+			oldnice = task_nice(current);
+			set_user_nice(current, MAX_NICE);
+			while (READ_ONCE(stutter_pause_test)) {
+				if (!(i++ & 0xffff)) {
+					set_current_state(TASK_INTERRUPTIBLE);
+					delay = 10 * NSEC_PER_USEC;
+					schedule_hrtimeout(&delay, HRTIMER_MODE_REL);
+				}
 				cond_resched();
+			}
+			set_user_nice(current, oldnice);
 		} else {
 			schedule_timeout_interruptible(round_jiffies_relative(HZ));
 		}