linux-kernel - Re: [PATCH] torture: Fix hang during kthread shutdown phase

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <20230103180404.GA4028633@paulmck-ThinkPad-P17-Gen-1>
Date:   Tue, 3 Jan 2023 10:04:04 -0800
From:   "Paul E. McKenney" <paulmck@...nel.org>
To:     Davidlohr Bueso <dave@...olabs.net>
Cc:     "Joel Fernandes (Google)" <joel@...lfernandes.org>,
        linux-kernel@...r.kernel.org,
        Frederic Weisbecker <fweisbec@...il.com>,
        Zhouyi Zhou <zhouzhouyi@...il.com>, stable@...r.kernel.org,
        Josh Triplett <josh@...htriplett.org>
Subject: Re: [PATCH] torture: Fix hang during kthread shutdown phase

On Mon, Jan 02, 2023 at 08:43:10AM -0800, Davidlohr Bueso wrote:
> On Sun, 01 Jan 2023, Joel Fernandes (Google) wrote:
> 
> > During shutdown of rcutorture, the shutdown thread in
> > rcu_torture_cleanup() calls torture_cleanup_begin() which sets fullstop
> > to FULLSTOP_RMMOD. This is enough to cause the rcutorture threads for
> > readers and fakewriters to breakout of their main while loop and start
> > shutting down.
> > 
> > Once out of their main loop, they then call torture_kthread_stopping()
> > which in turn waits for kthread_stop() to be called, however
> > rcu_torture_cleanup() has not even called kthread_stop() on those
> > threads yet, it does that a bit later.  However, before it gets a chance
> > to do so, torture_kthread_stopping() calls
> > schedule_timeout_interruptible(1) in a tight loop. Tracing confirmed
> > this makes the timer softirq constantly execute timer callbacks, while
> > never returning back to the softirq exit path and is essentially "locked
> > up" because of that. If the softirq preempts the shutdown thread,
> > kthread_stop() may never be called.
> > 
> > This commit improves the situation dramatically, by increasing timeout
> > passed to schedule_timeout_interruptible() 1/20th of a second. This
> > causes the timer softirq to not lock up a CPU and everything works fine.
> > Testing has shown 100 runs of TREE07 passing reliably, which was not the
> > case before because of RCU stalls.
> > 
> > Cc: Paul McKenney <paulmck@...nel.org>
> > Cc: Frederic Weisbecker <fweisbec@...il.com>
> > Cc: Zhouyi Zhou <zhouzhouyi@...il.com>
> > Cc: <stable@...r.kernel.org> # 6.0.x
> > Signed-off-by: Joel Fernandes (Google) <joel@...lfernandes.org>
> 
> Reviewed-by: Davidlohr Bueso <dave@...olabs.net>

Queued for further review and testing, thank you all!

One thing still puzzles me.  Assuming HZ=1000 and given 16 CPUs, each
timer hander must have consumed many tens of microseconds in order
to keep the system busy, which seems a bit longer than it should be.
Or am I underestimating the number of tasks involved?

							Thanx, Paul