linux-kernel - BUG-RT: scheduling while in atomic in the watchdog's hrtimer

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [thread-next>] [day] [month] [year] [list]

Message-ID: <d2eb047a-6c07-e176-aa2a-68a425efd0ae@redhat.com>
Date:   Thu, 21 Mar 2019 22:27:30 +0100
From:   Daniel Bristot de Oliveira <bristot@...hat.com>
To:     linux-rt-users <linux-rt-users@...r.kernel.org>
Cc:     Sebastian Andrzej Siewior <bigeasy@...utronix.de>,
        Peter Zijlstra <peterz@...radead.org>,
        Clark Williams <williams@...hat.com>,
        Tommaso Cucinotta <tommaso.cucinotta@...up.it>,
        Rômulo Silva de Oliveira 
        <romulo.deoliveira@...c.br>, LKML <linux-kernel@...r.kernel.org>
Subject: BUG-RT: scheduling while in atomic in the watchdog's hrtimer

Hi,

In the 4.19/5.0-rt configured !SMP, the following chain of events can
take place:

------------------------------ %< -------------------------------------
smp_apic_timer_interrupt(){
	hrtimer_interrupt() {
		__hrtimer_run_queues() {
			watchdog_timer_fn() {
				stop_one_cpu_nowait() {
					#ifdef !CONFIG_SMP
						schedule_work() {
							queue_work() {
								queue_work_on() {
								/* phew, long, ah!?.... */	
									local_lock_irqsave() {
										__local_lock_irqsave() {
											__local_lock_irq() {
												spin_lock_irqsave() {
													rt_spin_lock()...
														/* from here on you already know...*/
														/* a lot of } */
------------------------------ >% -------------------------------------

And this can cause a scheduling while in atomic.

Tracking down, the commit that introduced the stop_one_cpu_nowait()->
queue_work_on() to the path was:

------------------------------ %< -------------------------------------
commit 9cf57731b63e37ed995b46690adc604891a9a28f
Author: Peter Zijlstra 
Date:   Thu Jun 7 10:52:03 2018 +0200

    watchdog/softlockup: Replace "watchdog/%u" threads with cpu_stop_work
    
    Oleg suggested to replace the "watchdog/%u" threads with
    cpu_stop_work. That removes one thread per CPU while at the same time
    fixes softlockup vs SCHED_DEADLINE.
    
    But more importantly, it does away with the single
    smpboot_update_cpumask_percpu_thread() user, which allows
    cleanups/shrinkage of the smpboot interface.
    
    Suggested-by: Oleg Nesterov 
    Signed-off-by: Peter Zijlstra (Intel) 
    Cc: Linus Torvalds 
    Cc: Peter Zijlstra 
    Cc: Thomas Gleixner 
    Cc: linux-kernel@...r.kernel.org
    Signed-off-by: Ingo Molnar 
------------------------------ >% -------------------------------------

Later modified by:

------------------------------ %< -------------------------------------
commit be45bf5395e0886a93fc816bbe41a008ec2e42e2
Author: Peter Zijlstra 
Date:   Fri Jul 13 12:42:08 2018 +0200

    watchdog/softlockup: Fix cpu_stop_queue_work() double-queue bug
    
    When scheduling is delayed for longer than the softlockup interrupt
    period it is possible to double-queue the cpu_stop_work, causing list
    corruption.
    
    Cure this by adding a completion to track the cpu_stop_work's
    progress.
    
    Reported-by: kernel test robot 
    Tested-by: Rong Chen 
    Signed-off-by: Peter Zijlstra (Intel) 
    Cc: Linus Torvalds 
    Cc: Peter Zijlstra 
    Cc: Thomas Gleixner 
    Fixes: 9cf57731b63e ("watchdog/softlockup: Replace "watchdog/%u" threads with cpu_stop_work")
    Link: http://lkml.kernel.org/r/20180713104208.GW2494@hirez.programming.kicks-ass.net
    Signed-off-by: Ingo Molnar 
------------------------------ >% -------------------------------------

I hit it while trying the automata model against the 4.19-rt/5.0-rt 
kernels.

You can find more about it here:
http://bristot.me/another-bug-found-with-the-model/

Thanks
-- Daniel