linux-kernel - Re: INFO: possible circular locking dependency detected

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <1310729362.2586.325.camel@twins>
Date:	Fri, 15 Jul 2011 13:29:22 +0200
From:	Peter Zijlstra <a.p.zijlstra@...llo.nl>
To:	Ed Tomlinson <edt@....ca>
Cc:	paulmck@...ux.vnet.ibm.com, Steven Rostedt <rostedt@...dmis.org>,
	Sergey Senozhatsky <sergey.senozhatsky@...il.com>,
	Ingo Molnar <mingo@...e.hu>,
	Thomas Gleixner <tglx@...utronix.de>,
	Andrew Morton <akpm@...ux-foundation.org>,
	Dipankar Sarma <dipankar@...ibm.com>,
	linux-kernel@...r.kernel.org
Subject: Re: INFO: possible circular locking dependency detected

On Fri, 2011-07-15 at 07:05 -0400, Ed Tomlinson wrote:
> Jul 14 23:21:18 grover kernel: [  920.659426] -> #1 (rcu_node_level_0){..-...}:
> Jul 14 23:21:18 grover kernel: [  920.659426]        [<ffffffff8108b7e5>] lock_acquire+0x95/0x140
> Jul 14 23:21:18 grover kernel: [  920.659426]        [<ffffffff8157808b>] _raw_spin_lock+0x3b/0x50
> Jul 14 23:21:18 grover kernel: [  920.659426]        [<ffffffff810ba797>] __rcu_read_unlock+0x197/0x2d0
> Jul 14 23:21:18 grover kernel: [  920.659426]        [<ffffffff8103f2f5>] select_task_rq_fair+0x585/0xa80
> Jul 14 23:21:18 grover kernel: [  920.659426]        [<ffffffff8104633b>] try_to_wake_up+0x17b/0x360
> Jul 14 23:21:18 grover kernel: [  920.659426]        [<ffffffff81046575>] wake_up_process+0x15/0x20
> Jul 14 23:21:18 grover kernel: [  920.659426]        [<ffffffff810528f4>] irq_exit+0xb4/0x100
> Jul 14 23:21:18 grover kernel: [  920.659426]        [<ffffffff8158197e>] smp_apic_timer_interrupt+0x6e/0x99
> Jul 14 23:21:18 grover kernel: [  920.659426]        [<ffffffff81580c53>] apic_timer_interrupt+0x13/0x20
> Jul 14 23:21:18 grover kernel: [  920.659426]        [<ffffffff810ba6e9>] __rcu_read_unlock+0xe9/0x2d0 
> Jul 14 23:21:18 grover kernel: [  920.659426]        [<ffffffff814c20d4>] sock_def_readable+0x94/0xc0

Ed, are you perchance running with force_irqthreads?

Paul, what appears to be happening here is that some rcu_read_unlock()
gets interrupted, possibly before calling rcu_read_unlock_special(),
possibly not if the interrupt is itself the timer interrupt.

Supposing ->rcu_read_unlock_special is set before, any wakeup happening
from an interrupt hitting __rcu_read_unlock():

void __rcu_read_unlock(void)
{
        struct task_struct *t = current;
                
        barrier();  /* needed if we ever invoke rcu_read_unlock in rcutree.c */
        --t->rcu_read_lock_nesting;
        barrier();  /* decrement before load of ->rcu_read_unlock_special */
        if (t->rcu_read_lock_nesting == 0 &&
            unlikely(ACCESS_ONCE(t->rcu_read_unlock_special)))
                rcu_read_unlock_special(t);
#ifdef CONFIG_PROVE_LOCKING
        WARN_ON_ONCE(ACCESS_ONCE(t->rcu_read_lock_nesting) < 0);
#endif /* #ifdef CONFIG_PROVE_LOCKING */
}

After --t->rcu_read_lock_nesting, but before calling
rcu_read_unlock_special(), will trigger this lock inversion.

The alternative case, ->rcu_read_unlock_special is not set yet, it can
be set if the interrupt hitting in that same spot above, is the timer
interrupt, and the wakeup happens either from the softirq ran from the
hard IRQ tail, or as I suspect here happens, the wakeup of ksoftirqd/#.


--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/