linux-kernel - int3 doing rcu_read

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [thread-next>] [day] [month] [year] [list]

Message-ID: <20131018231351.28c2036d@gandalf.local.home>
Date:	Fri, 18 Oct 2013 23:13:51 -0400
From:	Steven Rostedt <rostedt@...dmis.org>
To:	"Paul E. McKenney" <paulmck@...ux.vnet.ibm.com>,
	LKML <linux-kernel@...r.kernel.org>
Cc:	Ingo Molnar <mingo@...nel.org>,
	"H. Peter Anvin" <hpa@...ux.intel.com>,
	Thomas Gleixner <tglx@...utronix.de>
Subject: int3 doing rcu_read_lock()

Hey Paul,

I hit this in my tests:

[ 1597.688015] ===============================
[ 1597.688015] [ INFO: suspicious RCU usage. ]
[ 1597.688015] 3.12.0-rc4-test+ #48 Not tainted
[ 1597.688015] -------------------------------
[ 1597.688015] /home/rostedt/work/git/linux-trace.git/include/linux/rcupdate.h:775 rcu_read_lock() used illegally while idle!
[ 1597.688015] 
[ 1597.688015] other info that might help us debug this:
[ 1597.688015] 
[ 1597.688015] 
[ 1597.688015] RCU used illegally from idle CPU!
[ 1597.688015] rcu_scheduler_active = 1, debug_locks = 0
[ 1597.688015] RCU used illegally from extended quiescent state!
[ 1597.688015] 1 lock held by swapper/0/0:
[ 1597.688015]  #0:  (rcu_read_lock){.+.+..}, at: [<ffffffff810689ab>] rcu_lock_acquire+0x0/0x29
[ 1597.688015] 
[ 1597.688015] stack backtrace:
[ 1597.688015] CPU: 0 PID: 0 Comm: swapper/0 Not tainted 3.12.0-rc4-test+ #48
[ 1597.688015] Hardware name: To Be Filled By O.E.M. To Be Filled By O.E.M./To be filled by O.E.M., BIOS SDBLI944.86P 05/08/2007
[ 1597.688015]  0000000000000001 ffff88007a609e78 ffffffff8151125f ffffffff81a44f38
[ 1597.688015]  ffffffff81a10490 ffff88007a609ea8 ffffffff8109abbe ffff88007a609f08
[ 1597.688015]  ffffffff81a407d0 0000000000000002 00000000ffffffff ffff88007a609ee8
[ 1597.688015] Call Trace:
[ 1597.688015]  <#DB>  [<ffffffff8151125f>] dump_stack+0x52/0x8b
[ 1597.688015]  [<ffffffff8109abbe>] lockdep_rcu_suspicious+0x109/0x112
[ 1597.688015]  [<ffffffff81519596>] __atomic_notifier_call_chain+0x70/0xee
[ 1597.688015]  [<ffffffff81519628>] atomic_notifier_call_chain+0x14/0x16
[ 1597.688015]  [<ffffffff81519658>] notify_die+0x2e/0x30
[ 1597.688015]  [<ffffffff81516e60>] do_int3+0x4f/0x9c
[ 1597.688015]  [<ffffffff815166c4>] int3+0x34/0x50
[ 1597.688015]  [<ffffffff8109a221>] ? trace_hardirqs_off_caller+0x3f/0xac
[ 1597.688015]  [<ffffffff8151ea61>] ? do_IRQ+0x1/0xa4
[ 1597.688015]  <<EOE>>  <IRQ>  [<ffffffff8151646f>] ? common_interrupt+0x6f/0x6f
[ 1597.688015]  <EOI>  [<ffffffff8109dff0>] ? trace_hardirqs_on+0xd/0xf
[ 1597.688015]  [<ffffffff8109a221>] ? trace_hardirqs_off_caller+0x3f/0xac
[ 1597.688015]  [<ffffffff8100b60f>] ? default_idle+0x21/0x32
[ 1597.688015]  [<ffffffff8100b60d>] ? default_idle+0x1f/0x32
[ 1597.688015]  [<ffffffff8100bd90>] arch_cpu_idle+0x18/0x22
[ 1597.688015]  [<ffffffff8108a28b>] cpu_startup_entry+0x10b/0x16c
[ 1597.688015]  [<ffffffff81509876>] rest_init+0x13a/0x141
[ 1597.688015]  [<ffffffff8150973c>] ? csum_partial_copy_generic+0x16c/0x16c
[ 1597.688015]  [<ffffffff81cc4d91>] start_kernel+0x41c/0x429
[ 1597.688015]  [<ffffffff81cc46e9>] ? repair_env_string+0x56/0x56
[ 1597.688015]  [<ffffffff81cc448d>] x86_64_start_reservations+0x2a/0x2c
[ 1597.688015]  [<ffffffff81cc457a>] x86_64_start_kernel+0xeb/0xf2

When function tracing is being enabled, to avoid stop machine we add a
break point to all functions that are about to be traced, convert them,
and then remove them. ftrace adds a breakpoint handler to be called by
int3 that simply skips the code.

The problem is that the do_int3 calls notify_die which does the notify
handler which does a rcu_read_lock().

The problem is if do_IRQ gets called during this transition (as it was
above) from idle. The breakpoint is hit at the beginning of do_IRQ()
before it gets to call irq_enter(), which means rcu_irq_enter() isn't
called either.

I wonder if we should have a rcu_bp_enter(), that basically does what
rcu_irq_enter() does, but it would not be traced.

Thinking about this more, it seems that because breakpoints are used
everywhere function tracing can be used, we may need to fix the
breakpoint code not to call rcu_read_lock() as it can be just as
dangerous to have as function tracing. We may need to have a different
kind of notifier that breakpoints use :-/

Something to talk about in Edinburgh ;-)

-- Steve
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/