linux-kernel - Re: [tip:sched/core] sched/preempt: Optimize preemption operations on _

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives
Hash Suite: Windows password security audit tool. GUI, reports in PDF.
[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]
Date:	Mon, 20 Jul 2015 18:52:35 -0400
From:	Sasha Levin <sasha.levin@...cle.com>
To:	Frederic Weisbecker <fweisbec@...il.com>
CC:	linux-kernel@...r.kernel.org, mingo@...nel.org, tglx@...utronix.de,
	hpa@...or.com, torvalds@...ux-foundation.org, peterz@...radead.org,
	"Paul E. McKenney" <paulmck@...ux.vnet.ibm.com>
Subject: Re: [tip:sched/core] sched/preempt: Optimize preemption operations
 on __schedule() callers

On 07/20/2015 06:48 PM, Frederic Weisbecker wrote:
> On Mon, Jul 20, 2015 at 06:14:27PM -0400, Sasha Levin wrote:
>> > On 05/19/2015 03:17 AM, tip-bot for Frederic Weisbecker wrote:
>>> > > Commit-ID:  b30f0e3ffedfa52b1d67a302ae5860c49998e5e2
>>> > > Gitweb:     http://git.kernel.org/tip/b30f0e3ffedfa52b1d67a302ae5860c49998e5e2
>>> > > Author:     Frederic Weisbecker <fweisbec@...il.com>
>>> > > AuthorDate: Tue, 12 May 2015 16:41:49 +0200
>>> > > Committer:  Ingo Molnar <mingo@...nel.org>
>>> > > CommitDate: Tue, 19 May 2015 08:39:12 +0200
>>> > > 
>>> > > sched/preempt: Optimize preemption operations on __schedule() callers
>>> > > 
>>> > > __schedule() disables preemption and some of its callers
>>> > > (the preempt_schedule*() family) also set PREEMPT_ACTIVE.
>>> > > 
>>> > > So we have two preempt_count() modifications that could be performed
>>> > > at once.
>>> > > 
>>> > > Lets remove the preemption disablement from __schedule() and pull
>>> > > this responsibility to its callers in order to optimize preempt_count()
>>> > > operations in a single place.
>>> > > 
>>> > > Suggested-by: Linus Torvalds <torvalds@...ux-foundation.org>
>>> > > Signed-off-by: Frederic Weisbecker <fweisbec@...il.com>
>>> > > Signed-off-by: Peter Zijlstra (Intel) <peterz@...radead.org>
>>> > > Cc: Peter Zijlstra <peterz@...radead.org>
>>> > > Cc: Thomas Gleixner <tglx@...utronix.de>
>>> > > Link: http://lkml.kernel.org/r/1431441711-29753-5-git-send-email-fweisbec@gmail.com
>>> > > Signed-off-by: Ingo Molnar <mingo@...nel.org>
>> > 
>> > Hi all,
>> > 
>> > I was seeing RCU stall warnings that appeared to have different backtrace
>> > each time, but would reliably trigger when fuzzing and looked something like this:
>> > 
>> > [4394886.414687] INFO: rcu_preempt detected stalls on CPUs/tasks:
>> > [4394886.416717]        Tasks blocked on level-0 rcu_node (CPUs 0-23): P13356
>> > [4394886.418699]        (detected by 2, t=20502 jiffies, g=3548, c=3547, q=120)
>> > [4394886.420712] trinity-c42     R  running task    26936 13356   9574 0x10000000
>> > [4394886.422466]  ffff8807d1c7fbe8 ffff8807d1c7fb88 0000000000000022 ffff8801081e1bb0
>> > [4394886.423711]  ffff8801081e1b88 ffff8807d1eb8780 ffff8801081e11d8 ffff8807e2564000
>> > [4394886.424956]  ffff8807d1eb8000 ffff8807d1c7fbd8 ffff8807d1c78000 0000000000000000
>> > [4394886.426196] Call Trace:
>> > [4394886.426627] preempt_schedule_irq (./arch/x86/include/asm/paravirt.h:807 kernel/sched/core.c:3218)
>> > [4394886.427582] ? lockdep_reset_lock (kernel/locking/lockdep.c:3105)
>> > [4394886.428548] ? kill_pid_info (include/linux/rcupdate.h:857 kernel/signal.c:1340)
>> > [4394886.429438] retint_kernel (arch/x86/entry/entry_64.S:578)
>> > [4394886.430286] ? kill_pid_info (include/linux/rcupdate.h:857 kernel/signal.c:1340)
>> > [4394886.431176] ? native_restore_fl (./arch/x86/include/asm/irqflags.h:35)
>> > [4394886.432376] lock_is_held (kernel/locking/lockdep.c:3661)
>> > [4394886.433241] ? kill_pid_info (include/linux/rcupdate.h:914 kernel/signal.c:1344)
>> > [4394886.434135] rcu_read_lock_held (kernel/rcu/update.c:275)
>> > [4394886.435012] pid_task (kernel/pid.c:440 (discriminator 5))
>> > [4394886.435656] kill_pid_info (kernel/signal.c:1341)
>> > [4394886.436351] ? kill_pid_info (include/linux/rcupdate.h:857 kernel/signal.c:1340)
>> > [4394886.437065] SYSC_kill (kernel/signal.c:1426 kernel/signal.c:2903)
>> > [4394886.437726] ? SYSC_kill (include/linux/rcupdate.h:857 kernel/signal.c:1425 kernel/signal.c:2903)
>> > [4394886.438419] ? find_get_pid (include/linux/rcupdate.h:914 kernel/pid.c:494)
>> > [4394886.439134] ? kill_pid (kernel/signal.c:2894)
>> > [4394886.439782] ? find_get_pid (kernel/pid.c:497)
>> > [4394886.440492] ? find_get_pid (kernel/pid.c:489)
>> > [4394886.441183] ? lock_is_held (kernel/locking/lockdep.c:3661)
>> > [4394886.441929] ? rcu_read_lock_sched_held (kernel/rcu/update.c:109)
>> > [4394886.442772] ? syscall_trace_enter_phase2 (arch/x86/entry/common.c:196)
>> > [4394886.443632] SyS_kill (kernel/signal.c:2893)
>> > [4394886.444255] tracesys_phase2 (arch/x86/entry/entry_64.S:270)
>> > 
>> > I worked with Paul to rule out RCU as the cause.
>> > 
>> > I've noticed that all traces had one thing in common: being stuck in preempt_schedule_irq(),
>> > so I've looked at recent changes there and noticed this commit.
>> > 
>> > I've tried testing the commit before that, and the problem went away. Checking out this
>> > commit the problem reappeared.
> Do you have any idea how to reproduce that?

Beyond running trinity, no. That's why I couldn't bisect it to begin with.


Thanks,
Sasha
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/