| lists.openwall.net | lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC | |
|
Open Source and information security mailing list archives
| ||
|
Message-ID: <CAOxq_8MM08-7AdUf5QxH_er7n5OMiiw7iRNZ2ZsOSWRFrQD9CQ@mail.gmail.com>
Date: Mon, 14 Dec 2015 21:54:41 +0530
From: Ani Sinha <ani@...sta.com>
To: Paul McKenney <paulmck@...ux.vnet.ibm.com>
Cc: Rik van Riel <riel@...hat.com>,
Randy Dunlap <rdunlap@...radead.org>,
Richard Weinberger <richard@....at>,
"linux-kernel@...r.kernel.org" <linux-kernel@...r.kernel.org>,
Ivan Delalande <colona@...sta.com>,
fruggeri <fruggeri@...sta.com>
Subject: Re: new warning on sysrq kernel crash trigger
Rik, any comments?
On Sat, Dec 12, 2015 at 6:33 AM, Paul E. McKenney
<paulmck@...ux.vnet.ibm.com> wrote:
> On Fri, Dec 11, 2015 at 04:16:37PM -0800, Ani Sinha wrote:
>> On Fri, 11 Dec 2015, Paul E. McKenney wrote:
>> > On Fri, Dec 11, 2015 at 05:10:43PM -0500, Rik van Riel wrote:
>> > > On 12/11/2015 03:44 PM, Ani Sinha wrote:
>> > > >
>> > > >
>> > > > On Thu, 10 Dec 2015, Paul E. McKenney wrote:
>> > > >
>> > > >> On Thu, Dec 10, 2015 at 03:57:09PM -0800, Ani Sinha wrote:
>> > > >>> Hi guys
>> > > >>>
>> > > >>> I am noticing a new warning in linux 3.18 which we did not see before
>> > > >>> in linux 3.4 :
>> > > >>>
>> > > >>> bash-4.1# echo c > /proc/sysrq-trigger
>> > > >>> [ 978.807185] BUG: sleeping function called from invalid context at
>> > > >>> ../arch/x86/mm/fault.c:1187
>> > > >>> [ 978.909816] in_atomic(): 0, irqs_disabled(): 0, pid: 4706, name: bash
>> > > >>> [ 978.987358] Preemption disabled at:[<ffffffff81484339>] printk+0x48/0x4a
>> > > >>>
>> > > >>>
>> > > >>> I have bisected this to the following change :
>> > > >>>
>> > > >>> commit 984d74a72076a12b400339973e8c98fd2fcd90e5
>> > > >>> Author: Rik van Riel <riel@...hat.com>
>> > > >>> Date: Fri Jun 6 14:38:13 2014 -0700
>> > > >>>
>> > > >>> sysrq: rcu-ify __handle_sysrq
>> > > >>>
>> > > >>>
>> > > >>> the rcu_read_lock() in handle_sysrq() bumps up
>> > > >>> current->rcu_read_lock_nesting. Hence, in __do_page_fault() when it
>> > > >>> calls might_sleep() in x86/mm/fault.c line 1191,
>> > > >>> preempt_count_equals(0) returns false and hence the warning is
>> > > >>> printed.
>> > > >>>
>> > > >>> One way to handle this would be to do something like this:
>> > > >>>
>> > > >>> diff --git a/arch/x86/mm/fault.c b/arch/x86/mm/fault.c
>> > > >>> index eef44d9..d4dbe22 100644
>> > > >>> --- a/arch/x86/mm/fault.c
>> > > >>> +++ b/arch/x86/mm/fault.c
>> > > >>> @@ -1132,7 +1132,7 @@ __do_page_fault(struct pt_regs *regs, unsigned
>> > > >>> long error_code,
>> > > >>> * If we're in an interrupt, have no user context or are running
>> > > >>> * in a region with pagefaults disabled then we must not take the fault
>> > > >>> */
>> > > >>> - if (unlikely(faulthandler_disabled() || !mm)) {
>> > > >>> + if (unlikely(faulthandler_disabled() || rcu_preempt_depth() || !mm)) {
>> > > >>
>> > > >> This works if CONFIG_PREEMPT=y, but if CONFIG_PREEMPT=n, then
>> > > >> rcu_preempt_depth() unconditionally returns zero. And if
>> > > >> CONFIG_PREEMPT_COUNT=y && CONFIG_PREEMPT=n, you would still see
>> > > >> the might_sleep() splat.
>> > > >>
>> > > >> Maybe use SRCU instead of RCU for this purpose?
>> > > >>
>> > > >
>> > > > From ae232ce3fb167b2ad363bfac7aab69001bc55a50 Mon Sep 17 00:00:00 2001
>> > > > From: Ani Sinha <ani@...sta.com>
>> > > > Date: Fri, 11 Dec 2015 12:07:42 -0800
>> > > > Subject: [PATCH 1/1] Fix 'sleeping function called from invalid context'
>> > > > warning in sysrq generated crash.
>> > > >
>> > > > Commit 984d74a72076a1 ("sysrq: rcu-ify __handle_sysrq")
>> > > > replaced spin_lock_irqsave() calls with
>> > > > rcu_read_lock() calls in sysrq. Since rcu_read_lock() does not
>> > > > disable preemption, faulthandler_disabled() in
>> > > > __do_page_fault() in x86/fault.c returns false. When the code
>> > > > later calls might_sleep() in the pagefault handler, we get the
>> > > > following warning:
>> > > >
>> > > > BUG: sleeping function called from invalid context at ../arch/x86/mm/fault.c:1187
>> > > > in_atomic(): 0, irqs_disabled(): 0, pid: 4706, name: bash
>> > > > Preemption disabled at:[<ffffffff81484339>] printk+0x48/0x4a
>> > > >
>> > > > To fix this, replace RCU call in handle_sysrq() to use SRCU.
>> > >
>> > > The sysrq code can be called from irq context.
>> > >
>> > > Trying to use SRCU from an irq context sounds like it could
>> > > be a bad idea, though admittedly I do not know enough about
>> > > SRCU to know for sure :)
>> >
>> > Indeed, not the best idea! ;-)
>> >
>> > I could imagine something like this:
>> >
>> > if (in_irq())
>> > rcu_read_lock();
>> > else
>> > idx = srcu_read_lock(&sysrq_rcu);
>> >
>> > And ditto for unlock. Then, for the update:
>> >
>> > synchronize_rcu_mult(call_rcu, call_sysrq_srcu);
>> >
>> > Where:
>> >
>> > static void call_sysrq_srcu(struct rcu_head *head, rcu_callback_t func)
>> > {
>> > call_srcu(&sysrq_rcu, head, func);
>> > }
>> >
>>
>> >From ae232ce3fb167b2ad363bfac7aab69001bc55a50 Mon Sep 17 00:00:00 2001
>> From: Ani Sinha <ani@...sta.com>
>> Date: Fri, 11 Dec 2015 12:07:42 -0800
>> Subject: [PATCH 1/1] Fix 'sleeping function called from invalid context'
>> warning in sysrq generated crash.
>>
>> Commit 984d74a72076a1 ("sysrq: rcu-ify __handle_sysrq")
>> replaced spin_lock_irqsave() calls with
>> rcu_read_lock() calls in sysrq. Since rcu_read_lock() does not
>> disable preemption, faulthandler_disabled() in
>> __do_page_fault() in x86/fault.c returns false. When the code
>> later calls might_sleep() in the pagefault handler, we get the
>> following warning:
>>
>> BUG: sleeping function called from invalid context at ../arch/x86/mm/fault.c:1187
>> in_atomic(): 0, irqs_disabled(): 0, pid: 4706, name: bash
>> Preemption disabled at:[<ffffffff81484339>] printk+0x48/0x4a
>>
>> To fix this, replace RCU call in handle_sysrq() to use SRCU
>> in non-irq context.
>>
>> Tested this patch on linux 3.18 by booting off one of our boards.
>>
>> Fixes: 984d74a72076a1 ("sysrq: rcu-ify __handle_sysrq")
>>
>> Signed-off-by: Ani Sinha <ani@...sta.com>
>
> From an RCU perspective:
>
> Acked-by: Paul E. McKenney <paulmck@...ux.vnet.ibm.com>
>
> But I must defer to Rik from an sysrq perspective.
>
> Thanx, Paul
>
>> ---
>> diff --git a/drivers/tty/sysrq.c b/drivers/tty/sysrq.c
>> index 5381a72..df7d747 100644
>> --- a/drivers/tty/sysrq.c
>> +++ b/drivers/tty/sysrq.c
>> @@ -54,6 +54,7 @@
>> /* Whether we react on sysrq keys or just ignore them */
>> static int __read_mostly sysrq_enabled = CONFIG_MAGIC_SYSRQ_DEFAULT_ENABLE;
>> static bool __read_mostly sysrq_always_enabled;
>> +DEFINE_STATIC_SRCU(sysrq_rcu);
>>
>> static bool sysrq_on(void)
>> {
>> @@ -519,10 +520,13 @@ void __handle_sysrq(int key, bool check_mask)
>> {
>> struct sysrq_key_op *op_p;
>> int orig_log_level;
>> - int i;
>> + int i, idx;
>>
>> rcu_sysrq_start();
>> - rcu_read_lock();
>> + if (in_irq())
>> + rcu_read_lock();
>> + else
>> + idx = srcu_read_lock(&sysrq_rcu);
>> /*
>> * Raise the apparent loglevel to maximum so that the sysrq header
>> * is shown to provide the user with positive feedback. We do not
>> @@ -564,7 +568,10 @@ void __handle_sysrq(int key, bool check_mask)
>> pr_cont("\n");
>> console_loglevel = orig_log_level;
>> }
>> - rcu_read_unlock();
>> + if (in_irq())
>> + rcu_read_unlock();
>> + else
>> + srcu_read_unlock(&sysrq_rcu, idx);
>> rcu_sysrq_end();
>> }
>>
>> @@ -1040,6 +1047,11 @@ int sysrq_toggle_support(int enable_mask)
>> return 0;
>> }
>>
>> +static void call_sysrq_srcu(struct rcu_head *head, rcu_callback_t func)
>> +{
>> + call_srcu(&sysrq_rcu, head, func);
>> +}
>> +
>> static int __sysrq_swap_key_ops(int key, struct sysrq_key_op *insert_op_p,
>> struct sysrq_key_op *remove_op_p)
>> {
>> @@ -1059,7 +1071,7 @@ static int __sysrq_swap_key_ops(int key, struct sysrq_key_op *insert_op_p,
>> * Wait for it to go away before returning, so the code for an old
>> * op is not freed (eg. on module unload) while it is in use.
>> */
>> - synchronize_rcu();
>> + synchronize_rcu_mult(call_rcu, call_sysrq_srcu);
>>
>> return retval;
>> }
>>
>
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
Powered by blists - more mailing lists