| lists.openwall.net | lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC | |
|
Open Source and information security mailing list archives
| ||
|
Date: Tue, 20 Jan 2015 11:09:19 +0800
From: Zhang Zhen <zhenzhang.zhang@...wei.com>
To: Don Zickus <dzickus@...hat.com>
CC: <paulmck@...ux.vnet.ibm.com>, <linux-kernel@...r.kernel.org>,
<morgan.wang@...wei.com>, <josh@...edesktop.org>,
<dipankar@...ibm.com>
Subject: Re: RCU CPU stall console spews leads to soft lockup disabled is
reasonable ?
On 2015/1/19 22:06, Don Zickus wrote:
> On Mon, Jan 19, 2015 at 05:04:29PM +0800, Zhang Zhen wrote:
>>>
>>> Did you really intend to acquire the same spinlock twice in a row,
>>> forcing a self-deadlock? If not, I of course suggest changing the second
>>> "spin_lock()" to "spin_unlock()".
>>>
>>
>> Yes, i acquire the same spinlock twice in order to reproduce the problem.
>>
>>> If your .config has CONFIG_TREE_RCU=y, the above is quite likely to
>>> give you an RCU CPU stall warning.
>>>
>>
>> In my .config CONFIG_TREE_RCU=y.
>>
>> If softlockup_thresh < rcu_cpu_stall_timeout, it will give soft lockup warning.
>> If softlockup_thresh > rcu_cpu_stall_timeout, it will likely to give RCU CPU stall warning
>> just like above and no give soft lockup warning.
>>
>> It means that RCU CPU stall console spews leads to soft lockup disabled.
>> Is this reasonable ?
>
> I believe so. In kernel v3.10.., all activity to the console executed
> touch_nmi_watchdog() which calls touch_softlockup_watchdog, which delayed
> the softlockup for another round of 'softlockup_thresh'.
>
Yeah, you are right. It's the real reason.
> Of course back then, touch_nmi_watchdog touched all cpus. So a problem
> like this was masked. I believe this upstream commit 62572e29bc53, solved
> the problem.
Thanks for your suggestion.
Commit 62572e29bc53 changed the semantics of touch_nmi_watchdog and make it
only touch local cpu not every one.
But watchdog_nmi_touch = true only guarantee no hard lockup check on this cpu.
Commit 62572e29bc53 didn't changed the semantics of touch_softlockup_watchdog.
>
> You can apply that commit and see if you if you get both RCU stall
> messages _and_ softlockup messages. I believe that is what you were
> expecting, correct?
>
Correct, i expect i can get both RCU stall messages _and_ softlockup messages.
I applied that commit, and i only got RCU stall messages.
/ #
/ # echo 60 > /proc/sys/kernel/watchdog_thresh
/ # busybox insmod softlockup_test.ko
[ 35.344060] INFO: rcu_preempt detected stalls on CPUs/tasks: {} (detected by 0, t=21002 jiffies, g=85, c=84, q=4)
[ 35.344060] INFO: Stall ended before state dump start
[ 98.349079] INFO: rcu_preempt detected stalls on CPUs/tasks: {} (detected by 0, t=84007 jiffies, g=85, c=84, q=4)
[ 98.349079] INFO: Stall ended before state dump start
[ 161.354100] INFO: rcu_preempt detected stalls on CPUs/tasks: {} (detected by 0, t=147012 jiffies, g=85, c=84, q=4)
[ 161.354100] INFO: Stall ended before state dump start
>
> Of course, on a non-virt guest, your test case would normally trigger a
> hardlockup warning first. And a later kernel version for the guest may
> actually do that (not quite sure if the emulated PMU stuff is upstream or
> not yet). Just to set your expectations correctly.
>
Yes, on a non-virt guest, my test case tiggered hardlockup warning firt.
Best regards!
> Cheers,
> Don
>
>>
>> Thanks!
>>
>>>> return 0;
>>>> }
>>>>
>>>> static int __init test_init(void)
>>>> {
>>>> hello_start();
>>>>
>>>> printk(KERN_INFO "Module init\n");
>>>> return 0;
>>>> }
>>>>
>>>> static void __exit test_exit(void)
>>>> {
>>>> printk(KERN_INFO "Module exit!\n");
>>>> }
>>>>
>>>> module_init(test_init)
>>>> module_exit(test_exit)
>>>> MODULE_LICENSE("GPL");
>>>> //
>>>>
>>>> My kernel version is v3.10.63, and i checked the kernel source code,
>>>>
>>>> update_process_times
>>>> -> run_local_timers
>>>> -> hrtimer_run_queues
>>>> -> __run_hrtimer
>>>> -> watchdog_timer_fn
>>>> -> is_softlockup
>>>>
>>>> -> rcu_check_callbacks
>>>> -> __rcu_pending
>>>> -> check_cpu_stall
>>>> -> print_cpu_stall
>>>>
>>>> If softlockup_thresh > rcu_cpu_stall_timeout, print_cpu_stall will print log to serial port.
>>>>
>>>> The 8250 serial driver will call serial8250_console_write => touch_nmi_watchdog() which reset
>>>> watchdog_touch_ts to 0. So the softlockup will not be triggered.
>>>>
>>>> Is this reasonable? Why?
>>>
>>> Is exactly what reasonable? ;-)
>>>
>>> Yes, it is reasonable that your code triggers an RCU CPU stall warning.
>>>
>>> No, it is not reasonable that the RCU CPU stall warning does not include
>>> a stack trace, and the fix for that bug will be going into the next merge
>>> window.
>>>
>>> Yes, is is reasonable that varying the softlockup and RCU CPU stall
>>> timeouts might change the behavior.
>>>
>>> No, your code is not reasonable, except perhaps as a test of the
>>> generation of softlockup and RCU CPU stall warnings. If you are not
>>> trying to test softlockup and RCU CPU stall warnings, you should of course
>>> not try to acquire any non-recursive exclusive lock that you already hold.
>>>
>>>> If it is not reasonable, we should adjust the printk loglevel from *KERN_ERR* to *KERN_INFO*
>>>> in print_cpu_stall.
>>>
>>> Given that RCU CPU stall warnings are supposed to be pointing out errors
>>> elsewhere in the kernel, and in this case are pointing out errors elsewhere
>>> in the kernel, namely in your hello_start() function, it is reasonable
>>> that the RCU CPU stall warnings use the KERN_ERR loglevel.
>>>
>>> Or am I missing something here?
>>>
>>> Thanx, Paul
>>>
>>>
>>> .
>>>
>>
>>
>
> .
>
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
Powered by blists - more mailing lists