linux-kernel - Re: linux-next: Tree for March 25 (Call trace: RCU|workqueues|block|VFS|ext4 related?)

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <AANLkTikzLm+ArYkG20F41JOgfNygfdhxC0+A4tV4ez-R@mail.gmail.com>
Date:	Sat, 26 Mar 2011 23:15:22 +0100
From:	Sedat Dilek <sedat.dilek@...glemail.com>
To:	paulmck@...ux.vnet.ibm.com
Cc:	Josh Triplett <josh@...htriplett.org>,
	linux-next <linux-next@...r.kernel.org>,
	LKML <linux-kernel@...r.kernel.org>,
	Stephen Rothwell <sfr@...b.auug.org.au>,
	Randy Dunlap <randy.dunlap@...cle.com>,
	"Theodore Ts'o" <tytso@....edu>, Jens Axboe <axboe@...nel.dk>,
	Tejun Heo <tj@...nel.org>, Al Viro <viro@...iv.linux.org.uk>,
	Nick Piggin <npiggin@...nel.dk>
Subject: Re: linux-next: Tree for March 25 (Call trace: RCU|workqueues|block|VFS|ext4
 related?)

On Sat, Mar 26, 2011 at 5:02 PM, Paul E. McKenney
<paulmck@...ux.vnet.ibm.com> wrote:
> On Sat, Mar 26, 2011 at 01:34:53PM +0100, Sedat Dilek wrote:
[...]
>> Just FYI: Changed to the following settings:
>>
>> - Enable Preemptible Kernel (Low-Latency Desktop)
>> - Enable Preemptible tree-based hierarchical RCU
>> - Enable RCU priority boosting
>> - Reset RCU CPU stall timeout to default (60 seconds)
>>
>> So far I see no RCU stalls in the logs and my system runs as expected.
>> ( I have noticed here some "stalling" in the webbrowser, but I can do
>> my daily business. )
>
> OK, good to see some progress!
>

On 1st impression thing went fine, but after a while jobs like opening
several tabs in firefox or doing a simple df command stalled the
machine. No, my system even got frozen and required a brutal reset.

>From the logs (more see file-attachment):

Mar 26 19:58:40 tbox kernel: [ 1440.640060] INFO: task systemd:1
blocked for more than 120 seconds.
Mar 26 19:58:40 tbox kernel: [ 1440.640074] "echo 0 >
/proc/sys/kernel/hung_task_timeout_secs" disables this message.

Following it -> NOPE
$ echo 0 > /proc/sys/kernel/hung_task_timeout_secs

> Is there a runaway process consuming CPU?  The reason that I ask is that
> an infinite loop in the kernel can result in a stall when PREEMPT=n
> but is less likely to if PREEMPT=y.  Could you please check with "top",
> "ps", or whatever?
>

Unsure what you mean by this, as you can see from the logs, it's not
only one special task "stalling".
BTW, I have systemd here running.

>> I am not sure what the change to PREEMPT exactly mean in the end.
>> ( Let's work with this new kernel and carefully check for possible
>> side-effects. )
>> For example CONFIG_RCU_FAST_NO_HZ=y is now dropped, where the Kconfig
>> descriptive text says some words on better energy saving. For a
>> notebook this is no good.
>
> CONFIG_RCU_FAST_NO_HZ is of no use on a uniprocessor system, so OK
> to disable it.  But are you saying that CONFIG_RCU_FAST_NO_HZ=y
> results in problems that are removed by CONFIG_RCU_FAST_NO_HZ=n?
> That would be a surprise, and I need to know if this is the case.
>

In my current setup (PREEMPT and TREE_PREEMPT_RCU)
CONFIG_RCU_FAST_NO_HZ is not considered and set via Kconfig-system
(see excerpt below).
But when you say for UP it is of no use, I need no more info.
Might be good to add some recommended (and/or useless) kernel-config
settings to RCU/UP.txt?

[ init/Kconfig ]
config RCU_FAST_NO_HZ
        bool "Accelerate last non-dyntick-idle CPU's grace periods"
        depends on TREE_RCU && NO_HZ && SMP
        default n

>> I have also questions to some Kconfig dependencies, for example why I
>> can't select TREE_PREEMPT_RCU if CONFIG_PREEMPT_VOLUNTARY=y, etc.
>> Intended?
>
> Yes.  There is no point in TREE_PREEMPT_RCU unless PREEMPT=y.
>

OK.

>> Maybe I collect all my askings in a separate email to RCU folks and ML
>> and do not disturb further people from other sub-trees.
>>
>> I enjoyed to read the numerous docs in Documentation/RCU/ (and noticed
>> some typos as well).
>> The RCU folk gave the word "FAQ" a new meaning: Frequenty Asked
>> Questions & Q*uiz* :-).
>>
>> Thanks for the helpful hints and explanations from the RCU folks!
>
> Glad you liked them!  ;-)
>

Other sub-trees lack of no good or up2date docs.

>> - Sedat -
>>
>> P.S.: Current RCU and HZ kernel-config settings
>>
>> # grep RCU /boot/config-$(uname -r)
>> # RCU Subsystem
>> CONFIG_TREE_PREEMPT_RCU=y
>> CONFIG_PREEMPT_RCU=y
>> CONFIG_RCU_TRACE=y
>> CONFIG_RCU_FANOUT=32
>> # CONFIG_RCU_FANOUT_EXACT is not set
>> CONFIG_TREE_RCU_TRACE=y
>> CONFIG_RCU_BOOST=y
>> CONFIG_RCU_BOOST_PRIO=1
>> CONFIG_RCU_BOOST_DELAY=500
>> # CONFIG_SPARSE_RCU_POINTER is not set
>> # CONFIG_RCU_TORTURE_TEST is not set
>> CONFIG_RCU_CPU_STALL_TIMEOUT=60
>> CONFIG_RCU_CPU_STALL_VERBOSE=y
>>
>> # grep _HZ /boot/config-$(uname -r)
>> CONFIG_NO_HZ=y
>> # CONFIG_HZ_100 is not set
>> CONFIG_HZ_250=y
>> # CONFIG_HZ_300 is not set
>> # CONFIG_HZ_1000 is not set
>> CONFIG_HZ=250
>
> OK, thank you for the info!
>

N.P.

>                                                        Thanx, Paul
>

I guees I will revert step-by-step the RCU commits in linux-next and report.
This weekend I wanted to enhance Debian's live-cd framework with
overlayfs support and a customized kernel.
But then came RCU :-(.

Can you say some words to kfree_rcu.2011.03.25b (rcu/kfree_rcu) GIT branch(es)?

- Sedat -

View attachment "syslog-hung_task_timeout_secs.txt" of type "text/plain" (10949 bytes)