lists.openwall.net   lists  /  announce  owl-users  owl-dev  john-users  john-dev  passwdqc-users  yescrypt  popa3d-users  /  oss-security  kernel-hardening  musl  sabotage  tlsify  passwords  /  crypt-dev  xvendor  /  Bugtraq  Full-Disclosure  linux-kernel  linux-netdev  linux-ext4  linux-hardening  linux-cve-announce  PHC 
Open Source and information security mailing list archives
 
Hash Suite: Windows password security audit tool. GUI, reports in PDF.
[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]
Date:	Sun, 27 Mar 2011 03:30:34 +0200
From:	Sedat Dilek <sedat.dilek@...glemail.com>
To:	paulmck@...ux.vnet.ibm.com
Cc:	Josh Triplett <josh@...htriplett.org>,
	linux-next <linux-next@...r.kernel.org>,
	LKML <linux-kernel@...r.kernel.org>,
	Stephen Rothwell <sfr@...b.auug.org.au>,
	Randy Dunlap <randy.dunlap@...cle.com>,
	"Theodore Ts'o" <tytso@....edu>, Jens Axboe <axboe@...nel.dk>,
	Tejun Heo <tj@...nel.org>, Al Viro <viro@...iv.linux.org.uk>,
	Nick Piggin <npiggin@...nel.dk>
Subject: Re: linux-next: Tree for March 25 (Call trace: RCU|workqueues|block|VFS|ext4
 related?)

On Sun, Mar 27, 2011 at 1:09 AM, Paul E. McKenney
<paulmck@...ux.vnet.ibm.com> wrote:
> On Sat, Mar 26, 2011 at 11:15:22PM +0100, Sedat Dilek wrote:
>> On Sat, Mar 26, 2011 at 5:02 PM, Paul E. McKenney
>> <paulmck@...ux.vnet.ibm.com> wrote:
>> > On Sat, Mar 26, 2011 at 01:34:53PM +0100, Sedat Dilek wrote:
>> [...]
>> >> Just FYI: Changed to the following settings:
>> >>
>> >> - Enable Preemptible Kernel (Low-Latency Desktop)
>> >> - Enable Preemptible tree-based hierarchical RCU
>> >> - Enable RCU priority boosting
>> >> - Reset RCU CPU stall timeout to default (60 seconds)
>> >>
>> >> So far I see no RCU stalls in the logs and my system runs as expected.
>> >> ( I have noticed here some "stalling" in the webbrowser, but I can do
>> >> my daily business. )
>> >
>> > OK, good to see some progress!
>> >
>>
>> On 1st impression thing went fine, but after a while jobs like opening
>> several tabs in firefox or doing a simple df command stalled the
>> machine. No, my system even got frozen and required a brutal reset.
>>
>> >From the logs (more see file-attachment):
>>
>> Mar 26 19:58:40 tbox kernel: [ 1440.640060] INFO: task systemd:1
>> blocked for more than 120 seconds.
>> Mar 26 19:58:40 tbox kernel: [ 1440.640074] "echo 0 >
>> /proc/sys/kernel/hung_task_timeout_secs" disables this message.
>>
>> Following it -> NOPE
>> $ echo 0 > /proc/sys/kernel/hung_task_timeout_secs
>
> These are tasks that are blocked, not tasks that are consuming CPU.
>
>> > Is there a runaway process consuming CPU?  The reason that I ask is that
>> > an infinite loop in the kernel can result in a stall when PREEMPT=n
>> > but is less likely to if PREEMPT=y.  Could you please check with "top",
>> > "ps", or whatever?
>>
>> Unsure what you mean by this, as you can see from the logs, it's not
>> only one special task "stalling".
>> BTW, I have systemd here running.
>
> Right.  But I need to know if there are tasks consuming lots of CPU,
> which is different than tasks that are stalled for a long time.  Look
> at the message: it says "blocked for more than 120 seconds", not
> "running for more than 120 seconds".
>
> For example, if one of the RCU kthreads is consuming lots of CPU, that
> would tell me that I should look for an RCU bug (and yank the patches
> from -next in the meantime).  On the other hand, if some other task is
> consuming lots of CPU, then that would be a hint as to where to find
> the bug.
>
>> >> I am not sure what the change to PREEMPT exactly mean in the end.
>> >> ( Let's work with this new kernel and carefully check for possible
>> >> side-effects. )
>> >> For example CONFIG_RCU_FAST_NO_HZ=y is now dropped, where the Kconfig
>> >> descriptive text says some words on better energy saving. For a
>> >> notebook this is no good.
>> >
>> > CONFIG_RCU_FAST_NO_HZ is of no use on a uniprocessor system, so OK
>> > to disable it.  But are you saying that CONFIG_RCU_FAST_NO_HZ=y
>> > results in problems that are removed by CONFIG_RCU_FAST_NO_HZ=n?
>> > That would be a surprise, and I need to know if this is the case.
>>
>> In my current setup (PREEMPT and TREE_PREEMPT_RCU)
>> CONFIG_RCU_FAST_NO_HZ is not considered and set via Kconfig-system
>> (see excerpt below).
>> But when you say for UP it is of no use, I need no more info.
>
> OK, "of no use" is overstating things a bit.  But CONFIG_RCU_FAST_NO_HZ's
> main purpose is to get the last CPU into dyntick-idle state
> (CONFIG_NO_HZ), which is most useful if the system has several CPUs.
>
>> Might be good to add some recommended (and/or useless) kernel-config
>> settings to RCU/UP.txt?
>>
>> [ init/Kconfig ]
>> config RCU_FAST_NO_HZ
>>         bool "Accelerate last non-dyntick-idle CPU's grace periods"
>>         depends on TREE_RCU && NO_HZ && SMP
>>         default n
>
> The "depends on TREE_RCU && NO_HZ && SMP" already excludes it from
> UP kernel builds, so no need to document.
>
>> >> I have also questions to some Kconfig dependencies, for example why I
>> >> can't select TREE_PREEMPT_RCU if CONFIG_PREEMPT_VOLUNTARY=y, etc.
>> >> Intended?
>> >
>> > Yes.  There is no point in TREE_PREEMPT_RCU unless PREEMPT=y.
>> >
>>
>> OK.
>>
>> >> Maybe I collect all my askings in a separate email to RCU folks and ML
>> >> and do not disturb further people from other sub-trees.
>> >>
>> >> I enjoyed to read the numerous docs in Documentation/RCU/ (and noticed
>> >> some typos as well).
>> >> The RCU folk gave the word "FAQ" a new meaning: Frequenty Asked
>> >> Questions & Q*uiz* :-).
>> >>
>> >> Thanks for the helpful hints and explanations from the RCU folks!
>> >
>> > Glad you liked them!  ;-)
>> >
>>
>> Other sub-trees lack of no good or up2date docs.
>>
>> >> - Sedat -
>> >>
>> >> P.S.: Current RCU and HZ kernel-config settings
>> >>
>> >> # grep RCU /boot/config-$(uname -r)
>> >> # RCU Subsystem
>> >> CONFIG_TREE_PREEMPT_RCU=y
>> >> CONFIG_PREEMPT_RCU=y
>> >> CONFIG_RCU_TRACE=y
>> >> CONFIG_RCU_FANOUT=32
>> >> # CONFIG_RCU_FANOUT_EXACT is not set
>> >> CONFIG_TREE_RCU_TRACE=y
>> >> CONFIG_RCU_BOOST=y
>> >> CONFIG_RCU_BOOST_PRIO=1
>> >> CONFIG_RCU_BOOST_DELAY=500
>> >> # CONFIG_SPARSE_RCU_POINTER is not set
>> >> # CONFIG_RCU_TORTURE_TEST is not set
>> >> CONFIG_RCU_CPU_STALL_TIMEOUT=60
>> >> CONFIG_RCU_CPU_STALL_VERBOSE=y
>> >>
>> >> # grep _HZ /boot/config-$(uname -r)
>> >> CONFIG_NO_HZ=y
>> >> # CONFIG_HZ_100 is not set
>> >> CONFIG_HZ_250=y
>> >> # CONFIG_HZ_300 is not set
>> >> # CONFIG_HZ_1000 is not set
>> >> CONFIG_HZ=250
>> >
>> > OK, thank you for the info!
>> >
>>
>> N.P.
>>
>> >                                                        Thanx, Paul
>> >
>>
>> I guees I will revert step-by-step the RCU commits in linux-next and report.
>> This weekend I wanted to enhance Debian's live-cd framework with
>> overlayfs support and a customized kernel.
>> But then came RCU :-(.
>
> Well, if it turns out to be a problem in RCU I will certainly apologize.
>

No, that's not so dramatic.
Dealing with this RCU issue has nice side-effects: I remembered (and
finally did) to use a reduced kernel-config set.
The base for it I created with 'make localmodconfig' and did some
manual fine-tuning afterwards (throw out media, rc, dvd, unneeded FSs,
etc.).
Also, I can use fresh gcc-4.6 (4.6.0-1) from the official Debian repos.

So, I started building with
"revert-rcu-patches/0001-Revert-rcu-introduce-kfree_rcu.patch".
I will let you know.

- Sedat -

>> Can you say some words to kfree_rcu.2011.03.25b (rcu/kfree_rcu) GIT branch(es)?
>
> These are just applying Lia Jiangshan's kfree_rcu() to a number of places
> in the kernel.  You can safely ignore them.
>
>                                                                Thanx, Paul
>
>> - Sedat -
>
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Powered by blists - more mailing lists