linux-kernel - Re: RCU stall when using function

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Date:   Wed, 9 Aug 2017 05:58:04 -0700
From:   "Paul E. McKenney" <paulmck@...ux.vnet.ibm.com>
To:     Pratyush Anand <panand@...hat.com>
Cc:     김동현 <austinkernel.kim@...il.com>,
        Daniel Lezcano <daniel.lezcano@...aro.org>,
        john.stultz@...aro.org, Steven Rostedt <rostedt@...dmis.org>,
        linux-kernel@...r.kernel.org
Subject: Re: RCU stall when using function_graph

On Wed, Aug 09, 2017 at 02:43:49PM +0530, Pratyush Anand wrote:
> 
> 
> On Sunday 06 August 2017 10:32 PM, Paul E. McKenney wrote:
> >On Sat, Aug 05, 2017 at 02:24:21PM +0900, 김동현 wrote:
> >>Dear All
> >>
> >>As for me, after configuring function_graph as below, crash disappears.
> >>"echo 0 > d/tracing/tracing_on"
> >>"sleep 1"
> >>
> >>"echo function_graph > d/tracing/current_tracer"
> >>"sleep 1"
> >>
> >>"echo smp_call_function_single > d/tracing/set_ftrace_filter"
> 
> It will limit trace output to only for the filtered function
> (smp_call_function_single).
> 
> >>adb shell "sleep 1"
> >>
> >>"echo 1 > d/tracing/tracing_on"
> >>adb shell "sleep 1"
> >>
> >>Right after function_graph is enabled, too many logs are traced upon IRQ
> >>transaction which many times eventually causes stall.
> >
> >That would do it!
> >
> >Hmmm...
> >
> >Steven, would it be helpful if RCU were to inform tracing (say) halfway
> >through the RCU CPU stall interval, allowing the tracer to do something
> >like cond_resched_rcu_qs()?  I can imagine all sorts of reasons why this
> >wouldn't work, for example, if all the tracing was with irqs disabled
> >or some such, but figured I should ask.
> >
> >Does Guillermo's approach work for others?
> 
> Limited output with a couple of filtered function will definitely
> not cause RCU schedule stall. But the question is whether we should
> expect a full function graph trace working on every platform or not
> (specially the one which generates high interrupts)?

It might well be that the user must disable RCU CPU stall warnings via
the rcu_cpu_stall_suppress sysfs entry (or increase their timeout via the
rcu_cpu_stall_timeout sysfs entry) before doing something that greatly
increases overhead.  Like enabling large quantities of tracing.  ;-)

It -might- be possible to do this automatically, but reliable
automation would require that tracing understand how often each
function was called, which sounds to me to be a bit of a stretch.

Thoughts?

							Thanx, Paul