linux-kernel - Re: [PATCH -tip v9 25/26] kprobes: Introduce kprobe cache to reduce cache misshits

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Date:	Thu, 24 Apr 2014 20:38:11 +0900
From:	Masami Hiramatsu <masami.hiramatsu.pt@...achi.com>
To:	Ingo Molnar <mingo@...nel.org>
Cc:	linux-kernel@...r.kernel.org, Andi Kleen <andi@...stfloor.org>,
	Ananth N Mavinakayanahalli <ananth@...ibm.com>,
	Sandeepa Prabhu <sandeepa.prabhu@...aro.org>,
	Frederic Weisbecker <fweisbec@...il.com>, x86@...nel.org,
	Steven Rostedt <rostedt@...dmis.org>, fche@...hat.com,
	mingo@...hat.com, systemtap@...rceware.org,
	"H. Peter Anvin" <hpa@...or.com>,
	Thomas Gleixner <tglx@...utronix.de>
Subject: Re: [PATCH -tip v9 25/26] kprobes: Introduce kprobe cache to reduce
 cache misshits

(2014/04/24 18:01), Ingo Molnar wrote:
> 
> * Masami Hiramatsu <masami.hiramatsu.pt@...achi.com> wrote:
> 
>> Introduce kprobe cache to reduce cache misshits for
>> massive multiple kprobes.
>> For stress testing kprobes, we need to activate kprobes
>> as many as possible. This situation causes cache miss
>> hit storm on kprobe hash-list. kprobe hashlist is already
>> enlarged to 4k entries and this is still small for 40k
>> kprobes.
>>
>> For example, when registering 40k probes on the hlist and
>> enabling 20k probes, perf tools shows still a lot of
>> cache-misses are on the get_kprobe.
>>   ----
>>   Samples: 633  of event 'cache-misses', Event count (approx.): 3414776
>>   +  68.13%  [k] get_kprobe
>>   +   4.38%  [k] ftrace_lookup_ip
>>   +   2.54%  [k] kprobe_ftrace_handler
>>   ----
>>
>> Also, I found that the most of the kprobes are not hit.
>> In that case, to reduce cache-misses, we can reduce the
>> random memory access by introducing a per-cpu cache which
>> caches the address of frequently used kprobe data structure
>> and its probe address.
>>
>> With kpcache enabled, the get_kprobe_cached goes down to
>> around 4-5% of cache-misses with 20k probes.
>>   ----
>>   Samples: 729  of event 'cache-misses', Event count (approx.): 690125
>>   +  14.49%  [k] ftrace_lookup_ip
>>   +   5.61%  [k] kprobe_trace_func
>>   +   5.17%  [k] kprobe_ftrace_handler
>>   +   4.62%  [k] get_kprobe_cached
>>   ----
>>
>> Of course this reduces the enabling time too.
>>
>> Without this fix (just enlarge hash table):
>> (2934 sec, 1 min intervals for each 2000 probes enabled)
>>
>>   ----
>>   Enabling trace events: start at 1393921862
>>   0 1393921864 a2mp_chan_alloc_skb_cb_38581
>>   ...
>>   19999 1393924928 nfs4_open_confirm_done_11785
>>   ----
>>
>> With this fix:
>> (2025 sec, 1 min intervals for each 2000 probes enabled)
> 
> That's a nice speedup.

Thanks :)

> 
> So I don't think this should be a Kconfig entry, just enable it 
> unconditionally. That will further simplify the code.

Hmm, it consumes some amount of memory (36KB/core) just for the
case of several thousand of kprobes. On enterprise servers and desktop
it's OK, no problem. But I think, some embedded systems with small
resources will not want that. So, how about enabling Kconfig by default?

Thank you,


-- 
Masami HIRAMATSU
Software Platform Research Dept. Linux Technology Research Center
Hitachi, Ltd., Yokohama Research Laboratory
E-mail: masami.hiramatsu.pt@...achi.com


--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/