linux-kernel - Re: [PATCH v6 0/4] perf: add support for profiling jitted code

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Date:	Tue, 31 Mar 2015 14:31:45 -0700
From:	Brendan Gregg <brendan.d.gregg@...il.com>
To:	Stephane Eranian <eranian@...gle.com>
Cc:	LKML <linux-kernel@...r.kernel.org>, acme@...hat.com,
	Peter Zijlstra <peterz@...radead.org>,
	Ingo Molnar <mingo@...e.hu>, Andi Kleen <ak@...ux.intel.com>,
	Jiri Olsa <jolsa@...hat.com>,
	Namhyung Kim <namhyung@...nel.org>,
	Rose Belcher <cel@...ibm.com>, sukadev@...ux.vnet.ibm.com,
	sonnyrao@...omium.org, johnmccutchan@...gle.com,
	David Ahern <dsahern@...il.com>, adrian.hunter@...el.com,
	Pawel Moll <pawel.moll@....com>
Subject: Re: [PATCH v6 0/4] perf: add support for profiling jitted code

On Tue, Mar 31, 2015 at 12:33 AM, Brendan Gregg
<brendan.d.gregg@...il.com> wrote:
> G'Day Stephane,
>
> On Mon, Mar 30, 2015 at 3:19 PM, Stephane Eranian <eranian@...gle.com> wrote:
> [...]
>> The current support only works when the runtime is monitored from
>> start to finish: perf record java --agentpath:libpfmjvmti.so my_class.
>>
>> Once the run is completed, the jitdump file needs to be injected into
>> the perf.data file. This is accomplished by using the perf inject command.
>> This will also generate an ELF image for each jitted function. The
>> inject MMAP records will point to those ELF images. The reasoning
>> behind using ELF images is that it makes processing for perf report
>> and annotate automatic and transparent. It also makes it easier to
>> package and analyze on a remote machine.
> [...]
>
> This is really impressive work. Do we have an idea of the overhead for
> running the java agent?
>
> Today, I'm using perf-map-agent, loaded dynamically, to dump a
> /tmp/perf*.map file as needed. My company has tens of thousands of
> Linux instances running Java, but very few need profiling, and we
> don't know which beforehand. So a snapshot-on-demand approach is
> ideal. An always-on approach, well, we'd have to know the overhead (I
> can build the agent and test...).

I built the agent and tested with an application framework
micro-benchmark, and saw the performance overhead drop after start
from about 13% initially (measured as a reduction in maximum req/sec
given fixed CPU capacity), to 1.1% after a minute, and then 0.13%
(which is really just noise) after several minutes of high load.

So the overhead is basically zero after (minutes of) warmup, at least
for my test. My jit.dump file reached 8 Mbytes, and was growing by a
tiny amount every 30 seconds or so (hence the near-zero overhead). I'm
much less concerned about overheads now.

I'll test with a production workload if I can... But I'm still curious
about why we're even doing this, instead of the previous method of
taking symbol snapshots. Is there a backstory? If it involves a case
of high symbol churn, then this should also mean non-zero overhead to
constantly log.

Brendan
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/