linux-kernel - Re: [RFC/PATCHSET 00/37] perf tools: Speed-up perf report by using multi thread (v1)

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <CABPqkBTamwaC_rueabTkuYpwZNycvejrULSFGQ7npna2gNQwLg@mail.gmail.com>
Date:	Wed, 7 Jan 2015 10:14:56 -0500
From:	Stephane Eranian <eranian@...gle.com>
To:	Namhyung Kim <namhyung@...nel.org>
Cc:	Andi Kleen <andi@...stfloor.org>,
	Arnaldo Carvalho de Melo <acme@...nel.org>,
	Ingo Molnar <mingo@...nel.org>,
	Peter Zijlstra <a.p.zijlstra@...llo.nl>,
	Jiri Olsa <jolsa@...hat.com>,
	LKML <linux-kernel@...r.kernel.org>,
	David Ahern <dsahern@...il.com>,
	Adrian Hunter <adrian.hunter@...el.com>,
	Frederic Weisbecker <fweisbec@...il.com>
Subject: Re: [RFC/PATCHSET 00/37] perf tools: Speed-up perf report by using
 multi thread (v1)

On Wed, Jan 7, 2015 at 2:13 AM, Namhyung Kim <namhyung@...nel.org> wrote:
> Hi Stephane,
>
> On Tue, Jan 06, 2015 at 10:50:44AM -0500, Stephane Eranian wrote:
>> On Mon, Jan 5, 2015 at 1:48 PM, Andi Kleen <andi@...stfloor.org> wrote:
>> > > This also requires to handle multiple files and to find a
>> > > corresponding machine state when processing samples.  On a large
>> > > profiling session, many tasks were created and exited so pid might be
>> > > recycled (even more than once!).  To deal with it, I managed to have
>> > > thread, map_groups and comm in time sorted.  The only remaining thing
>> > > is symbol loading as it's done lazily when sample requires it.
>> >
>> > FWIW there's often a lot of unnecessary information in this
>> > (e.g. mmaps that are not used). The Quipper page
>> > claims large saving in data files by avoided redundancies.
>> >
>> > It would be probably better if perf record avoided writing redundant
>> > information better (I realize that's not easy)
>> > >
>> > > With that being done, the stage 2 can be done by multiple threads.  I
>> > > also save each sample data (per-cpu or per-thread) in separate files
>> > > during record.  On perf report time, each file will be processed by
>> > > each thread.  And symbol loading is protected by a mutex lock.
>> >
>> > I really don't like the multiple files. See above. Also it could easily
>> > cause additional seeking on spinning disks.
>> >
>> having to manage two separate files is a major change which I don't
>> particularly like. It will cause problems. I don't see why this cannot
>> be appended to the perf.data file with a index at the beginning. There
>> is already an index for sections in file mode.
>
> I just thought it's easier to handle with multiple thread.  Maybe we
> can concatenate the files after recording.
>
>
>>
>> We use the pipe mode a lot and this would not work there. So no,
>> I don't like the 2 files solution. But I like the idea of using multiple
>> threads to speed up processing.
>
> Actually it's not 2 files, it's 1 + N files. :)  Anyway, I think the
> single file + index approach requires seeking to process them, is it
> ok for pipe-mode?
>
There is no seek possible in pipe mode.

The way this is done (as I remember) is by creating pseudo-record types
and injecting them in the stream.
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/