linux-kernel - Re: [PATCH RFC V4 5/6] perf top: switch to backward overwrite mode

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Date:   Mon, 2 Oct 2017 15:03:46 -0300
From:   "acme@...nel.org" <acme@...nel.org>
To:     "Liang, Kan" <kan.liang@...el.com>
Cc:     Jiri Olsa <jolsa@...hat.com>,
        "peterz@...radead.org" <peterz@...radead.org>,
        "mingo@...hat.com" <mingo@...hat.com>,
        "linux-kernel@...r.kernel.org" <linux-kernel@...r.kernel.org>,
        "jolsa@...nel.org" <jolsa@...nel.org>,
        "namhyung@...nel.org" <namhyung@...nel.org>,
        "Hunter, Adrian" <adrian.hunter@...el.com>,
        "Odzioba, Lukasz" <lukasz.odzioba@...el.com>,
        "wangnan0@...wei.com" <wangnan0@...wei.com>,
        "hekuang@...wei.com" <hekuang@...wei.com>,
        "ast@...nel.org" <ast@...nel.org>,
        "ak@...ux.intel.com" <ak@...ux.intel.com>
Subject: Re: [PATCH RFC V4 5/6] perf top: switch to backward overwrite mode

Em Mon, Oct 02, 2017 at 05:19:41PM +0000, Liang, Kan escreveu:
> 
> 
> > On Fri, Sep 29, 2017 at 07:47:56AM -0700, kan.liang@...el.com wrote:
> > > From: Kan Liang <kan.liang@...el.com>
> > >
> > > perf_top__mmap_read has severe performance issue in Knights
> > > Landing/Mill, when monitoring in heavy load system. It costs several
> > > minutes to finish, which is unacceptable.
> > >
> > > perf top was overwrite mode. But it is changed to non overwrite mode
> > > since commit 93fc64f14472 ("perf top: Switch to non overwrite mode").
> > > For non overwrite mode, it tries to read everything in the ring buffer
> > > and does not check the messup. Once there are lots of samples
> > > delivered shortly, the processing time could be very long.
> > > Knights Landing/Mill as a manycore processor contains a large number
> > > of small cores. Because of the huge core number, it will generated
> > > lots of samples in a heavy load system. Also, since the huge sample#,
> > > the mmap writer probably bite the tail and mess up the samples.
> > >
> > > Also, to avoid the problems which is described in commit 9ecda41acb97
> > > ("perf/core: Add ::write_backward attribute to perf event"), switch to
> > > backward overwrite mode.
> > > Pausing the ring-buffer during perf_top__mmap_read to ensure the
> > > ring-buffer is stable.
> > > There would be some records lost in backward overwrite mode. Removing
> > > the lost events checking.
> > 
> > I'm getting perf top hogging the cpu completely with this change
> > 
> 
> I think I find the root cause of the cpu hogging.
> perf_mmap__read_catchup discards the md->prev from previous mmap_read.
> Current mmap_read doesn't know which data has already been processed by
> previous mmap_read. So it has to go through all the valid data in the ring buffer,
> even most of the data has been processed by previous mmap_read.
> 
> Also, it looks perf record has the similar issue.
> The previous location will be discarded as well in backward overwrite mode.
> That will be an issue when --overwrite and --switch-output are enabled.
> The new output will always include the old data in the previous output, which
> should be wrong.
> 
> I think I will rewrite the perf_mmap__read_backward and perf_mmap__read_catchup
> to fix this issue in a separate thread. Those functions should be common backward
> mmap_read functions for all tools and tests.
> 
> BTW, are you OK with patch 1-4?
> Those patches multithreading the machine__synthesize_threads, which is
> irrelevant with the overwrite mode.
> I think they can be merged separately.

I'll try and process 1-4 while you work on the read_backward case.

- Arnaldo