linux-kernel - Re: [PATCH RFC 0/3] perf: add logic to collect off-cpu samples

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <CAD2QZ9ZLHQ62d5CDpf14DkrWFs_FggRMdqzGFz3qsmF_vmS00Q@mail.gmail.com>
Date: Sat, 13 Jul 2024 13:12:52 +0530
From: Ajay Kaher <ajay.kaher@...adcom.com>
To: Ian Rogers <irogers@...gle.com>
Cc: chu howard <howardchu95@...il.com>, peterz@...radead.org, mingo@...hat.com, 
	acme@...nel.org, namhyung@...nel.org, mark.rutland@....com, 
	rostedt@...dmis.org, alexander.shishkin@...ux.intel.com, jolsa@...nel.org, 
	adrian.hunter@...el.com, kan.liang@...ux.intel.com, yangjihong1@...wei.com, 
	zegao2021@...il.com, leo.yan@...ux.dev, asmadeus@...ewreck.org, 
	siyanteng@...ngson.cn, sunhaiyong@...ngson.cn, 
	linux-perf-users@...r.kernel.org, linux-kernel@...r.kernel.org, 
	alexey.makhalov@...adcom.com, vasavi.sirnapalli@...adcom.com, 
	Vamsi Krishna Brahmajosyula <vamsi-krishna.brahmajosyula@...adcom.com>, nadav.amit@...il.com
Subject: Re: [PATCH RFC 0/3] perf: add logic to collect off-cpu samples

On Fri, Jul 12, 2024 at 3:28 AM Ian Rogers <irogers@...gle.com> wrote:
>
> On Thu, Jul 11, 2024 at 5:16 AM Ajay Kaher <ajay.kaher@...adcom.com> wrote:
> >
> > Add --off-cpu-kernel option to capture off-cpu sample alongwith on-cpu
> > samples.
> >
> > off-cpu samples represent time spent by task when it was on wait queue
> > (schedule out to waiting for events, blocked on I/O, locks, timers,
> > paging/swapping, etc)
> >
> > Refer following links for more details:
> > https://lpc.events/event/17/contributions/1556/
> > https://www.youtube.com/watch?v=sF2faKGRnjs
>
> Hi Ajay,
>
> I wonder if Howard's improvements (not landed) for `perf record
> --off-cpu` would solve this problem for you?
> https://lore.kernel.org/lkml/20240424024805.144759-1-howardchu95@gmail.com/
> Or is that approach problematic due to the use of BPF?
>

Thanks Ian for your response and sharing Howard's improvements.

Yes, perf --off-cpu is based upon BPF and having following restrictions:

- target binary should be compiled with frame pointer, same mentioned
  in tools/perf/Documentation/perf-record.txt:
  Note that BPF can collect stack traces using frame pointer ("fp") only,
  as of now. So the applications built without the frame pointer might see
  bogus addresses.

- perf should be complied with BUILD_BPF_SKEL=1:
  Warning: option `off-cpu' is being ignored because no BUILD_BPF_SKEL=1

- off-cpu, on-cpu samples are not on the same result page.
  (I guess Howard has improve this, not tried his patches)

I have tried to collect the off-cpu sample same as on-cpu sample with the help
of  kernel/events/core.c. We will get one off-cpu sample from the target task
sched-out to sched-in. Or we can say off-cpu samples are not dependent on
frequency provided by the user to perf record.

I am also worried about having so many samples if sched-in/out
frequency is high.
Thinking to merge samples if attributes are the same (i.e. stacktrace)
and add the
off-cpu period to previous samples with the same attribute.

-Ajay

> Thanks,
> Ian
>
> > Ajay Kaher (3):
> >   perf/core: add logic to collect off-cpu sample
> >   perf/record: add options --off-cpu-kernel
> >   perf/report: add off-cpu samples
> >
> >  include/linux/perf_event.h            | 16 ++++++++++++++
> >  include/uapi/linux/perf_event.h       |  3 ++-
> >  kernel/events/core.c                  | 27 ++++++++++++++++++-----
> >  tools/include/uapi/linux/perf_event.h |  3 ++-
> >  tools/perf/builtin-record.c           |  2 ++
> >  tools/perf/util/events_stats.h        |  2 ++
> >  tools/perf/util/evsel.c               |  4 ++++
> >  tools/perf/util/hist.c                | 31 ++++++++++++++++++++++++---
> >  tools/perf/util/hist.h                |  1 +
> >  tools/perf/util/record.h              |  1 +
> >  tools/perf/util/sample.h              |  1 +
> >  11 files changed, 81 insertions(+), 10 deletions(-)
> >
> > --
> > 2.39.0
> >