Message-ID: <aBojTzsa5mSAGziP@x1>
Date: Tue, 6 May 2025 11:57:19 -0300
From: Arnaldo Carvalho de Melo <acme@...nel.org>
To: Dmitry Vyukov <dvyukov@...gle.com>
Cc: Namhyung Kim <namhyung@...nel.org>, Ingo Molnar <mingo@...nel.org>,
	Ian Rogers <irogers@...gle.com>,
	Kan Liang <kan.liang@...ux.intel.com>, Jiri Olsa <jolsa@...nel.org>,
	Adrian Hunter <adrian.hunter@...el.com>,
	Peter Zijlstra <peterz@...radead.org>,
	LKML <linux-kernel@...r.kernel.org>,
	linux-perf-users@...r.kernel.org, Andi Kleen <ak@...ux.intel.com>
Subject: Re: [RFC/PATCH] perf report: Support latency profiling in
 system-wide mode

On Mon, May 05, 2025 at 10:03:05AM +0200, Dmitry Vyukov wrote:
> On Sun, 4 May 2025 at 21:52, Namhyung Kim <namhyung@...nel.org> wrote:
> > On Sun, May 04, 2025 at 10:22:26AM +0200, Ingo Molnar wrote:
> > > > Here's an example:

> > > >   # perf record -a -- perf bench sched messaging

> > > > This basically forks each sender and receiver tasks for the run.

> > > >   # perf report --latency -s comm --stdio
> > > >   ...
> > > >   #
> > > >   #  Latency  Overhead  Command
> > > >   # ........  ........  ...............
> > > >   #
> > > >       98.14%    95.97%  sched-messaging
> > > >        0.78%     0.93%  gnome-shell
> > > >        0.36%     0.34%  ptyxis
> > > >        0.23%     0.23%  kworker/u112:0-
> > > >        0.23%     0.44%  perf
> > > >        0.08%     0.10%  KMS thread
> > > >        0.05%     0.05%  rcu_preempt
> > > >        0.05%     0.05%  kworker/u113:2-
> > > >        ...

> > > Just a generic user-interface comment: I had to look up what 'latency'
> > > means in this context, and went about 3 hops deep into various pieces
> > > of description until I found Documentation/cpu-and-latency-overheads.txt,
> > > where after a bit of head-scratching I realized that 'latency' is a
> > > weird alias for 'wall-clock time'...

> > > This is *highly* confusing terminology IMHO.

> > Sorry for the confusion.  I know I'm terrible at naming things. :)

> > Actually Dmitry used the term 'wall-clock' profiling at first when he
> > implemented this feature but I thought it was not clear how it meant
> > for non-cycle events.  As 'overhead' is also a generic term, we ended
> > up with 'latency'.
 
> Exactly :)

So, the 'cycles' event is the most commonly used, its the default if one
does 'perf record' and don't ask for a specific event.

When we notice that 'cycles' and 'instructions' are counted, we provide
an IPC, as that is a well known metric for that combo:

  root@...ber:~# perf stat -e cycles,instructions sleep 1
  
   Performance counter stats for 'sleep 1':
  
           1,149,493      cycles                                                                
           1,197,279      instructions                     #    1.04  insn per cycle            
  
         1.000645408 seconds time elapsed
  
         0.000599000 seconds user
         0.000000000 seconds sys
  
  
  root@...ber:~#

So maybe when we notice that cycles was used 'perf report/top' should
use the term 'wall-clock' for the column name?

So instead of having a --latency we could have a hint that would tell
the user that if this knob was set:

  $ perf config report.wall-clock=true

Then if 'cycles' is present we would have:

  # perf report -s comm --stdio
  ...
  #
  # Wall-clock  Overhead  Command
  # ..........  ........  ...............
  #
        98.14%    95.97%  sched-messaging
         0.78%     0.93%  gnome-shell
         0.36%     0.34%  ptyxis
         0.23%     0.23%  kworker/u112:0-
         0.23%     0.44%  perf
         0.08%     0.10%  KMS thread
         0.05%     0.05%  rcu_preempt
         0.05%     0.05%  kworker/u113:2-
         ...

And that would become the "default", i.e. show the "Wall-clock" column
when appropriate, which is something I think Dmitry partially (he used
it for all events, IIRC, this would be just if 'cycles' is present)
tried when submitting the first version of his patchset?

What term to use with other events remains as an exercise, but to a lot
of people wanting just the 'cycles' event, like was used in the above
patch log message:

  # perf record -a -- perf bench sched messaging

That would be enough, no?

- Arnaldo
 
> I've also talked with a bunch of people about this, and everybody
> proposes their own term and is confused by all other proposals.
 
> The problem is that we did not just lack this fundamental profiling
> capability in all profilers out there, but we, as a community, also
> still don't know how to even talk about these things...
 
> There is no terminology that would be clear for everybody. E.g. when
> some people hear wall-clock, they imply that it samples every thread
> (runnable and non-runnnable) every time unit (but that's a vastly
> different profile from this one).
 
> > > 'Latency' is a highly overloaded concept that almost never corresponds
> > > to 'wall clock time'. It usually means a relative delay value, which is
> > > why I initially thought this somehow means instruction-latency or
> > > memory-latency profiling ...

> > > Ie. 'latency' in its naive meaning, is on the exact opposite side of
> > > the terminology spectrum of where it should be: it suggests relative
> > > time, while in reality it's connected to wall-clock/absolute time ...

> > > *Please* use something else. Wall-clock is fine, as
> > > cpu-and-latency-overheads.txt uses initially, but so would be other
> > > combinations:

> > >    #1: 'CPU time' vs. 'real time'

> > >         This is short, although a disadvantage is the possible
> > >         'real-time kernel' source of confusion here.

> > >    #2: 'CPU time' vs. 'wall-clock time'

> > >         This is longer but OK and unambiguous.

> > >    #3: 'relative time' vs. 'absolute time'

> > >         This is short and straightforward, and might be my favorite
> > >         personally, because relative/absolute is such an unambiguous
> > >         and well-known terminology and often paired in a similar
> > >         fashion.

> > >    #4: 'CPU time' vs. 'absolute time'

> > >         This is a combination of #1 and #3 that keeps the 'CPU time'
> > >         terminology for relative time. The CPU/absolute pairing is not
> > >         that intuitive though.

> > >    #5: 'CPU time' vs. 'latency'

> > >         This is really, really bad and unintuitive. Sorry to be so
> > >         harsh and negative about this choice, but this is such a nice
> > >         feature, which suffers from confusing naming. :-)

> > Thanks for your seggestions.  My main concern is that it's not just
> > about cpu-time and wallclock-time.  perf tools can measure any events
> > that have different meanings.  So I think we need generic terms to cover
> > them.
 
> I did not see a conflict there. It's possible to sample, say, cache
> misses per unit of CPU time or per unit of wall-clock time.

Powered by blists - more mailing lists