linux-kernel - Re: [PATCH v5 0/8] perf report: Add latency and parallelism profiling

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [day] [month] [year] [list]

Message-ID: <Z6ZRVeoFxT0NV9yb@tassilo>
Date: Fri, 7 Feb 2025 10:30:45 -0800
From: Andi Kleen <ak@...ux.intel.com>
To: Dmitry Vyukov <dvyukov@...gle.com>
Cc: namhyung@...nel.org, irogers@...gle.com,
	linux-perf-users@...r.kernel.org, linux-kernel@...r.kernel.org,
	Arnaldo Carvalho de Melo <acme@...nel.org>
Subject: Re: [PATCH v5 0/8] perf report: Add latency and parallelism profiling

> > I assume it just works, but might be worth checking.
> 
> Yes, it seems to just work as one would assume. Things just combine as intended.

Great.

> 
> > It was intended to address some of these issues too.
> 
> What issue? Latency profiling? I wonder what approach you had in mind?

The problem with gaps in parallelism is usually how things change
over time. If you have e.g. idle periods they tend to look different
in the profile. with the full aggregation you don't see it, but with
a time series it tends to stand out.

But yes that approach usually only works for large gaps. I guess
yours is better for more fine grained issues.

And I guess it might not be the most intutive for less experienced
users.

This is BTW actually a case for using a perf data GUI like hotspot or
vtune which can visualize this better and you can zoom arbitrarily.
Standard perf has only timechart for it, but it's a bit clunky to use
and only shows the reschedules.

> Also (1) user still needs to understand the default profile is wrong,
> (2) be proficient with perf features, (3) manually aggregate lots of
> data (time slicing increases amount of data in the profile X times),
> (4) deal with inaccuracy caused by edge effects (e.g. slice is 1s, but
> program phase changed mid-second).

If you're lucky and the problem is not long tail you can use a high
percentage cut off (--percent-limit) to eliminate most of the data.

Then you just have "topN functions over time" which tends to be quite
readable. One drawback of that approach is that it doesn't show
the "other", but perhaps we'll fix that one day.

But yes that perf has too many options and is not intuitive and most
people miss most of the features is an inherent problem. I don't have
a good solution for that unfortunately, other than perhaps better
documentation.

> 
> But it does open some interesting capabilities in combination with a
> latency profile, e.g. the following shows how parallelism was changing
> over time.
> 
> for perf make profile:

Very nice! Looks useful.

Perhaps add that variant to tips.txt too.

> 
> perf report -F time,latency,parallelism --time-quantum=1s
> 
> # Time           Latency  Parallelism
> # ............  ........  ...........
> #
>   1795957.0000     1.42%            1
>   1795957.0000     0.07%            2
>   1795957.0000     0.01%            3
>   1795957.0000     0.00%            4
> 
>   1795958.0000     4.82%            1
>   1795958.0000     0.11%            2
>   1795958.0000     0.00%            3
> ...
>   1795964.0000     1.76%            2
>   1795964.0000     0.58%            4
>   1795964.0000     0.45%            1
>   1795964.0000     0.23%           10
>   1795964.0000     0.21%            6
> 
> /\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
> 
> Here it finally started running on more than 1 CPU.

-Andi