linux-kernel - Re: [PATCH v4 0/2] perf trace: Add more tests for BTF-augmented perf trace

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives
Hash Suite: Windows password security audit tool. GUI, reports in PDF.
[<prev] [next>] [<thread-prev] [day] [month] [year] [list]
Message-ID: <CAH0uvoit+6nhADW_d61VcaSE17Wp5DH7BtUVjQdT=kU9rMqaaA@mail.gmail.com>
Date: Fri, 31 Jan 2025 16:14:09 -0800
From: Howard Chu <howardchu95@...il.com>
To: Namhyung Kim <namhyung@...nel.org>
Cc: Arnaldo Carvalho de Melo <acme@...nel.org>, mark.rutland@....com, 
	alexander.shishkin@...ux.intel.com, jolsa@...nel.org, irogers@...gle.com, 
	adrian.hunter@...el.com, kan.liang@...ux.intel.com, 
	linux-perf-users@...r.kernel.org, linux-kernel@...r.kernel.org
Subject: Re: [PATCH v4 0/2] perf trace: Add more tests for BTF-augmented perf trace

Hello Namhyung,

Sorry for somehow missing this reply...

On Thu, Jan 23, 2025 at 3:53 PM Namhyung Kim <namhyung@...nel.org> wrote:
>
> On Tue, Jan 21, 2025 at 04:19:35PM -0800, Howard Chu wrote:
> > Hello Namhyung,
> >
> > On Sat, Jan 18, 2025 at 10:28 AM Namhyung Kim <namhyung@...nel.org> wrote:
> > >
> > > On Mon, Jan 13, 2025 at 12:57:23PM -0300, Arnaldo Carvalho de Melo wrote:
> > > > On Fri, Jan 10, 2025 at 04:52:08PM -0800, Howard Chu wrote:
> > > > > Hello Arnaldo,
> > > > >
> > > > > On Mon, Dec 23, 2024 at 11:40 AM Arnaldo Carvalho de Melo
> > > > > <acme@...nel.org> wrote:
> > > > > >
> > > > > > On Mon, Dec 23, 2024 at 04:02:43PM -0300, Arnaldo Carvalho de Melo wrote:
> > > > > > > On Mon, Dec 16, 2024 at 11:58:46AM -0800, Namhyung Kim wrote:
> > > > > > > > On Sun, Dec 15, 2024 at 11:07:09AM -0800, Howard Chu wrote:
> > > > > > > > > Changes in v4:
> > > > > > > > >  - Use if -f to check the existence of vmlinux BTF, and exit if it
> > > > > > > > >    doesn't, so trace_test_string will not overwrite $err, and keep
> > > > > > > > >    running the test.
> > > > > > > > >
> > > > > > > > > Changes in v3:
> > > > > > > > >  - Add vmlinux BTF check, and skip the tests if it doesn't exist
> > > > > > > > >
> > > > > > > > > v1, v2:
> > > > > > > > >
> > > > > > > > > The previous version of the perf trace BTF general augmentation tests
> > > > > > > > > didn't pass Shellcheck (thanks to Arnaldo Carvalho de Melo
> > > > > > > > > <acme@...nel.org> for pointing this out), this version uses bash instead
> > > > > > > > > of POSIX shell to pass Shellcheck.
> > > > > > > > >
> > > > > > > > > This patch series also adds documentation for the new option
> > > > > > > > > --force-btf, which is used in the tests.
> > > > > > > > >
> > > > > > > > > Link: https://lore.kernel.org/linux-perf-users/Zt9yiQq-n-W6I274@x1/
> > > > > > > > >
> > > > > > > > > Howard Chu (2):
> > > > > > > > >   perf trace: Add tests for BTF general augmentation
> > > > > > > > >   perf docs: Add documentation for --force-btf option
> > > > > > > >
> > > > > > > > Acked-by: Namhyung Kim <namhyung@...nel.org>
> > > > > > >
> > > > > > > Thanks, applied to perf-tools-next,
> > > > > >
> > > > > > It fails when running in parallel mode, sometimes:
> > > > > >
> > > > > > 109: perf trace BTF general tests                                    : FAILED!
> > > > > >
> > > > > > Then:
> > > > > >
> > > > > > root@...ber:~# perf stat --null -r 10 perf test "BTF general"
> > > > > > 109: perf trace BTF general tests                                    : Ok
> > > > > > 109: perf trace BTF general tests                                    : Ok
> > > > > > 109: perf trace BTF general tests                                    : Ok
> > > > > > 109: perf trace BTF general tests                                    : Ok
> > > > > > 109: perf trace BTF general tests                                    : FAILED!
> > > > > > 109: perf trace BTF general tests                                    : Ok
> > > > > > 109: perf trace BTF general tests                                    : Ok
> > > > > > 109: perf trace BTF general tests                                    : FAILED!
> > > > > > 109: perf trace BTF general tests                                    : Ok
> > > > > > 109: perf trace BTF general tests                                    : Ok
> > > > > >
> > > > > >  Performance counter stats for 'perf test BTF general' (10 runs):
> > > > > >
> > > > > >              2.148 +- 0.293 seconds time elapsed  ( +- 13.63% )
> > > > > >
> > > > > > root@...ber:~#
> > > > > >
> > > > > >
> > > > > > So its not just when running in paralell, anyway, its merged, we can go
> > > > > > on from what we got there.
> > > > >
> > > > > It never fails on my machine, I think the reason is my machine is not
> > > > > fully-loaded. Can you please run
> > > > > ```
> > > > > perf stat --null -r 10 perf test -vv "BTF general"
> > > > > ```
> > > > > And provide the output?
> > > >
> > > > root@...ber:~# grep -m1 'model name' /proc/cpuinfo
> > > > model name    : Intel(R) Core(TM) i7-14700K
> > > > root@...ber:~#
> > > >
> > > > root@...ber:~# perf stat --null -r 10 perf test -vv "BTF general"
> > > > 106: perf trace BTF general tests:
> > > > --- start ---
> > > > test child forked, pid 47824
> > > > Checking if vmlinux BTF exists
> > > > Testing perf trace's string augmentation
> > > > String augmentation test failed
> > > > ---- end(-1) ----
> > > > 106: perf trace BTF general tests                                    : FAILED!
> > > > 106: perf trace BTF general tests:
> > > > --- start ---
> > > > test child forked, pid 47842
> > > > Checking if vmlinux BTF exists
> > > > Testing perf trace's string augmentation
> > > > String augmentation test failed
> > > > ---- end(-1) ----
> > > > 106: perf trace BTF general tests                                    : FAILED!
> > > > ...
> > >
> > > I also see the failure on not fully-loaded machines (both Intel and
> > > AMD).
> > >
> > >   $ sudo ./perf test -v BTF
> > >   --- start ---
> > >   test child forked, pid 525025
> > >   Checking if vmlinux BTF exists
> > >   Testing perf trace's string augmentation
> > >   String augmentation test failed
> > >   ---- end(-1) ----
> > >   106: perf trace BTF general tests                                    : FAILED!
> > >
> > > Thanks,
> > > Namhyung
> > >
> >
> > Just wondering, did you and Arnaldo apply this patch:
> > https://lore.kernel.org/linux-perf-users/20241213023047.541218-1-howardchu95@gmail.com/
>
> I don't think we applied it, will add to perf-tools.
>
> >
> > This seems to solve the problem for me...
>
> But I'm not sure if it fixes the problem.  IIUC the patch fixes the
> loading problem.

Yes. However, if the machine is unable to load the BPF program, the
error message will be:

perf $ sudo ./perf test general -v
--- start ---
test child forked, pid 173990
Checking if vmlinux BTF exists
Testing perf trace's string augmentation
String augmentation test failed
---- end(-1) ----
106: perf trace BTF general tests                                    : FAILED!

Which is indistinguishable from an actual string augmentation error
that has a string mismatch...

> And for some reason, I don't see the problem any more.
> But I remember this sometimes happened in the past.

Do you mean that perf trace breaks occasionally? This sounds urgent.
Could you please provide me with your kernel version and Clang
version? Thank you very much! :)

Thanks,
Howard
>
> Thanks,
> Namhyung
>