| lists.openwall.net | lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC | |
|
Open Source and information security mailing list archives
| ||
|
Message-ID: <Y9PC2+Yqh3W8wyNU@kernel.org>
Date: Fri, 27 Jan 2023 09:26:03 -0300
From: Arnaldo Carvalho de Melo <acme@...nel.org>
To: Michael Petlan <mpetlan@...hat.com>
Cc: linux-perf-users@...r.kernel.org, acme@...hat.com,
qzhao@...hat.com, cjense@...gle.com,
Ian Rogers <irogers@...gle.com>,
Linux Kernel Mailing List <linux-kernel@...r.kernel.org>,
Namhyung Kim <namhyung@...nel.org>,
Jiri Olsa <jolsa@...nel.org>,
Adrian Hunter <adrian.hunter@...el.com>
Subject: Re: [PATCH 2/2] perf test: Fix JSON format linter test checks
Em Fri, Jan 20, 2023 at 02:40:39PM +0100, Michael Petlan escreveu:
> The test fails on CPUs with topdown metrics, where it is common to print
> two metrics per line. Since these are included in default event set for
> `perf stat -- something`, it causes the "no args" subtest to fail due to
> unexpected member count. We need to accept 7 or 9 members in that case.
>
> Coming to that, counting commas in the JSON line and consider it a number
> of elements is incorrect and misleading. There should be +1 element than
> the count of commas, while also, commas can potentially appear in the
> event names. Count " : " substrings rather, since these better fit to
> what should be actually counted.
Before I apply this first patch I can run, as root, the 'perf test' JSON
lint 100 times without problems:
[root@...co ~]# for a in $(seq 100) ; do echo -n $a ; perf test 98 ; done
1 98: perf stat JSON output linter : Ok
2 98: perf stat JSON output linter : Ok
3 98: perf stat JSON output linter : Ok
4 98: perf stat JSON output linter : Ok
<SNIP>
96 98: perf stat JSON output linter : Ok
97 98: perf stat JSON output linter : Ok
98 98: perf stat JSON output linter : Ok
99 98: perf stat JSON output linter : Ok
100 98: perf stat JSON output linter : Ok
[root@...co ~]#
After applying it it fails seemingly randomly, I'll remove both patches
from my tmp.perf/core branch and investigate.
[acme@...co perf]$ git log --oneline -1
fdafd42aac3629db (HEAD) perf stat: Fix JSON metric printout for multiple metrics per line
[acme@...co perf]$ perf -v
perf version 6.2.rc5.gfdafd42aac36
[acme@...co perf]$
[root@...co ~]# perf -v
perf version 6.2.rc5.gfdafd42aac36
[root@...co ~]# for a in $(seq 100) ; do echo -n $a ; perf test 98 ; done
[root@...co ~]# perf -v
perf version 6.2.rc5.gfdafd42aac36
[root@...co ~]# for a in $(seq 100) ; do echo -n $a ; perf test 98 ; done
1 98: perf stat JSON output linter : Ok
2 98: perf stat JSON output linter : Ok
3 98: perf stat JSON output linter : FAILED!
4 98: perf stat JSON output linter : Ok
5 98: perf stat JSON output linter : Ok
6 98: perf stat JSON output linter : Ok
7 98: perf stat JSON output linter : FAILED!
8 98: perf stat JSON output linter : Ok
9 98: perf stat JSON output linter : Ok
10 98: perf stat JSON output linter : Ok
11 98: perf stat JSON output linter : Ok
12 98: perf stat JSON output linter : Ok
13 98: perf stat JSON output linter : Ok
14 98: perf stat JSON output linter : Ok
15 98: perf stat JSON output linter : Ok
16 98: perf stat JSON output linter : FAILED!
17 98: perf stat JSON output linter : Ok
18 98: perf stat JSON output linter : Ok
19 98: perf stat JSON output linter : Ok
20 98: perf stat JSON output linter : FAILED!
21 98: perf stat JSON output linter : Ok
22 98: perf stat JSON output linter : Ok
23 98: perf stat JSON output linter : FAILED!
24 98: perf stat JSON output linter : Ok
25 98: perf stat JSON output linter : Ok
26 98: perf stat JSON output linter : Ok
27 98: perf stat JSON output linter : Ok
28 98: perf stat JSON output linter : FAILED!
29 98: perf stat JSON output linter : Ok
30 98: perf stat JSON output linter : Ok
31 98: perf stat JSON output linter : FAILED!
32 98: perf stat JSON output linter : FAILED!
33 98: perf stat JSON output linter : FAILED!
34 98: perf stat JSON output linter : Ok
35 98: perf stat JSON output linter : Ok
36 98: perf stat JSON output linter : Ok
37 98: perf stat JSON output linter : Ok
38 98: perf stat JSON output linter : FAILED!
39 98: perf stat JSON output linter : FAILED!
40 98: perf stat JSON output linter : FAILED!
41 98: perf stat JSON output linter : Ok
42 98: perf stat JSON output linter : Ok
43 98: perf stat JSON output linter : Ok
44 98: perf stat JSON output linter : Ok
45 98: perf stat JSON output linter : Ok
46 98: perf stat JSON output linter : FAILED!
47 98: perf stat JSON output linter : Ok
48 98: perf stat JSON output linter : FAILED!
49 98: perf stat JSON output linter : FAILED!
50 98: perf stat JSON output linter : FAILED!
51 98: perf stat JSON output linter : FAILED!
52 98: perf stat JSON output linter : Ok
53 98: perf stat JSON output linter : FAILED!
54 98: perf stat JSON output linter : Ok
55 98: perf stat JSON output linter : Ok
56 98: perf stat JSON output linter : FAILED!
57 98: perf stat JSON output linter : Ok
58 98: perf stat JSON output linter : Ok
59 98: perf stat JSON output linter : FAILED!
60 98: perf stat JSON output linter : Ok
61 98: perf stat JSON output linter : FAILED!
62 98: perf stat JSON output linter : Ok
63 98: perf stat JSON output linter : Ok
64 98: perf stat JSON output linter : Ok
65 98: perf stat JSON output linter : Ok
66 98: perf stat JSON output linter : Ok
67 98: perf stat JSON output linter : Ok
68 98: perf stat JSON output linter : Ok
69 98: perf stat JSON output linter : Ok
70 98: perf stat JSON output linter : Ok
71 98: perf stat JSON output linter : FAILED!
72 98: perf stat JSON output linter : FAILED!
73 98: perf stat JSON output linter : Ok
74 98: perf stat JSON output linter : FAILED!
75 98: perf stat JSON output linter : Ok
76 98: perf stat JSON output linter : FAILED!
77 98: perf stat JSON output linter : Ok
78 98: perf stat JSON output linter : FAILED!
79 98: perf stat JSON output linter : FAILED!
80 98: perf stat JSON output linter : FAILED!
81 98: perf stat JSON output linter : Ok
82 98: perf stat JSON output linter : FAILED!
83 98: perf stat JSON output linter : FAILED!
84 98: perf stat JSON output linter : Ok
85 98: perf stat JSON output linter : Ok
86 98: perf stat JSON output linter : Ok
87 98: perf stat JSON output linter : FAILED!
88 98: perf stat JSON output linter : FAILED!
89 98: perf stat JSON output linter : Ok
90 98: perf stat JSON output linter : FAILED!
91 98: perf stat JSON output linter : FAILED!
92 98: perf stat JSON output linter : Ok
93 98: perf stat JSON output linter : FAILED!
94 98: perf stat JSON output linter : FAILED!
95 98: perf stat JSON output linter : FAILED!
96 98: perf stat JSON output linter : FAILED!
97 98: perf stat JSON output linter : FAILED!
98 98: perf stat JSON output linter : Ok
99 98: perf stat JSON output linter : Ok
100 98: perf stat JSON output linter : FAILED!
[root@...co ~]#
> Fixes: 0c343af2a2f8 ("perf test: JSON format checking")
>
> Signed-off-by: Michael Petlan <mpetlan@...hat.com>
> ---
> .../tests/shell/lib/perf_json_output_lint.py | 16 ++++++++--------
> 1 file changed, 8 insertions(+), 8 deletions(-)
>
> diff --git a/tools/perf/tests/shell/lib/perf_json_output_lint.py b/tools/perf/tests/shell/lib/perf_json_output_lint.py
> index d90f8d102eb9..4f1bbb3f07ec 100644
> --- a/tools/perf/tests/shell/lib/perf_json_output_lint.py
> +++ b/tools/perf/tests/shell/lib/perf_json_output_lint.py
> @@ -40,17 +40,17 @@ def is_counter_value(num):
> return isfloat(num) or num == '<not counted>' or num == '<not supported>'
>
> def check_json_output(expected_items):
> - if expected_items != -1:
> + if expected_items:
> for line in Lines:
> if 'failed' not in line:
> count = 0
> - count = line.count(',')
> - if count != expected_items and count >= 1 and count <= 3 and 'metric-value' in line:
> + count = line.count(' : ')
> + if count not in expected_items and count >= 2 and count <= 4 and 'metric-value' in line:
> # Events that generate >1 metric may have isolated metric
> # values and possibly other prefixes like interval, core and
> # aggregate-number.
> continue
> - if count != expected_items:
> + if count not in expected_items:
> raise RuntimeError(f'wrong number of fields. counted {count} expected {expected_items}'
> f' in \'{line}\'')
> checks = {
> @@ -82,14 +82,14 @@ def check_json_output(expected_items):
>
> try:
> if args.no_args or args.system_wide or args.event:
> - expected_items = 6
> + expected_items = [7, 9]
> elif args.interval or args.per_thread or args.system_wide_no_aggr:
> - expected_items = 7
> + expected_items = [8, 10]
> elif args.per_core or args.per_socket or args.per_node or args.per_die:
> - expected_items = 8
> + expected_items = [9, 11]
> else:
> # If no option is specified, don't check the number of items.
> - expected_items = -1
> + expected_items = []
> check_json_output(expected_items)
> except:
> print('Test failed for input:\n' + '\n'.join(Lines))
> --
> 2.18.4
>
--
- Arnaldo
Powered by blists - more mailing lists