linux-kernel - Re: [PATCH v1 0/8] Run tests in parallel showing number of tests running

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <CAP-5=fULBYVqrgcQ1hHgou29HATOtXfZxR84r-Q2WTccT7SOBg@mail.gmail.com>
Date: Fri, 11 Oct 2024 09:26:19 -0700
From: Ian Rogers <irogers@...gle.com>
To: James Clark <james.clark@...aro.org>
Cc: Peter Zijlstra <peterz@...radead.org>, Ingo Molnar <mingo@...hat.com>, 
	Arnaldo Carvalho de Melo <acme@...nel.org>, Namhyung Kim <namhyung@...nel.org>, 
	Mark Rutland <mark.rutland@....com>, 
	Alexander Shishkin <alexander.shishkin@...ux.intel.com>, Jiri Olsa <jolsa@...nel.org>, 
	Adrian Hunter <adrian.hunter@...el.com>, Kan Liang <kan.liang@...ux.intel.com>, 
	Howard Chu <howardchu95@...il.com>, Athira Jajeev <atrajeev@...ux.vnet.ibm.com>, 
	Michael Petlan <mpetlan@...hat.com>, Veronika Molnarova <vmolnaro@...hat.com>, 
	Dapeng Mi <dapeng1.mi@...ux.intel.com>, Thomas Richter <tmricht@...ux.ibm.com>, 
	Ilya Leoshkevich <iii@...ux.ibm.com>, Colin Ian King <colin.i.king@...il.com>, 
	Weilin Wang <weilin.wang@...el.com>, Andi Kleen <ak@...ux.intel.com>, linux-kernel@...r.kernel.org, 
	linux-perf-users@...r.kernel.org
Subject: Re: [PATCH v1 0/8] Run tests in parallel showing number of tests running

On Fri, Oct 11, 2024 at 3:03 AM James Clark <james.clark@...aro.org> wrote:
>
>
>
> On 11/10/2024 8:35 am, Ian Rogers wrote:
> > Avoid waitpid so that stdout/stderr aren't destroyed prior to wanting
> > to read them for display. When running on a color terminal, display
> > the number of running tests (1 if sequential). To avoid previous
> > flicker, only delete and refresh the display line when it changes. An
> > earlier version of this code is here:
> > https://lore.kernel.org/lkml/20240701044236.475098-1-irogers@google.com/
> >
> > Add a signal handler for perf tests so that unexpected signals are
> > displayed and test clean up is possible.
> >
> > In perf test add an "exclusive" flag that causes a test to be run with
> > no other test. Set this flag manually for C tests and via a
> > "(exclusive)" in the test description for shell tests. Add the flag to
> > shell tests that may fail when run with other tests.
> >
> > Change the perf test loop to run in two passes. For parallel
> > execution, the first pass runs all tests that can be run in parallel
> > then the 2nd runs remaining tests sequentially. This causes the
> > "exclusive" tests to be run last and with test numbers moderately out
> > of alignment.
> >
> > Change the default to be to run tests in parallel. Running tests in
> > parallel brings the execution time down to less than half.
> >
> > Ian Rogers (8):
> >    tools subcmd: Add non-waitpid check_if_command_finished()
> >    perf test: Display number of remaining tests
> >    perf test: Reduce scope of parallel variable
> >    perf test: Avoid list test blocking on writing to stdout
> >    perf test: Tag parallel failing shell tests with "(exclusive)"
> >    perf test: Add a signal handler around running a test
> >    perf test: Run parallel tests in two passes
> >    perf test: Make parallel testing the default
> >
> >   tools/lib/subcmd/run-command.c                |  33 +++
> >   tools/perf/tests/builtin-test.c               | 267 ++++++++++++------
> >   tools/perf/tests/shell/list.sh                |   5 +-
> >   .../tests/shell/perftool-testsuite_report.sh  |   2 +-
> >   tools/perf/tests/shell/record.sh              |   2 +-
> >   tools/perf/tests/shell/record_lbr.sh          |   2 +-
> >   tools/perf/tests/shell/record_offcpu.sh       |   2 +-
> >   tools/perf/tests/shell/stat_all_pmu.sh        |   2 +-
> >   tools/perf/tests/shell/test_intel_pt.sh       |   2 +-
> >   .../perf/tests/shell/test_stat_intel_tpebs.sh |   2 +-
> >   tools/perf/tests/tests-scripts.c              |   5 +
> >   tools/perf/tests/tests.h                      |   1 +
> >   tools/perf/util/color.h                       |   1 +
> >   13 files changed, 226 insertions(+), 100 deletions(-)
> >
>
> Not really a big deal but remaining doesn't work when a subset of tests
> are run:
>
>   $ perf test 111 110
>   110: Check Arm64 callgraphs are complete in fp mode      : Ok
>   111: Check Arm CoreSight trace data recording and synthesized samples:
>     Running (150 remaining)

Thanks, I'd been so focussed on other issues that I'd missed testing
like this. v2 will fix the "remaining" but I've renamed it "active" as
the count doesn't reflect the remaining tests as exclusive tests will
be missing and they will show 1 test remaining which isn't accurate.

> Other than that:
>
> Tested-by: James Clark <james.clark@...aro.org>

Thanks,
Ian