lists.openwall.net   lists  /  announce  owl-users  owl-dev  john-users  john-dev  passwdqc-users  yescrypt  popa3d-users  /  oss-security  kernel-hardening  musl  sabotage  tlsify  passwords  /  crypt-dev  xvendor  /  Bugtraq  Full-Disclosure  linux-kernel  linux-netdev  linux-ext4  linux-hardening  linux-cve-announce  PHC 
Open Source and information security mailing list archives
 
Hash Suite: Windows password security audit tool. GUI, reports in PDF.
[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]
Message-ID: <05f7bf89-04a5-4b65-bf59-c19456aeb1f0@sirena.org.uk>
Date: Mon, 4 Mar 2024 22:20:03 +0000
From: Mark Brown <broonie@...nel.org>
To: Jakub Kicinski <kuba@...nel.org>
Cc: davem@...emloft.net, netdev@...r.kernel.org, edumazet@...gle.com,
	pabeni@...hat.com, shuah@...nel.org,
	linux-kselftest@...r.kernel.org, mic@...ikod.net,
	linux-security-module@...r.kernel.org, keescook@...omium.org,
	jakub@...udflare.com
Subject: Re: [PATCH v4 00/12] selftests: kselftest_harness: support using
 xfail

On Wed, Feb 28, 2024 at 04:59:07PM -0800, Jakub Kicinski wrote:

> When running selftests for our subsystem in our CI we'd like all
> tests to pass. Currently some tests use SKIP for cases they
> expect to fail, because the kselftest_harness limits the return
> codes to pass/fail/skip. XFAIL which would be a great match
> here cannot be used.
> 
> Remove the no_print handling and use vfork() to run the test in
> a different process than the setup. This way we don't need to
> pass "failing step" via the exit code. Further clean up the exit
> codes so that we can use all KSFT_* values. Rewrite the result
> printing to make handling XFAIL/XPASS easier. Support tests
> declaring combinations of fixture + variant they expect to fail.

This series landed in -next today and has caused breakage on all
platforms in the ALSA pcmtest-driver test.  When run on systems that
don't have the driver it needs loaded the test skip but since this
series was merged skipped tests are logged but then reported back as
failures:

# selftests: alsa: test-pcmtest-driver
# TAP version 13
# 1..5
# # Starting 5 tests from 1 test cases.
# #  RUN           pcmtest.playback ...
# #      SKIP      Can't read patterns. Probably, module isn't loaded
# # playback: Test failed
# #          FAIL  pcmtest.playback
# not ok 1 pcmtest.playback #  Can't read patterns. Probably, module isn't loaded
# #  RUN           pcmtest.capture ...
# #      SKIP      Can't read patterns. Probably, module isn't loaded
# # capture: Test failed
# #          FAIL  pcmtest.capture
# not ok 2 pcmtest.capture #  Can't read patterns. Probably, module isn't loaded
# #  RUN           pcmtest.ni_capture ...
# #      SKIP      Can't read patterns. Probably, module isn't loaded
# # ni_capture: Test failed
# #          FAIL  pcmtest.ni_capture
# not ok 3 pcmtest.ni_capture #  Can't read patterns. Probably, module isn't loaded
# #  RUN           pcmtest.ni_playback ...
# #      SKIP      Can't read patterns. Probably, module isn't loaded
# # ni_playback: Test failed
# #          FAIL  pcmtest.ni_playback
# not ok 4 pcmtest.ni_playback #  Can't read patterns. Probably, module isn't loaded
# #  RUN           pcmtest.reset_ioctl ...
# #      SKIP      Can't read patterns. Probably, module isn't loaded
# # reset_ioctl: Test failed
# #          FAIL  pcmtest.reset_ioctl
# not ok 5 pcmtest.reset_ioctl #  Can't read patterns. Probably, module isn't loaded
# # FAILED: 0 / 5 tests passed.
# # Totals: pass:0 fail:5 xfail:0 xpass:0 skip:0 error:0

I haven't completely isolated the issue due to some other breakage
that's making it harder that it should be to test.  

A sample full log can be seen at:

   https://lava.sirena.org.uk/scheduler/job/659576#L1349

but there's no more context.  I'm also seeing some breakage in the
seccomp selftests which also use kselftest-harness:

# #  RUN           TRAP.dfl ...
# # dfl: Test exited normally instead of by signal (code: 0)
# #          FAIL  TRAP.dfl
# not ok 56 TRAP.dfl
# #  RUN           TRAP.ign ...
# # ign: Test exited normally instead of by signal (code: 0)
# #          FAIL  TRAP.ign
# not ok 57 TRAP.ign

and looks suspiciously like it might also be related.  Again, not
drilled down fully yet or looked at the code but this series jumps out
as most likely relevant.

Full log at:

   https://validation.linaro.org/scheduler/job/4046205#L6066

Download attachment "signature.asc" of type "application/pgp-signature" (489 bytes)

Powered by blists - more mailing lists

Powered by Openwall GNU/*/Linux Powered by OpenVZ