linux-kernel - Re: [RFC] selftests: report proper exit statuses

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Date:	Mon, 14 Dec 2015 14:19:35 +1100
From:	Michael Ellerman <mpe@...erman.id.au>
To:	Brian Norris <computersforpeace@...il.com>,
	Shuah Khan <shuahkh@....samsung.com>
Cc:	linux-api@...r.kernel.org, linux-kernel@...r.kernel.org,
	Kees Cook <keescook@...omium.org>
Subject: Re: [RFC] selftests: report proper exit statuses

Hi Brian,

On Fri, 2015-12-11 at 15:15 -0800, Brian Norris wrote:

> There are several places where we don't report proper exit statuses, and
> this can have consequences -- for instance, the gen_kselftest_tar.sh
> script might try to produce a tarball for you, even if the 'make' or
> 'make install' steps didn't complete properly.
> 
> This is only an RFC (and really, it's more like a bug report), since I'm
> not really satisfied with my solution.

The changes to the tar script are probably OK.

But in general we do not want to exit after the first failure, which is what
your changes to the for loops would do.

The intention is to build and run as many tests as possible, on as many
architectures and machines as possible. So stopping the build because a header
or library is missing, or stopping the test run because one test fails, is the
exact opposite of what we want to happen.

For example a test might fail because it was written for x86 and doesn't work
on powerpc, if that caused all my powerpc tests to not run, I would be very
unpleased.

> It's probably not exhaustive, and
> there seem to be some major other deficiencies (e.g., verbose/useless
> output during build and run, non-paralle build, shell for-loops sidestep
> some normal 'make' behavior).

The goals for the kernel selftests are to make it as easy as possible to merge
tests, so that as many developers as possible create tests *and* merge them.

The current scheme supports that by not imposing much in the way of build
system requirements, or standards on what is or isn't appropriate output etc.

But if have ideas for improving things while still keeping the requirements on
test code low then I'm all ears.

cheers

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/