linux-kernel - Re: automated test? (was Re: Linux 2.6.17.7)

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <20060726130207.GC23701@stusta.de>
Date:	Wed, 26 Jul 2006 15:02:07 +0200
From:	Adrian Bunk <bunk@...sta.de>
To:	David Lang <dlang@...italinsight.com>
Cc:	Andrew de Quincey <adq_dvb@...skialf.net>,
	Arnaud Patard <apatard@...driva.com>, Greg KH <gregkh@...e.de>,
	linux-kernel@...r.kernel.org, stable@...nel.org
Subject: Re: automated test? (was Re: Linux 2.6.17.7)

On Tue, Jul 25, 2006 at 09:47:43AM -0700, David Lang wrote:
> On Tue, 25 Jul 2006, Andrew de Quincey wrote:
> 
> >On Tuesday 25 July 2006 10:55, Arnaud Patard wrote:
> >>Greg KH <gregkh@...e.de> writes:
> >>
> >>Hi,
> >>
> >>>We (the -stable team) are announcing the release of the 2.6.17.7 kernel.
> >>
> >>Sorry, but doesn't compile if DVB_BUDGET_AV is set :(
> >>
> >>>Andrew de Quincey:
> >>>      v4l/dvb: Fix budget-av frontend detection
> >
> >
> >In fact it is just this patch causing the problem:
> <SNIP>
> >Sorry, I had so much work going on in that area I must have diffed the 
> >wrong
> >kernel when I created this patch. :(
> 
> is it reasonable to have an aotomated test figure out what config options 
> are relavent to a patch (or patchset) and test compile all the combinations 
> to catch this sort of mistake?

If you think about it, you'll notice it's definitely not reasonable:

#include <linux/module.h> brings you a dependency on 5 config options.
#include <linux/pci.h> brings you a dependency on 6 config options.

By only including these two headers you are at 2048 combinations.
The number of valid configurations will be lower, but 500 test compiles 
sound realistically.

With have a dozen #include's you might need more than a million test 
compiles.

With a dozen #include's, you might need a trilion [1] test compiles.

Compile errors are quickly catched and don't cause any serious problem.

What bothers me more is that noone tested this patch against the kernel 
it was applied against.

The submitter didn't test it works (he didn't even test the compilation).

No user tested it.

Currently, -stable kernels get an 48 hours review on linux-kernel but 
zero testing.

How could this be improved?
Longer review/testing time?
Offer them as also one 2.6.17.7-rc1 patch?

> David Lang

cu
Adrian

[1] American English

-- 

       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/