Message-ID: <20140403131610.GB7292@suse.de>
Date:	Thu, 3 Apr 2014 14:16:10 +0100
From:	Mel Gorman <mgorman@...e.de>
To:	Theodore Ts'o <tytso@....edu>
Cc:	Dave Chinner <david@...morbit.com>,
	Sedat Dilek <sedat.dilek@...il.com>,
	lsf@...ts.linux-foundation.org, linux-ext4@...r.kernel.org,
	linux-fsdevel@...r.kernel.org, xfs@....sgi.com
Subject: Re: [Lsf] [PATCH] xfstests-bld: Simplify determination of number of
 CPUs in build-all

On Wed, Apr 02, 2014 at 10:26:20AM -0400, Theodore Ts'o wrote:
> On Wed, Apr 02, 2014 at 09:28:23AM +1100, Dave Chinner wrote:
> > > And of course, whether changes in the mainline kernel tree are
> > > manually propagated changes from the xfstests.git tree, or whether
> > > primary development happens in the kernel tree, is ultimately going to
> > > be up to you and the XFS developers who have stewardship of xfstests.
> > > I'm not sure I would be that excited about manual propagation of
> > > changes from one git tree to another, but that is of course, up to
> > > you.
> > 
> > And this is exactly my point, Ted. Again, you are presuming that the
> > implementation is going to require syncing commits across disparate
> > git trees and other such games will be needed to maintain separate
> > packages. Nothing could be further from the truth: we already have
> > this problem with the shared XFS kernel/userspace code and it's a
> > royal PITA keeping them in sync. Hence introducing the same
> > maintenance problem with new code and infrastructure is highly
> > undesirable and something we'll try to avoid at all costs.
> 
> Actually, I was presuming that the thing that makes the most sense was
> to move all or most of the tests in xfstests into the kernel tests
> tree.  And then you complained that I was making a presumption that
> this was the only sane thing to do.  That's why I said, "if you want
> to do something insane, be my guest".
> 

FWIW, I also don't think that xfstests or a framework should go into the
kernel tree for the following reasons

1. Tests can be created during the release cycle and it does not make
   sense to tie the availabilty of tests to the availability of a kernel.
   If someone is running tests against historical tests, they must first
   checkout the most recent kernel tree, build xfstests and then checkout
   the old tree. Yes, users can be given a patch against the kernel tree
   to add a new test but they still have this awkward switching back and
   forth between the branch that has the test and the kernel they are
   running the test against.

   Recent example -- trinity 1.3 does not trigger MM bugs. Until recently,
   the latest git version of trinity caused massive damage.  If trinity had
   been in the tree someone might have accidentally concluded that 3.13 was
   "fine"

2. It would be relatively easy for someone to mistakenly bisect xfstests
   instead of bisecting the kernel although my understanding is that tests
   do not change per-se, only new ones are added. There may be
   exceptions to this rule

3. With testing automation, it's not that hard to download the
   requirements for it. mmtests prefers to use a local mirror but falls
   back to downloading tarballs, checking out git trees or mercurcial
   trees as appropriate. It's just a few lines of code (most of which is
   auto-generated from a template).

4. Testing monoculture. Sometimes the value of a benchmark is because
   different people run it differently for whatever reason. My IO tests for
   example all tune the benchmark to at least 2*RAM but that misses the cases
   where there are regressions in IO to a file that fits in memory. xfstests
   might have similar problems but minimally it would be a shame if the
   framework always used the same mount options for example. FWIW I've
   had bugs reported that were real bugs and only revealed because the
   benchmark parameters made no sense but still found a bug by accident.

Overall I have little motivation to having all the tests in the kernel
tree. That said, I already have an automated framework I can use so I
would not feel as strongly about the problem.

The point is that putting tests in tree or out of tree is neither sane nor
insane. It's simply different approaches so people on the thread should
be wary of trying to label the other person as being mental just because
of where they want to stick a test.

-- 
Mel Gorman
SUSE Labs
--
To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Powered by blists - more mailing lists