Message-ID: <kxi6m3gi7xqv52bupvb7iskyk6e3spq6bbhq4il5pmfieacfmf@5iwcnsfkmfq4>
Date: Wed, 9 Oct 2024 00:17:35 -0400
From: Kent Overstreet <kent.overstreet@...ux.dev>
To: Theodore Ts'o <tytso@....edu>
Cc: Linus Torvalds <torvalds@...ux-foundation.org>, 
	linux-bcachefs@...r.kernel.org, linux-fsdevel@...r.kernel.org, linux-kernel@...r.kernel.org
Subject: Re: [GIT PULL] bcachefs fixes for 6.12-rc2

On Tue, Oct 08, 2024 at 10:51:39PM GMT, Theodore Ts'o wrote:
> On Sun, Oct 06, 2024 at 12:33:51AM -0400, Kent Overstreet wrote:
> > 
> > Correct me if I'm wrong, but your system isn't available to the
> > community, and I haven't seen a CI or dashboard for kdevops?
> 
> It's up on github for anyone to download, and I've provided pre-built
> test appliance so people don't have to have downloaded xfstests and
> all of its dependencies and build it from scratch.  (That's been
> automated, of course, but the build infrastructure is setup to use a
> Debian build chroot, and with the precompiled test appliances, you can
> use my test runner on pretty much any Linux distribution; it will even
> work on MacOS if you have qemu built from macports, although for now
> you have to build the kernel on Linux distro using Parallels VM[1].)

How many steps are required, start to finish, to test a git branch and
get the results?

Compare that to my setup, where I give you an account, we set up the
config file that lists tests to run and git branches to test, and then
results show up in the dashboard.

> I'll note that IMHO making testing resources available to the
> community isn't really the bottleneck.  Using cloud resources,
> especially if you spin up the VM's only when you need to run the
> tests, and shut them down once the test is complete, which
> gce-xfstests does, is actually quite cheap.  At retail prices, running
> a dozen ext4 file system configurations against xfstests's "auto"
> group will take about 24 hours of VM time, and including the cost of
> the block devices, costs just under two dollars USD.  Because the
> tests are run in parallel, the total wall clock time to run all of the
> tests is about two and a half hours.  Running the "quick" group on a
> single file system configuration costs pennies.  So the $300 of free
> GCE credits will actually get someone pretty far!

That's the same argument that I've been making - machine resources are
cheap these days.

And using bare metal machines significantly simplifies the backend
(watchdogs, catching full kernel and test output, etc.).

> No, the bottleneck is having someone knowledgeable enough to interpret
> the test results and then finding the root cause of the failures.
> This is one of the reasons why I haven't stressed all that much about
> dashboards.  Dashboards are only useful if the right person(s) is
> looking at them.  That's why I've been much more interested in making
> it stupidly easy to run tests on someone's local resources, e.g.:
> 
>      https://github.com/tytso/xfstests-bld/blob/master/Documentation/kvm-quickstart.md

Yes, it needs to be trivial to run the same test locally that gets run
by the automated infrastructure, I've got that as well.

But dashboards are important, as well. And the git log based dashboard
I've got drastically reduces time spent manually bisecting.

> In fact, for most people, the entry point that I envision as being
> most interesting is that they download the kvm-xfstests, and following
> the instructions in the quickstart, so they can run "kvm-xfstests
> smoke" before sending me an ext4 patch.  Running the smoke test only
> takes 15 minutes using qemu, and it's much more convenient for them to
> run that on their local machine than to trigger the test on some
> remote machine, whether it's in the cloud or someone's remote test
> server.
> 
> In any case, that's why I haven't been interesting in working with
> your test infrastructure; I have my own, and in my opinion, my
> approach is the better one to make available to the community, and so
> when I have time to improve it, I'd much rather work on
> {kvm,gce,android}-xfstests.

Well, my setup also isn't tied to xfstests, and it's fairly trivial to
wrap all of our other (mm, block) tests.

But like I said before, I don't particularly care which one wins, as
long as we're pushing forward with something.

Powered by blists - more mailing lists