Message-ID: <6dbbc85e-5a87-4605-8db6-92b191878d97@sirena.org.uk>
Date: Mon, 21 Oct 2024 22:41:54 +0100
From: Mark Brown <broonie@...nel.org>
To: Sasha Levin <sashal@...nel.org>
Cc: torvalds@...ux-foundation.org, ksummit@...ts.linux.dev,
	linux-kernel@...r.kernel.org
Subject: Re: linus-next: improving functional testing for to-be-merged pull
 requests

On Mon, Oct 21, 2024 at 12:07:13PM -0400, Sasha Levin wrote:

> During the Maintainer's Summit, Linus Torvalds expressed concerns about
> the quality of testing that code receives before he pulls it. The
> subsequent discussion side-tracked to the testability of linux-next, but
> we didn't directly address Linus's original concern about pre-pull
> testing quality.

> In an attempt to address the concerns, we're trying out a new "linus-next"
> tree is being created and maintained with the following characteristics:

> 	1. Composed of pull requests sent directly to Linus

> 	2. Contains branches destined for imminent inclusion by Linus

> 	3. Higher code quality expectation (these are pull requests that
> 	maintainers expect Linus to pull)

This feels kind of late for thorough testing, especially outside of the
merge window where it's mostly fixes.

> 	4. Continuous tree (not daily tags like in linux-next),
> 	facilitating easier bisection

Is this a pressing problem?  I routinely bisect -next, you have to base
things on Linus' tree (or pending-fixes) but otherwise it's not
especially problematic.  It is a bit helpful to remember commits that
you've run a given test on before to speed things up a bit but that's
not super essential.  The downside of not redoing the merge is that
once problematic stuff ends up in history it doesn't go away which can
itself be disruptive whenever there's multiple issues going on, you get
a window you have to skip through when bisecting.

> The linus-next tree aims to provide a more stable and testable
> integration point compared to linux-next, addressing the runtime issues
> that make testing linux-next challenging and focusing on code that's
> about to be pulled by Linus.

> linus-next is (expected to be) particularly effective before the merge
> window opens, as maintainers tend to send their pull requests early,
> allowing for more thorough testing of to-be-merged changes.

How many people send pull requests in advance, as opposed to getting
their tree stable, waiting for further testing, applying any fixes from
that testing and then sending a pull request after the merge window has
opened?

One big concern I have with this is that for non-fix changes the testing
coverage will be very heavily loaded to the end of the development cycle
and the merge window, with a lot of changes in flight at the same time.
That means it's going to be harder to effectively run more expensive
tests, and we'll potentially be a month or two on from when the changes
were landed by the time we actually get round to running whatever extra
tests get run here.

I do think it's useful to have coverage of specifically things actually
sent in pull requests, but I'm not sure we'd get much more thorough
testing than we already get via pending-fixes and -next.

> We also want to avoid altering the existing workflow. In particular:
> 
> 	1. No increase in latency. If anything, the expectation is that
> 	the cadence of merges would be improved given that Linus will
> 	need to do less builds and tests.

For fixes we do routinely have things landing within hours of the pull
request being sent, that's pretty aggressive for CI.

> 	2. Require "sign up" for the tree like linux-next does. Instead,
> 	pull requests are monitored and grabbed directly from the
> 	mailing list.

Like I mentioned above what's the plan for handling bad changes?  If an
issue is found in linus-next (BTW, I forsee confusion being caused by
typos there).  If a problem is found and then fixed with an incremental
patch which is sent in another pull request then the fix will eventually
land but if the offending patch is rebased out or the pull request just
dropped entirely then presumably someone will have to do something
manual.  How will that happen, and on what sort of schedule?

I do wonder if it's worth splitting the -next merge further and having
another pending-fixes type split where maintainers can differentiate
between changes where they're reasonably happy with the testing and
those that are bleeding edge.  It is volunteering Stephen to do even
more merges, and it does rely on people identifying what's stable, so
I'm not sure how useful it'd be but seems like it would mesh well with
the flow where people are doing their own stabalisation prior to sending
pull requests.  That's a different thing to increasing the coverage of
PRs like you're proposing here but it's another angle on the "-next is
too unstable" issue.

Download attachment "signature.asc" of type "application/pgp-signature" (489 bytes)

Powered by blists - more mailing lists