Message-ID: <20070426165950.GO3468@stusta.de>
Date:	Thu, 26 Apr 2007 18:59:50 +0200
From:	Adrian Bunk <bunk@...sta.de>
To:	Linus Torvalds <torvalds@...ux-foundation.org>
Cc:	Linux Kernel Mailing List <linux-kernel@...r.kernel.org>
Subject: Re: Linux 2.6.21

On Thu, Apr 26, 2007 at 08:47:26AM -0700, Linus Torvalds wrote:
> 
> 
> On Thu, 26 Apr 2007, Adrian Bunk wrote:
> > 
> > There is a conflict between Linus trying to release kernels every
> > 2 months and releasing with few regressions.
> 
> No. 
> 
> Regressions _increase_ with longer release cycles. They don't get fewer.
> 
> The fact is, we have a -stable series for a reason. The reason is that the 
> normal development kernel can work in three ways:
> 
>  (a) long release cycles, with two subcases:
> 	(a1) huge changes (ie a "long development series". This is what we 
> 	     used to have. There's no way to even track the regressions, 
> 	     because things just change too much.
> 	(a2) keep the development limited, just stretch out the 
> 	     "stabilization phase". This simply *does*not*work*. You might 
> 	     want it to work, but it's against human psychology. People 
> 	     get bored, and start wasting their time discussing esoteric 
> 	     scheduler issues which weren't regressions at all.
>  (b) Short and staggered release cycle: keep changes limited (like a2), 
>      but recognize when it gets counter-productive, and cut a release so 
>      that the stable team can continue with it, while most developers (who 
>      wouldn't have worked on the stable kernel _anyway_) don't get 
>      frustrated.

<SCNR>

They get frustrated because they focussed on developing new features 
instead of fixing regressions, and now it takes longer until their new 
features get merged because noone fixed the regressions...

</SCNR>

> And yes, we've gone for (b). With occasional "I'm not taking any half-way 
> scary things at _all_" releases, like 2.6.20 was.
> 
> > Trying to avoid regressions might in the worst case result in an -rc12 
> > and 4 months between releases. If the focus is on avoiding regressions 
> > this has to be accepted.
> 
> No. You are ignoring the reality of development. The reality is that you 
> have to balance things. If you have a four-month release cycle, where 

I'm not saying it always have to be 4 months.

> three and a half months are just "wait for reports to trickle in from 
> testers", you simply won't get _anything_ done. People will throw their 
> hands up in frustration and go somewhere else.

"wait for reports to trickle in from testers" is exactly the opposite of 
our problem.

I started the regression lists originally to prove the fairy tale
"noone tests -rc kernels" some kernel developers spread as wrong.

Look at the facts:
8 out of 14 regressions in my current list were reported in March or earlier.
And for many regressions fixed it took several weeks until debugging 
by a kernel developer was started.

We do not lack testers for getting bug reports quickly.
We lack developer manpower for debugging the many regression reports.

>...
> > 0 regressions is never realistic (especially since many regressions 
> > might not be reported during -rc), but IMHO we could do much better than 
> > what happened in 2.6.20 and 2.6.21.
> 
> 2.6.20 was actually really good. Yes, it had some regressions, but I do 
> believe that it was one of the least buggy releases we've had. The process 
> _worked_. 

In the country of the blind the one-eyed man is king...

> 2.6.21 was much less pleasant, but the timer thing really was 
> 
> > I'm not satisfied with the result, and the world won't stop turning when 
> > I'm not tracking 2.6.22-rc regressions.
> 
> True. However, it's sad that you feel like you can't bother to track them. 
> They were _very_ useful. The fact that you felt they weren't is just 
> becasue I think you had unrealistic expectations, and you think that the 
> stable people shouldn't have to have anything to do.
> 
> You're maintaining 2.6.16 yourself - do you not see what happens when you 
> decide that "zero regressions" is the target? You have to stop 
> development. And while that may sound like a good thing at any particular 
> time, it's a total *disaster* in the long run (not even very long, 
> actually: in the two-to-three release cycle kind of run), because while 
> you are in a "regression fix" mode, people still go on developing, and 
> you're just causing problems for the _next_ release by holding things up 
> too long.
> 
> That's the *real* reality: 5 to 7 _million_ lines of diffs in a release 
> every two to three months. Do you really think those changes stop just 
> because of a release process? No. If you drag out the releases to be 4+ 
> months, you'll just have 10-15 million lines of changes instead (or, more 
> likely, you'll have developers who can't be bothered any more, and you may 
> have just 2 million lines, and three years later you have a kernel that 
> isn't relevant any more. Look at any of the other Unixes).

There's not a realistic chance for 0 regressions, and 4 months was 
a worst case, not the average case.

But I am not happy with the current state of released kernels.

> In other words, there's a _reason_ we have staggered development. We have 
> the "crazy development trees" (aka -mm and various other trees), we have 
> the "development tree" (aka Linus' tree), and we have the -stable tree. If 
> the stable tree has a dozen known issues that they'll have to sort out 
> over the next two months, that's *fine*. That's kind of the point of the 
> stable tree.

And all the people who have to upgrade to 2.6.21 for getting an 
important security fix run into a dozen known (and many unknown) 
regressions.

I don't think that's fine.

> And you would helpe them with the 2.6.22-stable releases if you'd maintain 
> that list. Even if it is _designed_ not to go down to zero.
> 
> I suspect that you got overly optimistic from the fact that 2.6.20 really 
> _was_ an easy release. It was designed that way. You feel that it was bad 
> or average, but that's actually because of _your_ unrealistic 
> expectations, not becasue there was anything wrong with 2.6.20.

If we had the developer manpower to get each reported regression 
debugged and fixed [1] within three weeks, 2.6.21 might be in the shape 
I would have liked it to be today.

But there are the three interdependent variables time, developer 
manpower and quality. And few developer manpower and few time results in 
a lower quality of the release I'm not happy with.

Life has taught me that sometimes I'm right, sometimes I'm wrong, and 
sometimes both sides have a possible solution. We might agree to 
disagree, and you are the one who's opinion counts. I can only say that 
I am not happy with the result, and that I do therefore not spend my 
time on maintaining regression lists for 2.6.22 - and maintaining such 
lists is not something special noone else could do equally well.

> 		Linus

cu
Adrian

[1] "fixed" can also be e.g. "patch reverted" or "not a bug"

-- 

       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Powered by blists - more mailing lists