Message-ID: <20180322202558.GA28958@kroah.com>
Date:   Thu, 22 Mar 2018 21:25:58 +0100
From:   Greg Kroah-Hartman <gregkh@...uxfoundation.org>
To:     Guenter Roeck <linux@...ck-us.net>
Cc:     Brian Norris <briannorris@...omium.org>,
        Linux Kernel <linux-kernel@...r.kernel.org>,
        stable <stable@...r.kernel.org>,
        Leif Liddy <leif.linux@...il.com>,
        Matthias Kaehlcke <mka@...omium.org>,
        Daniel Drake <drake@...lessm.com>,
        Kai-Heng Feng <kai.heng.feng@...onical.com>,
        Hans de Goede <hdegoede@...hat.com>,
        Marcel Holtmann <marcel@...tmann.org>
Subject: Re: [PATCH 4.4 095/108] Bluetooth: btusb: Restore QCA Rome
 suspend/resume fix with a "rewritten" version

On Thu, Mar 22, 2018 at 11:56:35AM -0700, Guenter Roeck wrote:
> On Thu, Mar 22, 2018 at 06:52:51PM +0100, Greg Kroah-Hartman wrote:
> 
> [ ... ]
> > 
> > And that't the point to drive home here.  If you stay away from updating
> > to stable patches, you have a huge boatload of KNOWN SECURITY HOLES in
> > your product.  If you take them, you have the _possiblity_ of some bugs
> > added, but overall, the rate is _VERY_ small.  Guenter has numbers of
> > 2-4 patches per year cause problems.  That's lower than ANY other
> > development model I have ever seen anywhere.
> > 
> Unfortunately, people tend to be irrational. Yes, the regression rate I have
> observed is in the 0.1..0.15% range for v4.4.y and v4.14.y. Yet, there are
> still people who believe that we should not merge stable releases due to the
> regressions it causes (though they are much less vocal nowadays).
> > So, stick with known buggy/insecure devices?  Or take the updates and
> > handle the 1-2 problems a year they provide you.  I think the
> > cost-analysis is easy to make here :)
> > 
> 
> Agreed, on an objective basis. Unfortunately, one does not get credit for
> fixing bugs which have never been observed in the field because they have
> been fixed before they showed up. But one _does_ get blame for regressions.

Someone has half-way joked that they were going to turn an intern on the
stable releases and get a CVE assigned for every patch in them.  Just to
highlight just how many "real" things we are fixing before anyone
notices.

Some days I think that is going to be the only way people pay attention :(

> Even though there have been very few regressions in absolute numbers, the
> default reaction to newly observed problems is "it must be due to a stable
> release merge", even though it almost always turns out to be incorrect.
> 
> The only way to deal with that is to reduce regressions to 0, or as close
> to 0 as possible. 0.1% is good, but not good enough.

For some platforms, it is 0%.  Facebook has published numbers showing
this for a 2 year run of stable kernel releases.  When you start dealing
with crazy embedded/odd hardware platforms, the numbers does go up, just
because no one is testing those platforms before I do a release.

Hence the push to do the testing on the real hardware, which is why
kernel.ci and Linaro are now doing this.  If you note, we also have
people doing merges on their phones, and I get private emails from a
number of SoC companies showing that their merge/test cycle worked as
well.

And one note from that SoC testing, in the past 6 months since it has
started, I have _NO_ reported regressions on any stable release so far.
Not bad...

> Also, while I agree that we are much better off in respect to security,
> the verdict is still out if stable release merges actually improve release
> stability; I don't see a clear trend even with chromeos-4.4. Of course,
> it is all but impossible to say if this is due to 4.4.y or due to the
> 13,000+ patches we have on top of v4.4.y in chromeos-4.4.

Yeah, _THATS_ the major issue here.  The interaction of the 3+million
lines of out-of-tree crazyness in device trees still scares me.  But, as
the SoCs are now reporting, so far it's going well, but it's only been 6
months.  But it has been an "interesting" 6 months :)

As for "improve" stability, well, given that we are fixing
known-root-holes, yes, that does increase stability.  Again, I can crash
any phone shipping today except for 2 of them, because those 2 updated
to newer kernel versions.  Do I need to start publishing reproducers?

Actually, along those lines, I have seen people start putting tests for
reported kernel bugs into some regression tests.  When those start being
more popular (i.e. people start running them on devices that are not
updated), then you will start to see the reports of "instability".

Oh well, back to patch reviewing, I'm preaching to the choir here...

thanks,

greg k-h

Powered by blists - more mailing lists