Message-ID: <20080414074349.24fa90f8@laptopd505.fenrus.org>
Date:	Mon, 14 Apr 2008 07:43:49 -0700
From:	Arjan van de Ven <arjan@...radead.org>
To:	Andrew Morton <akpm@...ux-foundation.org>
Cc:	Al Viro <viro@...IV.linux.org.uk>, Willy Tarreau <w@....eu>,
	david@...g.hm, Stephen Clark <sclark46@...thlink.net>,
	Evgeniy Polyakov <johnpol@....mipt.ru>,
	"Rafael J. Wysocki" <rjw@...k.pl>, Tilman Schmidt <tilman@...p.cc>,
	Valdis.Kletnieks@...edu, Mark Lord <lkml@....ca>,
	David Miller <davem@...emloft.net>, jesper.juhl@...il.com,
	yoshfuji@...ux-ipv6.org, jeff@...zik.org,
	linux-kernel <linux-kernel@...r.kernel.org>, git@...r.kernel.org,
	netdev@...r.kernel.org
Subject: Re: Reporting bugs and bisection

On Mon, 14 Apr 2008 01:04:12 -0700
> 
> The steps to be taken are:
> 
> a) agree that we have a problem
> 


I for one do not agree that we have a problem.

Based on actual data on oopses (which very clearly excludes other kinds of bugs, so I know I only see part of the story)
we are doing reasonably well. Lets look at the 2.6.25 cycle. 
We got a total of roughly 2700 reports of oopses/warn_ons from users. (This may sound high to those of you only reading
lkml, but this includes automatically collected oopses from Fedora 9 beta testers).
Out of these 2700, the top 20 issues account for 75% of the total reports.

Out of these 20 issues, 9 were from still out of tree drivers (wireless.git and drm.git included in F9). These were
caught before they even got close to mainline.
The remaining 11 issues can be split in
1) The ones we caught and fixed
2) TCP/IP warnings that DaveM and co are chasing down hard (but have trouble finding reproducers)
3) An EXT3 bug that in theory can cause data corruption, but in practice seems to happen after you yank out a USB stick
  with an EXT3 filesystem on (so it can't corrupt the disk data). Ted is working on this
4) A bug (double free) that hits in the skb layer, probably caused by a bug in the ipv4 code
   (a first analysis + potential patch was mailed to netdev this weekend)
5) sysfs "existing file added" warning, mostly in the USB stack
   (gregkh claims he fixed this recently, I'm not entirely sure he got all cases)

And when I look beyond the first 20, the same pattern arises, we fixed the majority of the issues before -rc9.
At position 25 we have less than 20 reports per bug. At position 35 we have less than 10 reports per bug. 
At position 50 we have less than 5 reports per bug. Conclusion there: the bugs people actually hit fall of dramatically;
there's a core set of issues that gets hit a lot, the rest quickly gets reduced to noise levels.


To me this does not sound like we have a huge quality problem because
1) The distribution of the bugs is such that there is a relatively small set of core issues
   that are widely hit, and then there's a near exponential drop after that
2) We are fixing the important bugs by and large before they hit a release
   (important as defined by the number of people actually hitting the bug)


 
I'll be writing a report with more details about this soon with more analysis and statistics
(I'll be looking at more detail around the top 25 issues, when they got introduced, when they got fixed etc)







-- 
If you want to reach me at my work email, use arjan@...ux.intel.com
For development, discussion and tips for power savings, 
visit http://www.lesswatts.org
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Powered by blists - more mailing lists