Message-ID: <e473a90f-7a20-52e8-0158-94f283638f34@leemhuis.info>
Date:	Mon, 1 Aug 2016 21:41:53 +0200
From:	Thorsten Leemhuis <regressions@...mhuis.info>
To:	Linux Kernel Mailing List <linux-kernel@...r.kernel.org>
Cc:	Linus Torvalds <torvalds@...ux-foundation.org>,
	Jonathan Corbet <corbet@....net>, Takashi Iwai <tiwai@...e.de>,
	"Rafael J. Wysocki" <rjw@...ysocki.net>
Subject: Insights from doing regression tracking for Linux 4.7

In case anyone wonders if I regret doing regression tracking for Linux
4.7: No, that is not the case. It isn't really fun, but well, I didn't
expect it to be ;-) But FWIW, find below a few thoughts about the whole
regression tracking thing I thought might be good to write down and
share while they are still fresh in my head.

The TLDR version: I currently think it would help a lot to have
something like patchwork (https://github.com/getpatchwork/patchwork )
that is able to track regressions instead.

This is just a vague idea of mine right now I plan to investigate
further into sooner or later (I'm just coming back from a holiday and
have various conference talks to prepare -- besides doing my real job
and regression tracking for 4.8).

Here is the long version with the details that got me thinking into
above mentioned direction (in no particular order):

 * Some regression tracking work (track date and place of initial
report; are things proceeding, compile a report at least once a week,
...) feel time consuming and boring, because most of it is mechanical
work a computer should do, as that's what they were designed for ;-)

 * My work afaics helped to get a few regressions fixed that otherwise
might have been forgotten; but quite a few times I noticed pull request
mentioning fixes for regression that I had not been aware of; I wonder
how many regressions are still out there because both me and the
relevant subsystem maintainer missed them.

 * To make regression tracking really work well the workload must be
spread among multiple people. It would afaics be best to get developers
and maintainers involved more and give them something at hand to make
them track regressions in their area of work. But to make that happen we
need to have some kind of regression tracking software somewhere; and to
make developers and maintainers actually use it is has to make their
life easier somehow (similar to how patch tracking in patchwork makes
things easier at least for some people).

 * It's hard to get aware of all regression, as they are reported to
various places (LKML, bugzilla.kernel.org, as well as lots of other
mailing lists and a few other bugzillas). So a human or automatic
regression tracker need to get told about regressions. That currently
only sometimes happens for various reasons (I'm new to this; some people
prefer if their regressions are not mentioned in the spotlight; some
developers don't like to deal with bugzilla; ...). An email alias or a
dedicated mailing might be a step in the right direction; a better
solution might be a computer program that semi-automatically picks up
regressions in those places where they are reported currently.

 * Manually created regression reports (that's how I did them) sometimes
quickly get out of date; a more automated solution that compiles and
publishes up2date reports on the web would be better.

 * bugzilla.kernel.org is afaics not a good solution when it comes to
track regressions that are reported in other places. Main reason: Make
someone (maintainers or someone that tracks regression) create and
update bug entries for regressions mentioned in other places is a
solution that to me seems unlikely to fly due to the overhead bugzilla
has. Yes, Rafael did it like that when he did regression tracking, but
I'm currently think that time is better spend elsewhere; and having one
issue discussed in two places can quickly lead to confusion. Another
reason why bugzilla.kernel.org is afaics not a good solution: it
sometimes confusing to have tracked regressions and new, not yet
verified regressions in one place.

Those were my main thoughts on the whole thing; here are a few other:

* I'll try to do regression tracking for the 4.7 stable series (until
4.8 comes out), but doing it for mainline at least for 4.8 (not sure if
I can continue my work after that point) has higher priority for now, as
weeks only have 168 hours :-/

* For now I won't get into regression tracking for longterm kernels, as
the time afaics is better spend on getting regression tracking on a more
solid track.

 * Right now I only poke people manually when I think something needs to
be done to get things rolling again. Rafael sent automatic reminder
mails (together with the weekly reports I think). Maybe that is
something I should do, too; but OTOH it can quickly result in people
ignoring those mails.

* I should have helped to get
https://git.kernel.org/torvalds/c/262e2bfd7d1e1f1ee48b870e5dfabb87c06b975e
more quickly to mainline as the commit that introduced the regression
already had made it to the stable and broke things there for a few weeks
(and there was an earlier patch from jthumshirn that should have fixed
the issue, too) :-/

* I had hoped to spend more time helping users to (a) identify if their
problems actually are regressions and (b) getting their regressions
heard by the right people.  :-/

* I didn't do any statistics yet on how much my work helped (some
regressions I noticed were added and removed from my list between two
reports and thus for outsiders never really were on the reports); for
now that's not worth the work afaics.

* Something I avoided doing for now, because it's afaics quite a bit of
work: make bugzilla work better. It works well for some issues, but not
at all for others (most are somewhere in between). I for example suspect
the list of default assignees needs a major overhaul.

* Something else I'd like to do some when: Make it easier for novice
users to test for regressions. That among other means: improve the
documentation so people that want to test find their way in; for example
create docs that explain "how to test mainline rc kernels on the major
distributions easily and without risk", "how to use localmodconfig" or
"bisect regressions for dummies" in a central place would be nice to
have. And maybe even form groups of people that do regression testing
regularly; ideally together with Arch Linux, Fedora, openSUSE
Tumbleweed, and other distros that regularly ship the latest Linux
versions (and thus have a interest that new mainline versions do not
have to many regressions).

That it for now. HTH.

Ciao, Thorsten

Powered by blists - more mailing lists