Message-ID: <4E26B721.9020103@steinhoff.de>
Date:	Wed, 20 Jul 2011 13:08:17 +0200
From:	Armin Steinhoff <armin@...inhoff.de>
To:	Thomas Gleixner <tglx@...utronix.de>
CC:	LKML <linux-kernel@...r.kernel.org>,
	linux-rt-users <linux-rt-users@...r.kernel.org>,
	Peter Zijlstra <peterz@...radead.org>,
	Ingo Molnar <mingo@...e.hu>, Carsten Emde <ce@...g.ch>,
	Clark Williams <williams@...hat.com>,
	"Paul E. McKenney" <paulmck@...ux.vnet.ibm.com>,
	Kumar Gala <galak@...e.crashing.org>,
	Ralf Baechle <ralf@...ux-mips.org>
Subject: Re: [ANNOUNCE] 3.0-rc7-rt0


Thomas,

congratulations !

The results are really amazing !!

https://www.osadl.org/Latency-plot-of-system-in-rack-4-slot.qa-latencyplot-r4s6.0.html?latencies=&showno=&slider=57

--Armin



Thomas Gleixner wrote:
> Dear RT Folks,
>
> I'm pleased to announce the first drop of the 3.0-rc7 based RT
> patch.
>
> It's been quite a while since 2.6.33-rt, but I went through a very
> painful experience while trying to get a 2.6.38-rt stabilized. The
> beast insisted on destroying filesystems with reproduction times
> measured in days and the total refusal to reveal at least a
> minimalistic hint to debug the root cause. Staring into completely
> useless traces for months is not a very pleasant pastime.
>
> That's the very first problem in the RT history which I gave up on.
>
> [The truth: Linus avoiding the final 2.6.42 release made all my
>   ultimate plans go down the drain ... ]
>
> Though while trying to analyse the problem I had plenty of time to
> twist my brain around the existing RT approach and its shortcomings.
>
> The main issue which RT is fighting with is the ever growing per cpu
> variable usage and the assumptions which are built around it. The
> existing RT approach to work around this with PER_CPU_LOCKED
> constructs and hand the CPU number around simply does not work anymore
> because the number of sites which need to be patched is way too large
> and the resulting mess in the code is neither acceptable nor
> maintainable.
>
> After lenghty and fruitful discussions with Peter Zijlstra - thanks a
> lot Peter! - we finally agreed on trying a totally different approach
> to tackle these issues: disabling migration over spinlock and get_cpu
> sections. This had been discussed before, but nobody ever considered
> to sit down and make it work.
>
> This keeps the semantics which are expected by the per cpu users,
> while keeping the regions preemptible. As a side effect, it allows us
> to run softirq handlers directly from irq threads on local_bh_enable
> which was a long desired feature to lower the performance impact of
> RT.
>
> Changing this required a major refactoring of the RT patch queue,
> which took some time as I had to go through every single patch, fold
> fixes back into the right places and sort them into various categories:
>
>   - Mainline ready (raw lock annotations, infrastructure patches, code
>     restructuring...)
>
>   - Preparatory (_rt()/_nort() variants of preempt_*(), local_irq_*(),
>     BUG*(), WARN*() and the annotations in various places)
>
>   - Base patches (Reworking the slab/page_alloc code, bit_spinlock
>     replacements, migrate disable infrastructure ...)
>
>   - Full RT patches (sleeping spinlocks and the resulting fixups here
>     and there)
>
> In course of that exercise I weeded out a lot of historically grown
> hackery and dropped stuff which was not essential for getting it up
> and running. Thanks to Carsten for reintegrating the tracer addons
> which he's using for the OSADL test farm:
>
>    https://www.osadl.org/?id=1042
>
> I probably have missed a few bits and pieces, but the overall outcome
> is stable and survived testing on various systems. The latency
> behaviour with cyclictest is on par with 33-rt at least on x86_64/32.
>
> The overall patch size has shrunk significantly and the readability
> (except for the missing changelogs in various patches) is at an
> acceptable level.
>
> If you download the quilt tarball, you'll find various sections:
>
> - upstream fixes: Stuff broken upstream which we managed to trip
>    over. This section contains real weird stuff from simple fixes, over
>    mainline code which claims to contain (complete bogus) RT support up
>    to an archaeologic bug in the floppy driver code.
>
>    8 patches (size 8892)
>    7 files changed, 59 insertions(+), 51 deletions(-)
>
> - upstream submitted: Stuff which is on LKML already and needs some
>    follow up.
>
>    4 patches (size 9741)
>    4 files changed, 81 insertions(+), 119 deletions(-)
>
> - upstream ready: Stuff which needs a bit polishing and upstream
>    submission
>
>    79 patches (size 232566)
>    192 files changed, 1204 insertions(+), 1097 deletions(-)
>
> - upstream needs work: Stuff which should go upstream, but needs some
>    or lots of care.
>
>    7 patches (size 164120)
>    49 files changed, 3292 insertions(+), 253 deletions(-)
>
> - the real rt stuff:
>
>    125 patches (size 280665)
>    162 files changed, 4327 insertions(+), 592 deletions(-)
>
> The overall patch is now:
>    223 patches (size 680054)
>    374 files changed, 8950 insertions(+), 2099 deletions(-)
>
> Compared that to 2.6.33-rt:
>    462 patches (size 1396505)
>    690 files changed, 15994 insertions(+), 5123 deletions(-)
>
> That's a significant reduction in size and impact. Some of it is due
> to the new approach, but we also got quite a lot of the infrastructure
> patches upstream in the last few kernel releases. Thanks to all folks
> who have helped to get that done, especially to Peter Zijlstra for
> getting the preemptible mmu gather problem and lots of the scheduler
> issues which we discovered in RT over time sorted out!!!
>
> What's new in 3.0-rt ?
>
>   - No more split soft interrupt threads. We need to analyze whether
>     this is a good decision.
>
>   - softirq handling from the end of interrupt threads and on all
>     thread sites where a nested local_bh disabled section ends
>
>   - SPARSE interrupts and IOMMU interrupt remapping work now
>
>   - Split config option CONFIG_PREEMPT_RT into CONFIG_PREEMPT_RT_BASE
>     and CONFIG_PREEMPT_RT_FULL. RT_BASE covers some of the more complex
>     changes (e.g. mm/* where we substitute interrupt disabled sections
>     with per cpu locks and the bit_spinlock to spinlock conversion).
>     RT_BASE allows us to test and verify these changes independently of
>     the big RT_FULL modifications. That's mainly a debugability and
>     maintainability issue.
>
> What's the state:
>
>     We've done quite some testing on x86 32/64 bit and basic tests on
>     some ARM/MIPS/POWERPC platforms. Thank God, no file system eating so
>     far :)
>
>     Given the fact that it is a major rewrite it's amazinlgy stable and
>     I consider it to be the best -rt1 release we ever had. That doesn't
>     mean that there are no bugs, since it has not had the proper test
>     coverage yet.
>
>     Thanks to Carsten, Clark and Peter for all the help to get this far!
>
> Want to help?
>
>     Many people offered help in the past and I had to turn them down so
>     far as refactoring that stuff really is not a task which can be
>     shared easily. Though now is the point where I can use all the help
>     you promised to provide.
>
>     What's needed?
>
>     - Testing, testing, testing ... you know the drill (good bug
>       reports are 98% of the solution)
>
>     - Compare and analyze the performance/troughput impact of the new
>       approach with 33-rt
>
>     - Help mainlining the "upstream ready section"
>
>       That means reviewing the patches, cleaning them up, fixing the
>       changelogs, submitting them through the proper channels ...
>
>       Please do not blindly pick any of these patches and submit them
>       to mailing lists w/o doing the above. Also please coordinate on
>       the #linux-rt IRC channel on oftc.net so redundant and
>       conflicting work can be avoided
>
>     - Help getting the "upstream needs work" section into shape
>
>       All of these patches need a close look and (especially the
>       hwlatency detector) major cleanups. Please coordinate with the
>       patch authors and lookout for previous discussions of some of
>       those on LKML.
>
>     - Tend to the FIXME annotations in the RT stuff section
>
>       I have annotated some places with /* FIXME ... comments. These
>       sections are not for the faint hearted and need some serious
>       review and thought.
>
>     - Help with the RCU modifications
>
>       That's an easy one. We have a volunteer signed up for this
>       involuntarily already. Thanks Paul!
>
>     - Twist your brain around the schedulability impact of the
>       migrate_disable() approach.
>
>       A really interesting research topic for our friends from the
>       academic universe. Relevant and conclusive (even short notice)
>       papers and/or talks on that topic have a reserved slot in the
>       Kernel developers track at the Realtime Linux Workshop in Prague
>       in October this year.
>
> Enough marketing, here comes the real stuff.
>
>    Patch against 3.0-rc7 can be found here:
>
>      http://www.kernel.org/pub/linux/kernel/projects/rt/patch-3.0-rc7-rt0.patch.bz2
>
>    The split quilt queue is available at:
>
>      http://www.kernel.org/pub/linux/kernel/projects/rt/patches-3.0-rc7-rt0.tar.gz
>
> There is no git tree for now.
>
> I'm not yet convinced that moving RT to git was a good idea as quilt
> allows me to move stuff around in a way more flexible manner. So for
> now no git version until someone comes up with a brilliant idea which
> allows me to keep my workflow sane (do not even try to suggest stgit&
> co!).
>
> That said, have fun and make sure that you have the fire extinguisher
> ready when you start using this!
>
> Thanks,
>
> 	tglx
> --
> To unsubscribe from this list: send the line "unsubscribe linux-rt-users" in
> the body of a message to majordomo@...r.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Powered by blists - more mailing lists