Message-ID: <20071120064747.2a8036cd@laptopd505.fenrus.org>
Date:	Tue, 20 Nov 2007 06:47:47 -0800
From:	Arjan van de Ven <arjan@...radead.org>
To:	Nick Piggin <nickpiggin@...oo.com.au>
Cc:	Mark Lord <lkml@....ca>, Andrew Morton <akpm@...ux-foundation.org>,
	Linus Torvalds <torvalds@...l.org>,
	Ingo Molnar <mingo@...e.hu>,
	Linux Kernel <linux-kernel@...r.kernel.org>
Subject: Re: CONFIG_IRQBALANCE for 64-bit x86 ?

On Tue, 20 Nov 2007 18:37:39 +1100
Nick Piggin <nickpiggin@...oo.com.au> wrote:
> > actually.... no. IRQ balancing is not a "fast" decision; every time
> > you
> 
> I didn't say anything of the sort. But IRQ load could still fluctuate
> a lot more rapidly than we'd like to wake up the irqbalancer.

irq load fluctuates by definition. but acting on it faster isn't the
right thing.
> 
> 
> > move an interrupt around, you end up causing a really a TON of cache
> > line bounces, and generally really bad performance
> 
> All the more reason why the kernel should do it. When I say move it to
> the kernel, I don't mean because I want to move IRQs 1 000 000 times
> per second and can't sustain enough context switches to do it in
> userspace. Userspace basically has insufficient information to do it
> as well as kernel.

like what?
Assuming this is a "once every few seconds" decision (and really it is,
esp for networking)....
> 
> 
> > (esp if you do it 
> > for networking ones, since you destroy the packet reassembly stuff
> > in the tcp/ip stack).
> >
> > Instead, what ends up working is if you do high level categories of
> > interrupt classes and balance within those (so that no 2 networking
> > irqs are on the same core/package unless you have more nics than
> > cores)
> 
> Sure, but you say that like it is difficult information for the kernel
> to know about. Actually it is much easier. Note that you can still
> bind interrupts to specific CPUs.

I assume you've read what/how irqbalance does; good luck convincing
people that that kind of policy belongs in the kernel.
> 
> 
> > etc. Balancing on a 10 second scale seems to work quite well; no
> > need to pull that complexity into the kernel....
> 
> My perspective is that it isn't a good idea to have such a critical
> piece of infrastructure outside the kernel.

kernel or kernel source? If there was a good place in the kernel source
I'd not be against moving irqbalance there. In the kernel... not needed.
(also because on single socket machines, the irqbalancer basically has
a one-shot task because there balancing is effectively a static setup)

The same ("critical piece of infrastructure') can be said about other
things, like udev and ... even hal. Nobody is arguing for moving those
into the kernel though....

> 
> I want the kernel to balance interrupts and tasks fairly; 

with irqthreads that will come for free soon.

>maybe move
> interrupts closer to the tasks they are interacting with (instead of,
> or combined with our current policy of moving tasks near the
> interrupts, which can be much more damaging for cache and NUMA);

interrupts and tasks have an N:M relationship.... or sometimes 1:M
where tasks only depend on one irq. Moving the irq around then tends to
be a loss. For NUMA, you actually very likely want the IRQ on the node
that the IO is associdated with.

> move
> all interrupts to a single core when there is enough capacity and we
> are balancing for power savings; 

irqbalance does that today.

>do exponential interrupt balancing
> backoff when it isn't required; etc. Not easy to do all that in
> userspace.
> 
> Any reason you actually think it is a good idea, aside from the fact
> that a userspace solution was able to be better than a crappy old
> kernel one?

I listed a few;
1) it's policy 
2) the memory is only needed for a short time (20 seconds or so) on
single-socket machines
3) it makes decisions on "subjective" information such as interrupt
device classes that the kernel currently just doesn't have (it could
grow that obviously), and is clearly policy information.




-- 
If you want to reach me at my work email, use arjan@...ux.intel.com
For development, discussion and tips for power savings, 
visit http://www.lesswatts.org
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Powered by blists - more mailing lists