linux-kernel - Re: CONFIG_IRQBALANCE for 64-bit x86 ?

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <20071119213727.16d5917b@laptopd505.fenrus.org>
Date:	Mon, 19 Nov 2007 21:37:27 -0800
From:	Arjan van de Ven <arjan@...radead.org>
To:	Nick Piggin <nickpiggin@...oo.com.au>
Cc:	Mark Lord <lkml@....ca>, Andrew Morton <akpm@...ux-foundation.org>,
	Linus Torvalds <torvalds@...l.org>,
	Ingo Molnar <mingo@...e.hu>,
	Linux Kernel <linux-kernel@...r.kernel.org>
Subject: Re: CONFIG_IRQBALANCE for 64-bit x86 ?

On Tue, 20 Nov 2007 15:17:15 +1100
Nick Piggin <nickpiggin@...oo.com.au> wrote:

> On Tuesday 20 November 2007 15:12, Mark Lord wrote:
> > On 32-bit x86, we have CONFIG_IRQBALANCE available,
> > but not on 64-bit x86.  Why not?

because the in-kernel one is actually quite bad.

> > My QuadCore box works very well in 32-bit mode with IRQBALANCE,
> > but responsiveness sucks bigtime when run in 64-bit mode (no
> > IRQBALANCE) during periods of multiple heavy I/O streams (USB flash
> > drives).

please run the userspace irq balancer, see http://www.irqbalance.org
afaik most distros ship that by default anyway.

> > As near as I can tell, when IRQBALANCE is not configured,
> > all I/O device interrupts go to CPU#0.

that depends on your chipset; some chipsets do worse than that.

>
> > I don't think our CPU scheduler takes that into account when
> > assigning tasks to CPUs, so anything sent to CPU0 runs with very
> > high latencies.
> >
> > Or something like that.
> >
> > Why no IRQ_BALANCE in 64-bit mode ?
> 
> For that matter, I'd like to know why it has been decided that the
> best place for IRQ balancing is in userspace. It should be in kernel
> IMO, and it would probably allow better power saving, performance,
> fairness, etc. if it were to be integrated with the task balancer as
> well.

actually.... no. IRQ balancing is not a "fast" decision; every time you
move an interrupt around, you end up causing a really a TON of cache
line bounces, and generally really bad performance (esp if you do it
for networking ones, since you destroy the packet reassembly stuff in
the tcp/ip stack).

Instead, what ends up working is if you do high level categories of
interrupt classes and balance within those (so that no 2 networking
irqs are on the same core/package unless you have more nics than cores)
etc. Balancing on a 10 second scale seems to work quite well; no need
to pull that complexity into the kernel.... 

-- 
If you want to reach me at my work email, use arjan@...ux.intel.com
For development, discussion and tips for power savings, 
visit http://www.lesswatts.org
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/