linux-kernel - Re: [RFC/PATCHv2] x86/irq: round-robin distribution of irqs to cpus w/in node

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <20100929171941.GH3096@sgi.com>
Date:	Wed, 29 Sep 2010 10:19:41 -0700
From:	Arthur Kepner <akepner@....com>
To:	"Eric W. Biederman" <ebiederm@...ssion.com>
Cc:	Thomas Gleixner <tglx@...utronix.de>, linux-kernel@...r.kernel.org,
	x86@...nel.org
Subject: Re: [RFC/PATCHv2] x86/irq: round-robin distribution of irqs to
	cpus w/in node


(Compendium reply to 2 emails.)

On Mon, Sep 27, 2010 at 05:17:07PM -0700, Eric W. Biederman wrote:
> Thomas Gleixner <tglx@...utronix.de> writes:
> 
> > On Mon, 27 Sep 2010, Arthur Kepner wrote:
> >
> ......
> The deep bug is that create_irq_nr allocates a vector (which it does
> because at the time there was no better way to mark an irq in use on
> x86).  In the case of msi-x we really don't know the node that irq is
> going to be used on until we get a request irq.  We simply know which
> node the device is on.
> 
> If you want to see what is going follow the call trace looks like.
> pci_enable_msix 
>   arch_setup_msi_irqs
>     create_irq_nr
> 
> After pci_enable_msix is finished then the driver goes and makes all
> of the irqs per cpu irqs.
> 
> There are goofy things that happen when hardware asks for 1 irq per cpu.
> But since msi can ask for up to 4096 irqs (assuming the hardware
> supports it) I can totally see putting all 256 of those irqs on a single
> cpu, before you go to user space and let user space or something
> reassign all of those irqs in a per cpu way.
> 

Yes, that's exactly the problem. All of the vectors on the lowest 
numbered CPUs get used. Any subsequent request for an interrupt on 
one of the low numbered CPUs will fail.

> .....

On Tue, Sep 28, 2010 at 03:59:33AM -0700, Eric W. Biederman wrote:
> Thomas Gleixner <tglx@...utronix.de> writes:
> 
> > On Mon, 27 Sep 2010, Eric W. Biederman wrote:
> >> > On Mon, 27 Sep 2010, Arthur Kepner wrote:
> >> The deep bug is that create_irq_nr allocates a vector (which it does
> >> because at the time there was no better way to mark an irq in use on
> >> x86).  In the case of msi-x we really don't know the node that irq is
> >> going to be used on until we get a request irq.  We simply know which
> >> node the device is on.
> >
> > Bah. So the whole per node allocation business is completely useless
> > at this point.
> 
> Probably.

Huh? No, the patch that started this thread spreads the irqs around 
and avoids the problem of a single CPU's vectors all being consumed.

> ...
> 
> Understood.  It has taken a couple of years before this bug finally
> bit anyone waiting a release or two to get it fixed properly seems
> reasonable.
> ....

And so what are we to do in the meantime? At the moment we're 
disabling MSIX, which is a pretty unattractive workaround.

-- 
Arthur
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/