[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]
Date: Sat, 08 Aug 2009 07:08:20 -0400
From: Andrew Gallatin <gallatin@...i.com>
To: Bill Fink <billfink@...dspring.com>
CC: Brice Goglin <Brice.Goglin@...ia.fr>,
Linux Network Developers <netdev@...r.kernel.org>,
Yinghai Lu <yhlu.kernel@...il.com>
Subject: Re: Receive side performance issue with multi-10-GigE and NUMA
Bill Fink wrote:
> On Fri, 07 Aug 2009, Andrew Gallatin wrote:
>
>> Bill Fink wrote:
>>
>>> All sysfs local_cpus values are the same (00000000,000000ff),
>>> so yes they are also wrong.
>> How were you handling IRQ binding? If local_cpus is wrong,
>> the irqbalance will not be able to make good decisions about
>> where to bind the NICs' IRQs. Did you try manually binding
>> each NICs's interrupt to a separate CPU on the correct node?
>
> Yes, all the NIC IRQs were bound to a CPU on the local NUMA node,
> and the nuttcp application had its CPU affinity set to the same
> CPU with its memory affinity bound to the same local NUMA node.
> And the irqbalance daemon wasn't running.
I must be misunderstanding something. I had thought that
alloc_pages() on NUMA would wind up doing alloc_pages_current(), which
would allocate based on default policy which (if not interleaved)
should allocate from the current NUMA node. And since restocking the
RX ring happens from a the driver's NAPI softirq context, then it
should always be restocking on the same node the memory is destined to
be consumed on.
Do I just not understand how alloc_pages() works on NUMA?
Thanks,
Drew
--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Powered by blists - more mailing lists