netdev - Re: [PATCH 0/2] ixgbe, fix numa issues

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Date:	Mon, 24 Feb 2014 14:34:33 -0500
From:	Prarit Bhargava <prarit@...hat.com>
To:	Alexander Duyck <alexander.h.duyck@...el.com>
CC:	netdev@...r.kernel.org, Jeff Kirsher <jeffrey.t.kirsher@...el.com>,
	Jesse Brandeburg <jesse.brandeburg@...el.com>,
	Bruce Allan <bruce.w.allan@...el.com>,
	Carolyn Wyborny <carolyn.wyborny@...el.com>,
	Don Skidmore <donald.c.skidmore@...el.com>,
	Greg Rose <gregory.v.rose@...el.com>,
	John Ronciak <john.ronciak@...el.com>,
	Mitch Williams <mitch.a.williams@...el.com>,
	"David S. Miller" <davem@...emloft.net>, nhorman@...hat.com,
	agospoda@...hat.com, e1000-devel@...ts.sourceforge.net
Subject: Re: [PATCH 0/2] ixgbe, fix numa issues



On 02/24/2014 02:23 PM, Alexander Duyck wrote:
> On 02/24/2014 10:51 AM, Prarit Bhargava wrote:
>> The ixgbe driver makes some assumptions about the layout of cpus in the
>> system which are not always correct given a particular system layout.  The
>> ixgbe driver allocates one MSI/cpu for queue usage but the code does not take
>> into account that devices are located on NUMA nodes and that the cpus in a node
>> are not contiguous.
>>
>> These issues were found while doing cpu hotplug testing, however, both of these
>> issues can lead to obvious system performance issues as they defeat the
>> purpose of having one MSI processing a queue per cpu.
>>
>> Cc: Jeff Kirsher <jeffrey.t.kirsher@...el.com>
>> Cc: Jesse Brandeburg <jesse.brandeburg@...el.com>
>> Cc: Bruce Allan <bruce.w.allan@...el.com>
>> Cc: Carolyn Wyborny <carolyn.wyborny@...el.com>
>> Cc: Don Skidmore <donald.c.skidmore@...el.com>
>> Cc: Greg Rose <gregory.v.rose@...el.com>
>> Cc: Alex Duyck <alexander.h.duyck@...el.com>
>> Cc: John Ronciak <john.ronciak@...el.com>
>> Cc: Mitch Williams <mitch.a.williams@...el.com>
>> Cc: "David S. Miller" <davem@...emloft.net>
>> Cc: nhorman@...hat.com
>> Cc: agospoda@...hat.com
>> Cc: e1000-devel@...ts.sourceforge.net
>>
>> Prarit Bhargava (2):
>>   ixgbe, make interrupt allocations NUMA aware
>>   ixgbe, don't assume mapping of numa node cpus
>>
>>  drivers/net/ethernet/intel/ixgbe/ixgbe.h       |    2 ++
>>  drivers/net/ethernet/intel/ixgbe/ixgbe_lib.c   |   44 ++++++++++++++++++------
>>  drivers/net/ethernet/intel/ixgbe/ixgbe_main.c  |    6 ++--
>>  drivers/net/ethernet/intel/ixgbe/ixgbe_sriov.c |    5 +--
>>  4 files changed, 42 insertions(+), 15 deletions(-)
>>
> 
> This is a step in the right direction but totally defeats the purpose of
> ATR.  With this change we might as well defeature ATR all together since

ATR?  First hit on google is Americans for Tax Reform ;)

> things are now back to RSS w/ NUMA specific allocations which is what we
> had a couple of years ago.  The code as it is written now would be a
> better for for igb which doesn't have ATR than ixgbe.

The big(ger) problem here is that the ixgbe (and other drivers IIUC) do not do a
good job of handling MSIs, making sure they are launched on the right cpus, and
cleaning up during cpu hotplug operations.  This code looks like it needs a bit
of work so your advice is appreciated.

> 
> ATR is supposed to map 1:1 queues to CPUs.  The problem is RSS is also a
> factor and not especially smart or NUMA aware.  The ideal solution would
> be to allocate the first N CPUs, where N is the number in the local node
> for ATR/RSS.  

Okay ... I'll look into that.

Then map all other queues as ATR with a 1:1 mapping to CPUs.
> 

Hmmm ... but what if off-node CPUs cannot reach the device?  Part of the puzzle
here is that ACPI may be not only telling us that the device is on a specific
node, but that the device is physically separated on a root bus.

P.
--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html