netdev - Re: Receive side performance issue with multi-10-GigE and NUMA

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-Id: <20090807185525.602c1607.billfink@mindspring.com>
Date:	Fri, 7 Aug 2009 18:55:25 -0400
From:	Bill Fink <billfink@...dspring.com>
To:	Brice Goglin <Brice.Goglin@...ia.fr>
Cc:	Linux Network Developers <netdev@...r.kernel.org>,
	Yinghai Lu <yhlu.kernel@...il.com>, gallatin@...i.com
Subject: Re: Receive side performance issue with multi-10-GigE and NUMA

On Sat, 08 Aug 2009, Brice Goglin wrote:

> Bill Fink wrote:
> > OK.  The tests were run on a 2.6.29.6 kernel so presumably should
> > have included the fix you mentioned.
> 
> Yes, but I wanted to emphasize that new platforms sometime need some new
> code to handle this kind of things. Some Nehalem-specific changes might
> be needed now.

Thanks for the clarification.

> >>>> Is the corresponding local_cpus sysfs file wrong as well ?
> >>>>         
> >>> All sysfs local_cpus values are the same (00000000,000000ff),
> >>> so yes they are also wrong.
> >>>       
> >> And hyperthreading is enabled, right?
> >>     
> >
> > No, hyperthreading is disabled.  It's a dual quad-core system so there
> > are a total of 8 cores, 4 on NUMA node 0 and 4 on NUMA node2.
> 
> So numa_node says that the device is close to node 0 while local_cpus
> says that it's close to all 8 cores ie close to both node0 and node2
> (which may well be wrong as well).

I believe it is wrong.  The basic system arcitecture is:

      Memory----CPU1----QPI----CPU2----Memory
                  |              |
                  |              |
                 QPI            QPI
                  |              |
                  |              |
                5520----QPI----5520
                ||||           ||||
                ||||           ||||
                ||||           ||||
                PCIe           PCIe

There are 2 x8, 1 x16, and 1 x4 PCIe 2.0 interfaces on each of the
Intel 5520 I/O Hubs.  The Myricom dual-port 10-GigE NICs are in the
six x8 or better slots.  eth2 through eth7 are on the second
Intel 5520 I/O Hub, so they should presumably show up on NUMA node 2,
and have local CPUs 1, 3, 5, and 7.  eth8 through eth13 are on the
first Intel 5520 I/O Hub, and thus should be on NUMA node 0 with
local CPUs 0, 2, 4, and 6 (CPU info derived from /proc/cpinfo).

						-Bill

--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html