netdev - Re: Receive side performance issue with multi-10-GigE and NUMA

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <4A8A9671.8060002@myri.com>
Date:	Tue, 18 Aug 2009 07:54:25 -0400
From:	Andrew Gallatin <gallatin@...i.com>
To:	Bill Fink <billfink@...dspring.com>
CC:	Jesse Barnes <jbarnes@...tuousgeek.org>,
	"Brandeburg, Jesse" <jesse.brandeburg@...el.com>,
	Neil Horman <nhorman@...driver.com>,
	Brice Goglin <Brice.Goglin@...ia.fr>,
	Linux Network Developers <netdev@...r.kernel.org>,
	Yinghai Lu <yhlu.kernel@...il.com>
Subject: Re: Receive side performance issue with multi-10-GigE and NUMA

Bill Fink wrote:

> <         Latency: 0, Cache Line Size: 64 bytes

<...>

>>       Latency: 0, Cache Line Size: 256 bytes

A cache line size of 256 clearly seems wrong for a Xeon.  I assume all
devices on the SuperMicro show the same value?

> Interestingly, the "WC Enabled" is only indicated on the first two

The WC is probably a red herring.

What does ethtool -S show for the DMA write bandwidth of the
NICs on the SuperMicro?

These values are obtained serially, as the driver resets
the NIC (reset happens at load time, and ifconfig up),
so they could easily sum to more than the memory bandwidth
of the system.  But it would be good to check for any anomalies.

I can send you a pointer to a tool we use internally, which loads
some custom firmware on the NIC, and can exercise the DMA engines
on all the NICs in parallel.  This would give an idea of the
aggregate DMA bandwidth available on the system.  Let me know
if you're interested.

Drew
--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html