netdev - Re: Receive side performance issue with multi-10-GigE and NUMA

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [thread-next>] [day] [month] [year] [list]

Date:	Fri, 14 Aug 2009 12:38:32 -0400
From:	Bill Fink <billfink@...dspring.com>
To:	Andrew Gallatin <gallatin@...i.com>
Cc:	netdev@...r.kernel.org
Subject: Re: Receive side performance issue with multi-10-GigE and NUMA

Hi Drew,

On Fri, 14 Aug 2009, Andrew Gallatin wrote:

> Hi Bill,
> 
> A few questions.   I was looking at the manual for the
> X8DAH+-F, and it claims to support both I/OAT and DCA.
> Do you have either or both enabled?

I did not explicitly set either one, and the manual indicates they
are both enabled by default, which I also vaguely seem to recall
was the way they were set.  I'm not in at the office today so I
can't physically check.

> If yes, then
> what happens if you disable ioatdma (by setting
> net.ipv4.tcp_dma_copybreak=2147483647 with sysctl)?
> How about if you disable myri10ge's use of dca (load driver
> with myri10ge_dca=0).
> 
> Do you see any changes?

Good suggestions but unfortunately it didn't help (or hurt).
It may have helped a little bit on the transmit side (I saw one
test at 102 Gbps when the previous high I had seen was 101 Gbps),
but the receive side was still at 55 Mbps.

Would there be any difference between disabling I/OAT and DCA in
the BIOS versus the myri10ge module parameter and sysctl setting?
I can try any BIOS changes on Monday.

> I'm worried about ioatdma because I've seen problems with it
> before.  At least on Linux, it tends to busywait for the DMA
> to complete, which is actually slower than a memory copy in
> most cases that I've seen.
> 
> I'm worried about DCA because you've shown that the BIOS is buggy,
> so the tag table could be wrong (resulting in bad prefetching hints).

The new BIOS seems to be better at setting the NUMA node info.

> I'm also worried about DCA because I've never had the chance to
> use it on a 5520 based system, and there is always the chance
> that we may be doing something wrong ourselves in the NIC firmware
> (again resulting in bad prefetching hints).  Bad prefetching hints
> can cause cross-CPU chatter, and kill performance by wasting
> memory bandwidth, and dirtying a cache on another CPU
> for no reason.

Is there any easy way to monitor active memory bandwidth usage?

> Drew

						-Thanks

						-Bill

P.S.  I don't know if it's at all significant, but one time after
      a reboot that required an fsck because of exceeding the number
      of mounts without an fsck, thus incurring a significant delay
      in the boot process, the transmit performance dropped from
      its normal ~100 Gbps to 57 Gbps (similar to the receive side
      performance).  Another reboot restored the normal ~100 Gbps
      transmit side performance.  I have no idea why this might be,
      but I saw it once before when an fsck was required on boot,
      so it may not be a fluke.
--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html