[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]
Message-ID: <D12839161ADD3A4B8DA63D1A134D084026E48BA6AE@ESGSCCMS0001.eapac.ericsson.se>
Date: Sat, 9 Apr 2011 11:27:03 +0800
From: Wei Gu <wei.gu@...csson.com>
To: Eric Dumazet <eric.dumazet@...il.com>
CC: Alexander Duyck <alexander.h.duyck@...el.com>,
netdev <netdev@...r.kernel.org>,
"Kirsher, Jeffrey T" <jeffrey.t.kirsher@...el.com>
Subject: RE: Low performance Intel 10GE NIC (3.2.10) on 2.6.38 Kernel
HI Eric,
If I try to bind the 8 tx&rx queue to different NUMA Node to (core 3,7,11,15,19,23,27,31), looks doesn't help on the rx_missing_error anymore.
I still think the best performance would be binding NIC to one sock of CPU with it's local memory node.
I did a lot of combination on 2.6.32 kernel, by bind the eth10 to NODE2/3 could gain 20% more performance compare to NODE0/1.
So I guess the CPU Socket 2&3 was locally with the eth10.
Thanks
WeiGu
-----Original Message-----
From: Eric Dumazet [mailto:eric.dumazet@...il.com]
Sent: Friday, April 08, 2011 11:07 PM
To: Wei Gu
Cc: Alexander Duyck; netdev; Kirsher, Jeffrey T
Subject: RE: Low performance Intel 10GE NIC (3.2.10) on 2.6.38 Kernel
Le vendredi 08 avril 2011 à 22:10 +0800, Wei Gu a écrit :
> Hi,
> Got you mean.
> But as I decribed before, I start the eth10 with 8 rx queues and 8 tx
> queues, and then I binding these 8 tx&rx queue each to CPU core 24-32
> (NUMA3), which I think could gain the best performance in my case
> (It's true on Linux 2.6.32) single queue ->single CPU
Try with other cpus ? Maybe a mix.
Maybe your thinking is not good, and you chose the cpus that were not the best candidates. This was OK in 2.6.32 because you were lucky.
Using cpus from an unique NUMA node is not very good, since only one NUMA node is going to be used, and other NUMA nodes are idle.
NUMA binding is tricky. Linux try to use local node, hoping that all cpus are running and use local memory. In the end, global throughput is better.
But if your workload use cpus from one single node, then it means you lose part of the memory bandwidth.
> Then I can descibe a little bit with packet generator, I config the
> IXIA to continues increase the dest ip address towards the test
> server, so the packet was evenly distributed to each receving queues
> of the eth10. And according the IXIA tools the transmit sharp was
> really good, no too much peaks
>
> What I observed on Linux 2.6.38 during the test, there is no softqd
> was stressed (< 03% on SI for each core(24-31)) while the packet lost
> happens, so we are not really stress the CPU:), It looks like we are
> limited on some memory bandwidth (DMA) on this release
That would mean you chose the wrong cpus to handle this load.
>
> And with same test case on 2.6.32, no such problem at all. It running
> pretty stable > 2Mpps without rx_missing_error. There is no HW
> limitation on this DL580
>
>
> BTW what is these "swapper"
> + 0.80% swapper [ixgbe] [k]
> ixgbe_poll
> + 0.79% perf [ixgbe] [k]
> ixgbe_poll
> Why the ixgbe_poll was on swapper/perf?
>
softirq are run behalf the current interrupted thread, unless you enter ksoftirqd if load is high.
It can be "idle task" or the "perf" task, or another ones...
--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Powered by blists - more mailing lists