[<prev] [next>] [<thread-prev] [day] [month] [year] [list]
Message-ID: <542D48E3.2090403@mellanox.com>
Date: Thu, 2 Oct 2014 15:45:23 +0300
From: Amir Vadai <amirv@...lanox.com>
To: Eric Dumazet <eric.dumazet@...il.com>
CC: Or Gerlitz <gerlitz.or@...il.com>,
Alexei Starovoitov <ast@...mgrid.com>,
"David S. Miller" <davem@...emloft.net>,
Jesper Dangaard Brouer <brouer@...hat.com>,
Eric Dumazet <edumazet@...gle.com>,
John Fastabend <john.r.fastabend@...el.com>,
Linux Netdev List <netdev@...r.kernel.org>,
"Or Gerlitz" <or.gerlitz@...il.com>, <amira@...lanox.com>,
<idos@...lanox.com>, Yevgeny Petrilin <yevgenyp@...lanox.com>,
<eyalpe@...lanox.com>
Subject: Re: [PATCH v2 net-next] mlx4: optimize xmit path
On 10/2/2014 3:07 PM, Eric Dumazet wrote:
> On Thu, 2014-10-02 at 14:56 +0300, Amir Vadai wrote:
>> After making sure the sender thread and the TX completions are not on
>> the same CPU, I see the expected improvement. +0.5Mpps with tx
>> optimizations.
>
> I got 40% here.
>
> Hmm... both cpus are on the same socket, right ?
Yes. And on a NUMA node close to the NIC
>
> TX coalescing is properly setup ?
I will try to play with it, if it is too aggressive, the queue is stopped.
I will play again with it and the ring size to find a better spot.
>
> What interrupt rate do you get ?
~50K per second.
>
> You can take a look at where cycles are spent
>
> perf record -a -g sleep 5
> perf report
>
>
I will.
In case your curious, I have this on the CPU doing completions:
+ 44.86% swapper [kernel.kallsyms] [k] consume_skb
+ 21.38% swapper [kernel.kallsyms] [k] mlx4_en_poll_tx_cq
+ 6.81% swapper [kernel.kallsyms] [k] mlx4_en_free_tx_desc.isra.24
+ 6.07% swapper [kernel.kallsyms] [k] intel_idle
+ 2.98% swapper [kernel.kallsyms] [k] __dev_kfree_skb_any
+ 2.66% rngd rngd [.] 0x0000000000002749
+ 1.79% rngd [kernel.kallsyms] [k] consume_skb
+ 1.28% swapper [kernel.kallsyms] [k] irq_entries_start
+ 1.16% rngd [kernel.kallsyms] [k] mlx4_en_poll_tx_cq
+ 0.86% swapper [kernel.kallsyms] [k] swiotlb_unmap_page
+ 0.79% swapper [kernel.kallsyms] [k] unmap_single
+ 0.63% swapper [kernel.kallsyms] [k] irqtime_account_irq
+ 0.55% swapper [kernel.kallsyms] [k] mlx4_eq_int
+ 0.51% swapper [kernel.kallsyms] [k] eq_set_ci.isra.14
and this on the xmit thread:
+ 72.00% kpktgend_6 [kernel.kallsyms] [k] mlx4_en_xmit
+ 13.11% kpktgend_6 [kernel.kallsyms] [k] pktgen_thread_worker
+ 5.34% kpktgend_6 [kernel.kallsyms] [k] _raw_spin_lock
+ 3.80% kpktgend_6 [kernel.kallsyms] [k] swiotlb_map_page
+ 2.08% kpktgend_6 [kernel.kallsyms] [k] __iowrite64_copy
+ 1.21% kpktgend_6 [kernel.kallsyms] [k] skb_clone_tx_timestamp
+ 0.87% kpktgend_6 [kernel.kallsyms] [k] kthread_should_stop
+ 0.64% kpktgend_6 [kernel.kallsyms] [k] swiotlb_dma_mapping_error
+ 0.51% kpktgend_6 [kernel.kallsyms] [k] __local_bh_enable_ip
Thanks,
Amir
--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Powered by blists - more mailing lists