netdev - Re: [PATCH net-next] mlx4: exploit skb->xmit_more to conditionally send doorbell

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <1411659380.16953.29.camel@edumazet-glaptop2.roam.corp.google.com>
Date:	Thu, 25 Sep 2014 08:36:20 -0700
From:	Eric Dumazet <eric.dumazet@...il.com>
To:	Jesper Dangaard Brouer <brouer@...hat.com>
Cc:	David Miller <davem@...emloft.net>,
	netdev <netdev@...r.kernel.org>, Amir Vadai <amirv@...lanox.com>,
	John Fastabend <john.r.fastabend@...el.com>
Subject: Re: [PATCH net-next] mlx4: exploit skb->xmit_more to conditionally
 send doorbell

On Thu, 2014-09-25 at 08:06 -0700, Eric Dumazet wrote:

> 
> But... Nothing yet in qdisc layer sets xmit_more, unless TSO is
> disabled, and a GSO packet is segmented.
> 
> So you might notice some improvement with this patch, but not if you use
> TSO, and a 40GB NIC is better with TSO on you know ;)
> 

So playing with TSO=off (but GSO=on), we see about 12 % increase of
throughput on a TCP_STREAM flow.

And kernel profile clearly shows the difference of sending or not the
doorbell, as mlx4_en_xmit() shifts. Note also mlx4_en_free_tx_desc()
disappears, this might point to some false sharing or something worth
investigating.

Before patch :

    26.36%  [kernel]  [k] __copy_skb_header             
    16.59%  [kernel]  [k] mlx4_en_xmit                  
     5.62%  [kernel]  [k] __alloc_skb                   
     5.48%  [kernel]  [k] copy_user_enhanced_fast_string
     4.48%  [kernel]  [k] skb_segment                   
     2.46%  [kernel]  [k] mlx4_en_free_tx_desc.isra.27  
     2.31%  [kernel]  [k] _raw_spin_lock                
     2.10%  [kernel]  [k] memcpy                        
     2.01%  [kernel]  [k] tcp_sendmsg                   
     1.62%  [kernel]  [k] __iowrite64_copy              


After patch :

    32.78%  [kernel]  [k] __copy_skb_header             
     8.26%  [kernel]  [k] mlx4_en_xmit                  
     7.25%  [kernel]  [k] __alloc_skb                   
     7.18%  [kernel]  [k] copy_user_enhanced_fast_string
     4.39%  [kernel]  [k] skb_segment                   
     2.87%  [kernel]  [k] memcpy                        
     2.59%  [kernel]  [k] tcp_sendmsg                   
     2.50%  [kernel]  [k] _raw_spin_lock                
     2.38%  [kernel]  [k] dev_hard_start_xmit           
     1.52%  [kernel]  [k] tcp_gso_segment               
     1.50%  [kernel]  [k] kmem_cache_alloc_node_trace   
     1.40%  [kernel]  [k] kmem_cache_alloc_node         
     1.16%  [kernel]  [k] ip_send_check                 


--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html