netdev - Re: vxlan/veth performance issues on net.git + latest kernels

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <1386337011.30495.253.camel@edumazet-glaptop2.roam.corp.google.com>
Date:	Fri, 06 Dec 2013 05:36:51 -0800
From:	Eric Dumazet <eric.dumazet@...il.com>
To:	Or Gerlitz <or.gerlitz@...il.com>
Cc:	Alexei Starovoitov <ast@...mgrid.com>,
	David Miller <davem@...emloft.net>,
	Joseph Gasparakis <joseph.gasparakis@...el.com>,
	Jerry Chu <hkchu@...gle.com>,
	Or Gerlitz <ogerlitz@...lanox.com>,
	Eric Dumazet <edumazet@...gle.com>,
	Pravin B Shelar <pshelar@...ira.com>,
	"netdev@...r.kernel.org" <netdev@...r.kernel.org>
Subject: Re: vxlan/veth performance issues on net.git + latest kernels

On Fri, 2013-12-06 at 11:06 +0200, Or Gerlitz wrote:
> On Wed, Dec 4, 2013 at 3:23 AM, Eric Dumazet <eric.dumazet@...il.com> wrote:
> > skb->dev is in the first cache line, where we access skb->next anyway.
> > I could use skb->cb[] like the following patch :
> 
> Hi Eric, I applied on the net tree the patch you posted yesterday
> "net: introduce dev_consume_skb_any()" along with the network drivers
> part of this patch, unless I got it wrong, I assume both pieces are
> needed?
> 
> So I re-run the vxlan/veth test that we suspect goes through packet drops on TX.
> 
> With the patches applied I have almost no samples of that event
> 
> $ ./perf report -i perf.data

How did you get this perf.data file ? There are a few drops.

> 
> Samples: 89  of event 'skb:kfree_skb', Event count (approx.): 89
> +  39.33%  ksoftirqd/2  [kernel.kallsyms]  [k] net_tx_action
> 
>                     ?
> +  28.09%      swapper  [kernel.kallsyms]  [k] net_tx_action
> 
>                     ?
> +  28.09%         sshd  [kernel.kallsyms]  [k] net_tx_action
> 
>                     ?
> +   2.25%      swapper  [kernel.kallsyms]  [k] kfree_skb
> 
>                     ?
> +   1.12%  kworker/2:2  [kernel.kallsyms]  [k] net_tx_action
> 
>                     ?
> +   1.12%        iperf  [kernel.kallsyms]  [k] net_tx_action
> 
>  ./perf report -i perf.data --sort dso,symbol
> Samples: 89  of event 'skb:kfree_skb', Event count (approx.): 89
> +  97.75%  [kernel.kallsyms]  [k] net_tx_action
> +   2.25%  [kernel.kallsyms]  [k] kfree_skb
> --

OK, this means your driver drops few packets in its ndo_start_xmit()
handler.

Could you give us "ifconfig -a" reports as I already asked  ?

You could temporary change the dev_kfree_skb_any() in mlx4_en_xmit()
to call kfree_skb(skb) instead, to get  a stack trace (perf record -a -g
-e skb:kfree_skb sleep 20 ; perf report)

diff --git a/drivers/net/ethernet/mellanox/mlx4/en_tx.c b/drivers/net/ethernet/mellanox/mlx4/en_tx.c
index f54ebd5a1702..53130f27dec0 100644
--- a/drivers/net/ethernet/mellanox/mlx4/en_tx.c
+++ b/drivers/net/ethernet/mellanox/mlx4/en_tx.c
@@ -873,7 +873,7 @@ tx_drop_unmap:
 	}
 
 tx_drop:
-	dev_kfree_skb_any(skb);
+	kfree_skb(skb);
 	priv->stats.tx_dropped++;
 	return NETDEV_TX_OK;
 }


--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html