netdev - Re: vxlan/veth performance issues on net.git + latest kernels

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <CAJZOPZLqVxS_=9CLfodz2=ncZ0K2xEJaOVoUWUgoSDY7q4yKWg@mail.gmail.com>
Date:	Sat, 7 Dec 2013 23:20:40 +0200
From:	Or Gerlitz <or.gerlitz@...il.com>
To:	Eric Dumazet <eric.dumazet@...il.com>
Cc:	Alexei Starovoitov <ast@...mgrid.com>,
	David Miller <davem@...emloft.net>,
	Joseph Gasparakis <joseph.gasparakis@...el.com>,
	Jerry Chu <hkchu@...gle.com>,
	Or Gerlitz <ogerlitz@...lanox.com>,
	Eric Dumazet <edumazet@...gle.com>,
	Pravin B Shelar <pshelar@...ira.com>,
	"netdev@...r.kernel.org" <netdev@...r.kernel.org>
Subject: Re: vxlan/veth performance issues on net.git + latest kernels

On Fri, Dec 6, 2013 at 3:36 PM, Eric Dumazet <eric.dumazet@...il.com> wrote:
> On Fri, 2013-12-06 at 11:06 +0200, Or Gerlitz wrote:
>> On Wed, Dec 4, 2013 at 3:23 AM, Eric Dumazet <eric.dumazet@...il.com> wrote:
>> > skb->dev is in the first cache line, where we access skb->next anyway.
>> > I could use skb->cb[] like the following patch :
>>
>> Hi Eric, I applied on the net tree the patch you posted yesterday
>> "net: introduce dev_consume_skb_any()" along with the network drivers
>> part of this patch, unless I got it wrong, I assume both pieces are
>> needed?
>>
>> So I re-run the vxlan/veth test that we suspect goes through packet drops on TX.
>>
>> With the patches applied I have almost no samples of that event
>>
>> $ ./perf report -i perf.data
>
> How did you get this perf.data file ? There are a few drops.

Using the command you suggested on the active side while running
traffic (iperf tcp)

$ ./perf record -e skb:kfree_skb -g -a sleep 10

>> Samples: 89  of event 'skb:kfree_skb', Event count (approx.): 89
>> +  39.33%  ksoftirqd/2  [kernel.kallsyms]  [k] net_tx_action
>> +  28.09%      swapper  [kernel.kallsyms]  [k] net_tx_action
>> +  28.09%         sshd  [kernel.kallsyms]  [k] net_tx_action
>> +   2.25%      swapper  [kernel.kallsyms]  [k] kfree_skb
>> +   1.12%  kworker/2:2  [kernel.kallsyms]  [k] net_tx_action
>> +   1.12%        iperf  [kernel.kallsyms]  [k] net_tx_action

>>  ./perf report -i perf.data --sort dso,symbol
>> Samples: 89  of event 'skb:kfree_skb', Event count (approx.): 89
>> +  97.75%  [kernel.kallsyms]  [k] net_tx_action
>> +   2.25%  [kernel.kallsyms]  [k] kfree_skb
>> --

> OK, this means your driver drops few packets in its ndo_start_xmit()
> handler.
>
> Could you give us "ifconfig -a" reports as I already asked  ?

will do tomorrow while infront of the setup.

When I provided you the info last time
http://marc.info/?l=linux-netdev&m=138610891121531&w=2 it included the
ifconfig -a output and no drops were seen there


>
> You could temporary change the dev_kfree_skb_any() in mlx4_en_xmit()
> to call kfree_skb(skb) instead, to get  a stack trace (perf record -a -g
> -e skb:kfree_skb sleep 20 ; perf report)

yes, tomorrow

>
> diff --git a/drivers/net/ethernet/mellanox/mlx4/en_tx.c b/drivers/net/ethernet/mellanox/mlx4/en_tx.c
> index f54ebd5a1702..53130f27dec0 100644
> --- a/drivers/net/ethernet/mellanox/mlx4/en_tx.c
> +++ b/drivers/net/ethernet/mellanox/mlx4/en_tx.c
> @@ -873,7 +873,7 @@ tx_drop_unmap:
>         }
>
>  tx_drop:
> -       dev_kfree_skb_any(skb);
> +       kfree_skb(skb);
>         priv->stats.tx_dropped++;
>         return NETDEV_TX_OK;
>  }
>
>
--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html