| lists.openwall.net | lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC | |
|
Open Source and information security mailing list archives
| ||
|
Message-ID: <52A46184.6070707@mellanox.com>
Date: Sun, 8 Dec 2013 14:09:40 +0200
From: Or Gerlitz <ogerlitz@...lanox.com>
To: Eric Dumazet <eric.dumazet@...il.com>,
Or Gerlitz <or.gerlitz@...il.com>
CC: Alexei Starovoitov <ast@...mgrid.com>,
David Miller <davem@...emloft.net>,
Joseph Gasparakis <joseph.gasparakis@...el.com>,
Jerry Chu <hkchu@...gle.com>,
Eric Dumazet <edumazet@...gle.com>,
Pravin B Shelar <pshelar@...ira.com>,
"netdev@...r.kernel.org" <netdev@...r.kernel.org>
Subject: Re: vxlan/veth performance issues on net.git + latest kernels
On 06/12/2013 15:36, Eric Dumazet wrote:
> On Fri, 2013-12-06 at 11:06 +0200, Or Gerlitz wrote:
>> On Wed, Dec 4, 2013 at 3:23 AM, Eric Dumazet <eric.dumazet@...il.com> wrote:
>>> skb->dev is in the first cache line, where we access skb->next anyway.
>>> I could use skb->cb[] like the following patch :
>> Hi Eric, I applied on the net tree the patch you posted yesterday
>> "net: introduce dev_consume_skb_any()" along with the network drivers
>> part of this patch, unless I got it wrong, I assume both pieces are
>> needed?
>>
>> So I re-run the vxlan/veth test that we suspect goes through packet drops on TX.
>>
>> With the patches applied I have almost no samples of that event
>>
>> $ ./perf report -i perf.data
> How did you get this perf.data file ? There are a few drops.
>
>> Samples: 89 of event 'skb:kfree_skb', Event count (approx.): 89
>> + 39.33% ksoftirqd/2 [kernel.kallsyms] [k] net_tx_action
>> + 28.09% swapper [kernel.kallsyms] [k] net_tx_action
>> + 28.09% sshd [kernel.kallsyms] [k] net_tx_action
>> + 2.25% swapper [kernel.kallsyms] [k] kfree_skb
>> + 1.12% kworker/2:2 [kernel.kallsyms] [k] net_tx_action
>> + 1.12% iperf [kernel.kallsyms] [k] net_tx_action
>>
>> ./perf report -i perf.data --sort dso,symbol
>> Samples: 89 of event 'skb:kfree_skb', Event count (approx.): 89
>> + 97.75% [kernel.kallsyms] [k] net_tx_action
>> + 2.25% [kernel.kallsyms] [k] kfree_skb
>> --
> OK, this means your driver drops few packets in its ndo_start_xmit() handler.
I wasn't sure to follow how the above lead you to think the drops occur
at the driver -- but, anyway, I applied your other patches and the one
below which made the mlx4 driver to make a kfree_skb(skb) call - and
don't see mlx4 hits in either the client or server side
client side:
Samples: 133 of event 'skb:kfree_skb', Event count (approx.): 133
+ 40.60% ksoftirqd/2 [kernel.kallsyms] [k] net_tx_action
+ 25.56% iperf [kernel.kallsyms] [k] net_tx_action
+ 24.06% swapper [kernel.kallsyms] [k] net_tx_action
+ 3.01% iperf [kernel.kallsyms] [k] kfree_skb
+ 2.26% swapper [kernel.kallsyms] [k] kfree_skb
+ 2.26% kworker/2:1 [kernel.kallsyms] [k] net_tx_action
+ 0.75% rcuc/2 [kernel.kallsyms] [k] net_tx_action
+ 0.75% kworker/2:2 [kernel.kallsyms] [k] net_tx_action
+ 0.75% ypbind [kernel.kallsyms] [k] net_tx_action
server side:
Samples: 57 of event 'skb:kfree_skb', Event count (approx.): 57
+ 47.37% swapper [kernel.kallsyms] [k] kfree_skb
+ 22.81% iperf [kernel.kallsyms] [k] kfree_skb
+ 8.77% ksoftirqd/2 [kernel.kallsyms] [k] kfree_skb
+ 7.02% hald-addon-acpi [kernel.kallsyms] [k] kfree_skb
+ 7.02% ls [kernel.kallsyms] [k] kfree_skb
+ 3.51% umount.nfs [kernel.kallsyms] [k] kfree_skb
+ 1.75% udevd [kernel.kallsyms] [k] kfree_skb
+ 1.75% rpc.idmapd [kernel.kallsyms] [k] kfree_skb
I will provide you OOB the full perf report files, but I have scrolled
into the hits and didn't see one in mlx4... any idea where/how to take
this from here?
>
> Could you give us "ifconfig -a" reports as I already asked ?
sure, I see on both sides there are some drops on 1Gbs NIC which is not
part of the test
client side (mlx4 NIC is eth6)
r-dcs44-005 perf]# ifconfig -a
br1 Link encap:Ethernet HWaddr 0E:08:CC:54:78:44
inet addr:192.168.52.144 Bcast:192.168.52.255 Mask:255.255.255.0
inet6 addr: fe80::c0c0:45ff:feff:bfed/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1450 Metric:1
RX packets:11 errors:0 dropped:0 overruns:0 frame:0
TX packets:8 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:732 (732.0 b) TX bytes:648 (648.0 b)
eth0 Link encap:Ethernet HWaddr 00:50:56:25:4B:05
inet addr:10.212.75.5 Bcast:10.212.255.255 Mask:255.255.0.0
inet6 addr: fe80::250:56ff:fe25:4b05/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:70485 errors:0 dropped:467 overruns:0 frame:0
TX packets:35821 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:58838076 (56.1 MiB) TX bytes:20632115 (19.6 MiB)
eth6 Link encap:Ethernet HWaddr 00:02:C9:E9:BB:B2
inet addr:192.168.30.144 Bcast:192.168.30.255 Mask:255.255.255.0
inet6 addr: fe80::2:c900:1e9:bbb2/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:58503788 errors:0 dropped:0 overruns:0 frame:0
TX packets:389819285 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:6604815254 (6.1 GiB) TX bytes:589822872168 (549.3 GiB)
eth7 Link encap:Ethernet HWaddr 52:54:00:86:B6:48
BROADCAST MULTICAST MTU:1500 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:0 (0.0 b) TX bytes:0 (0.0 b)
lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING MTU:65536 Metric:1
RX packets:137 errors:0 dropped:0 overruns:0 frame:0
TX packets:137 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:13860 (13.5 KiB) TX bytes:13860 (13.5 KiB)
veth0 Link encap:Ethernet HWaddr E6:95:68:49:A6:3D
inet addr:192.168.62.144 Bcast:192.168.62.255 Mask:255.255.255.0
inet6 addr: fe80::e495:68ff:fe49:a63d/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1450 Metric:1
RX packets:58510472 errors:0 dropped:0 overruns:0 frame:0
TX packets:55440581 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:3680050172 (3.4 GiB) TX bytes:552349060836 (514.4 GiB)
veth1 Link encap:Ethernet HWaddr 5A:4D:A3:4B:B1:97
inet6 addr: fe80::584d:a3ff:fe4b:b197/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1450 Metric:1
RX packets:55440581 errors:0 dropped:0 overruns:0 frame:0
TX packets:58510475 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:552349060836 (514.4 GiB) TX bytes:3680050334 (3.4 GiB)
vxlan42 Link encap:Ethernet HWaddr 0E:08:CC:54:78:44
inet addr:192.168.42.144 Bcast:192.168.42.255 Mask:255.255.255.0
inet6 addr: fe80::c08:ccff:fe54:7844/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1450 Metric:1
RX packets:58510461 errors:0 dropped:0 overruns:0 frame:0
TX packets:55440599 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:2860902728 (2.6 GiB) TX bytes:553236159764 (515.2 GiB)
server side (mlx4 NIC is eth2)
r-dcs47-005 perf]# ifconfig -a
br1 Link encap:Ethernet HWaddr 2A:9B:C5:5F:FA:AB
inet addr:192.168.52.147 Bcast:192.168.52.255 Mask:255.255.255.0
inet6 addr: fe80::cca:f9ff:fead:4210/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1450 Metric:1
RX packets:33 errors:0 dropped:0 overruns:0 frame:0
TX packets:8 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:2172 (2.1 KiB) TX bytes:648 (648.0 b)
eth0 Link encap:Ethernet HWaddr 00:50:56:25:4A:05
inet addr:10.212.74.5 Bcast:10.212.255.255 Mask:255.255.0.0
inet6 addr: fe80::250:56ff:fe25:4a05/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:15274 errors:0 dropped:18 overruns:0 frame:0
TX packets:9190 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:36265311 (34.5 MiB) TX bytes:5736966 (5.4 MiB)
eth2 Link encap:Ethernet HWaddr 00:02:C9:E9:C0:82
inet addr:192.168.30.147 Bcast:192.168.30.255 Mask:255.255.255.0
inet6 addr: fe80::2:c900:1e9:c082/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:442423939 errors:0 dropped:0 overruns:0 frame:0
TX packets:66524628 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:669425541728 (623.4 GiB) TX bytes:7511309282 (6.9 GiB)
eth3 Link encap:Ethernet HWaddr 52:54:00:5D:70:D9
BROADCAST MULTICAST MTU:1500 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:0 (0.0 b) TX bytes:0 (0.0 b)
lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING MTU:65536 Metric:1
RX packets:32 errors:0 dropped:0 overruns:0 frame:0
TX packets:32 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:2196 (2.1 KiB) TX bytes:2196 (2.1 KiB)
veth0 Link encap:Ethernet HWaddr 56:3D:34:30:86:68
inet addr:192.168.62.147 Bcast:192.168.62.255 Mask:255.255.255.0
inet6 addr: fe80::543d:34ff:fe30:8668/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1450 Metric:1
RX packets:442491685 errors:0 dropped:0 overruns:0 frame:0
TX packets:66538057 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:647403574534 (602.9 GiB) TX bytes:4185942998 (3.8 GiB)
veth1 Link encap:Ethernet HWaddr 2A:9B:C5:5F:FA:AB
inet6 addr: fe80::289b:c5ff:fe5f:faab/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1450 Metric:1
RX packets:66538066 errors:0 dropped:0 overruns:0 frame:0
TX packets:442491738 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:4185943484 (3.8 GiB) TX bytes:647403652126 (602.9 GiB)
vxlan42 Link encap:Ethernet HWaddr F6:E2:99:BD:D6:58
inet addr:192.168.42.147 Bcast:192.168.42.255 Mask:255.255.255.0
inet6 addr: fe80::f4e2:99ff:febd:d658/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1450 Metric:1
RX packets:442491767 errors:0 dropped:0 overruns:0 frame:0
TX packets:66538082 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:641208829864 (597.1 GiB) TX bytes:5250554092 (4.8 GiB)
>
> You could temporary change the dev_kfree_skb_any() in mlx4_en_xmit()
> to call kfree_skb(skb) instead, to get a stack trace (perf record -a -g
> -e skb:kfree_skb sleep 20 ; perf report)
>
> diff --git a/drivers/net/ethernet/mellanox/mlx4/en_tx.c b/drivers/net/ethernet/mellanox/mlx4/en_tx.c
> index f54ebd5a1702..53130f27dec0 100644
> --- a/drivers/net/ethernet/mellanox/mlx4/en_tx.c
> +++ b/drivers/net/ethernet/mellanox/mlx4/en_tx.c
> @@ -873,7 +873,7 @@ tx_drop_unmap:
> }
>
> tx_drop:
> - dev_kfree_skb_any(skb);
> + kfree_skb(skb);
> priv->stats.tx_dropped++;
> return NETDEV_TX_OK;
> }
>
>
--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Powered by blists - more mailing lists