lists.openwall.net | lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC | |
Open Source and information security mailing list archives
| ||
|
Date: Sun, 8 Dec 2013 14:09:40 +0200 From: Or Gerlitz <ogerlitz@...lanox.com> To: Eric Dumazet <eric.dumazet@...il.com>, Or Gerlitz <or.gerlitz@...il.com> CC: Alexei Starovoitov <ast@...mgrid.com>, David Miller <davem@...emloft.net>, Joseph Gasparakis <joseph.gasparakis@...el.com>, Jerry Chu <hkchu@...gle.com>, Eric Dumazet <edumazet@...gle.com>, Pravin B Shelar <pshelar@...ira.com>, "netdev@...r.kernel.org" <netdev@...r.kernel.org> Subject: Re: vxlan/veth performance issues on net.git + latest kernels On 06/12/2013 15:36, Eric Dumazet wrote: > On Fri, 2013-12-06 at 11:06 +0200, Or Gerlitz wrote: >> On Wed, Dec 4, 2013 at 3:23 AM, Eric Dumazet <eric.dumazet@...il.com> wrote: >>> skb->dev is in the first cache line, where we access skb->next anyway. >>> I could use skb->cb[] like the following patch : >> Hi Eric, I applied on the net tree the patch you posted yesterday >> "net: introduce dev_consume_skb_any()" along with the network drivers >> part of this patch, unless I got it wrong, I assume both pieces are >> needed? >> >> So I re-run the vxlan/veth test that we suspect goes through packet drops on TX. >> >> With the patches applied I have almost no samples of that event >> >> $ ./perf report -i perf.data > How did you get this perf.data file ? There are a few drops. > >> Samples: 89 of event 'skb:kfree_skb', Event count (approx.): 89 >> + 39.33% ksoftirqd/2 [kernel.kallsyms] [k] net_tx_action >> + 28.09% swapper [kernel.kallsyms] [k] net_tx_action >> + 28.09% sshd [kernel.kallsyms] [k] net_tx_action >> + 2.25% swapper [kernel.kallsyms] [k] kfree_skb >> + 1.12% kworker/2:2 [kernel.kallsyms] [k] net_tx_action >> + 1.12% iperf [kernel.kallsyms] [k] net_tx_action >> >> ./perf report -i perf.data --sort dso,symbol >> Samples: 89 of event 'skb:kfree_skb', Event count (approx.): 89 >> + 97.75% [kernel.kallsyms] [k] net_tx_action >> + 2.25% [kernel.kallsyms] [k] kfree_skb >> -- > OK, this means your driver drops few packets in its ndo_start_xmit() handler. I wasn't sure to follow how the above lead you to think the drops occur at the driver -- but, anyway, I applied your other patches and the one below which made the mlx4 driver to make a kfree_skb(skb) call - and don't see mlx4 hits in either the client or server side client side: Samples: 133 of event 'skb:kfree_skb', Event count (approx.): 133 + 40.60% ksoftirqd/2 [kernel.kallsyms] [k] net_tx_action + 25.56% iperf [kernel.kallsyms] [k] net_tx_action + 24.06% swapper [kernel.kallsyms] [k] net_tx_action + 3.01% iperf [kernel.kallsyms] [k] kfree_skb + 2.26% swapper [kernel.kallsyms] [k] kfree_skb + 2.26% kworker/2:1 [kernel.kallsyms] [k] net_tx_action + 0.75% rcuc/2 [kernel.kallsyms] [k] net_tx_action + 0.75% kworker/2:2 [kernel.kallsyms] [k] net_tx_action + 0.75% ypbind [kernel.kallsyms] [k] net_tx_action server side: Samples: 57 of event 'skb:kfree_skb', Event count (approx.): 57 + 47.37% swapper [kernel.kallsyms] [k] kfree_skb + 22.81% iperf [kernel.kallsyms] [k] kfree_skb + 8.77% ksoftirqd/2 [kernel.kallsyms] [k] kfree_skb + 7.02% hald-addon-acpi [kernel.kallsyms] [k] kfree_skb + 7.02% ls [kernel.kallsyms] [k] kfree_skb + 3.51% umount.nfs [kernel.kallsyms] [k] kfree_skb + 1.75% udevd [kernel.kallsyms] [k] kfree_skb + 1.75% rpc.idmapd [kernel.kallsyms] [k] kfree_skb I will provide you OOB the full perf report files, but I have scrolled into the hits and didn't see one in mlx4... any idea where/how to take this from here? > > Could you give us "ifconfig -a" reports as I already asked ? sure, I see on both sides there are some drops on 1Gbs NIC which is not part of the test client side (mlx4 NIC is eth6) r-dcs44-005 perf]# ifconfig -a br1 Link encap:Ethernet HWaddr 0E:08:CC:54:78:44 inet addr:192.168.52.144 Bcast:192.168.52.255 Mask:255.255.255.0 inet6 addr: fe80::c0c0:45ff:feff:bfed/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1450 Metric:1 RX packets:11 errors:0 dropped:0 overruns:0 frame:0 TX packets:8 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:732 (732.0 b) TX bytes:648 (648.0 b) eth0 Link encap:Ethernet HWaddr 00:50:56:25:4B:05 inet addr:10.212.75.5 Bcast:10.212.255.255 Mask:255.255.0.0 inet6 addr: fe80::250:56ff:fe25:4b05/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:70485 errors:0 dropped:467 overruns:0 frame:0 TX packets:35821 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:58838076 (56.1 MiB) TX bytes:20632115 (19.6 MiB) eth6 Link encap:Ethernet HWaddr 00:02:C9:E9:BB:B2 inet addr:192.168.30.144 Bcast:192.168.30.255 Mask:255.255.255.0 inet6 addr: fe80::2:c900:1e9:bbb2/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:58503788 errors:0 dropped:0 overruns:0 frame:0 TX packets:389819285 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:6604815254 (6.1 GiB) TX bytes:589822872168 (549.3 GiB) eth7 Link encap:Ethernet HWaddr 52:54:00:86:B6:48 BROADCAST MULTICAST MTU:1500 Metric:1 RX packets:0 errors:0 dropped:0 overruns:0 frame:0 TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:0 (0.0 b) TX bytes:0 (0.0 b) lo Link encap:Local Loopback inet addr:127.0.0.1 Mask:255.0.0.0 inet6 addr: ::1/128 Scope:Host UP LOOPBACK RUNNING MTU:65536 Metric:1 RX packets:137 errors:0 dropped:0 overruns:0 frame:0 TX packets:137 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:13860 (13.5 KiB) TX bytes:13860 (13.5 KiB) veth0 Link encap:Ethernet HWaddr E6:95:68:49:A6:3D inet addr:192.168.62.144 Bcast:192.168.62.255 Mask:255.255.255.0 inet6 addr: fe80::e495:68ff:fe49:a63d/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1450 Metric:1 RX packets:58510472 errors:0 dropped:0 overruns:0 frame:0 TX packets:55440581 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:3680050172 (3.4 GiB) TX bytes:552349060836 (514.4 GiB) veth1 Link encap:Ethernet HWaddr 5A:4D:A3:4B:B1:97 inet6 addr: fe80::584d:a3ff:fe4b:b197/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1450 Metric:1 RX packets:55440581 errors:0 dropped:0 overruns:0 frame:0 TX packets:58510475 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:552349060836 (514.4 GiB) TX bytes:3680050334 (3.4 GiB) vxlan42 Link encap:Ethernet HWaddr 0E:08:CC:54:78:44 inet addr:192.168.42.144 Bcast:192.168.42.255 Mask:255.255.255.0 inet6 addr: fe80::c08:ccff:fe54:7844/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1450 Metric:1 RX packets:58510461 errors:0 dropped:0 overruns:0 frame:0 TX packets:55440599 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:2860902728 (2.6 GiB) TX bytes:553236159764 (515.2 GiB) server side (mlx4 NIC is eth2) r-dcs47-005 perf]# ifconfig -a br1 Link encap:Ethernet HWaddr 2A:9B:C5:5F:FA:AB inet addr:192.168.52.147 Bcast:192.168.52.255 Mask:255.255.255.0 inet6 addr: fe80::cca:f9ff:fead:4210/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1450 Metric:1 RX packets:33 errors:0 dropped:0 overruns:0 frame:0 TX packets:8 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:2172 (2.1 KiB) TX bytes:648 (648.0 b) eth0 Link encap:Ethernet HWaddr 00:50:56:25:4A:05 inet addr:10.212.74.5 Bcast:10.212.255.255 Mask:255.255.0.0 inet6 addr: fe80::250:56ff:fe25:4a05/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:15274 errors:0 dropped:18 overruns:0 frame:0 TX packets:9190 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:36265311 (34.5 MiB) TX bytes:5736966 (5.4 MiB) eth2 Link encap:Ethernet HWaddr 00:02:C9:E9:C0:82 inet addr:192.168.30.147 Bcast:192.168.30.255 Mask:255.255.255.0 inet6 addr: fe80::2:c900:1e9:c082/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:442423939 errors:0 dropped:0 overruns:0 frame:0 TX packets:66524628 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:669425541728 (623.4 GiB) TX bytes:7511309282 (6.9 GiB) eth3 Link encap:Ethernet HWaddr 52:54:00:5D:70:D9 BROADCAST MULTICAST MTU:1500 Metric:1 RX packets:0 errors:0 dropped:0 overruns:0 frame:0 TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:0 (0.0 b) TX bytes:0 (0.0 b) lo Link encap:Local Loopback inet addr:127.0.0.1 Mask:255.0.0.0 inet6 addr: ::1/128 Scope:Host UP LOOPBACK RUNNING MTU:65536 Metric:1 RX packets:32 errors:0 dropped:0 overruns:0 frame:0 TX packets:32 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:2196 (2.1 KiB) TX bytes:2196 (2.1 KiB) veth0 Link encap:Ethernet HWaddr 56:3D:34:30:86:68 inet addr:192.168.62.147 Bcast:192.168.62.255 Mask:255.255.255.0 inet6 addr: fe80::543d:34ff:fe30:8668/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1450 Metric:1 RX packets:442491685 errors:0 dropped:0 overruns:0 frame:0 TX packets:66538057 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:647403574534 (602.9 GiB) TX bytes:4185942998 (3.8 GiB) veth1 Link encap:Ethernet HWaddr 2A:9B:C5:5F:FA:AB inet6 addr: fe80::289b:c5ff:fe5f:faab/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1450 Metric:1 RX packets:66538066 errors:0 dropped:0 overruns:0 frame:0 TX packets:442491738 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:4185943484 (3.8 GiB) TX bytes:647403652126 (602.9 GiB) vxlan42 Link encap:Ethernet HWaddr F6:E2:99:BD:D6:58 inet addr:192.168.42.147 Bcast:192.168.42.255 Mask:255.255.255.0 inet6 addr: fe80::f4e2:99ff:febd:d658/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1450 Metric:1 RX packets:442491767 errors:0 dropped:0 overruns:0 frame:0 TX packets:66538082 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:641208829864 (597.1 GiB) TX bytes:5250554092 (4.8 GiB) > > You could temporary change the dev_kfree_skb_any() in mlx4_en_xmit() > to call kfree_skb(skb) instead, to get a stack trace (perf record -a -g > -e skb:kfree_skb sleep 20 ; perf report) > > diff --git a/drivers/net/ethernet/mellanox/mlx4/en_tx.c b/drivers/net/ethernet/mellanox/mlx4/en_tx.c > index f54ebd5a1702..53130f27dec0 100644 > --- a/drivers/net/ethernet/mellanox/mlx4/en_tx.c > +++ b/drivers/net/ethernet/mellanox/mlx4/en_tx.c > @@ -873,7 +873,7 @@ tx_drop_unmap: > } > > tx_drop: > - dev_kfree_skb_any(skb); > + kfree_skb(skb); > priv->stats.tx_dropped++; > return NETDEV_TX_OK; > } > > -- To unsubscribe from this list: send the line "unsubscribe netdev" in the body of a message to majordomo@...r.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Powered by blists - more mailing lists