Date:	Thu, 26 Jun 2014 12:30:22 -0700
From:	Eric Wheeler <netdev@...ts.ewheeler.net>
To:	netdev@...r.kernel.org
Cc:	rmody@...cade.com
Subject: Re: Slow VLAN-to-VLAN routing and many stacktraces of
 skb_warn_bad_offload+0xc3/0xd0()


> We are getting a large number of stack traces with skb_warn_bad_offload
> in dmesg running 3.14.6.  If you believe this is fixed in a later
> release, please let me know and I can test whatever later version you
> propose:

I tried Linus's git tree with the same .config and the stack trace went
away, so whatever that issue was has been fixed.  

We tried a gbit intel card instead of the brocade 1020 (bna) 10gbe card
and the VLAN performance was much better.  

Rasesh,

Are you aware of any VLAN tagging performance issues with the brocade
1020? 

When routing in eth0.102 (192.168.102.110) and out eth0.10
(192.168.100.33) we get a max of ~300mbit.  If we do the same thing with
an Intel (igb) gigabit card we get ~950mbit as expected.

I'll try different firmware and report back.  At the moment its not
clear why tagging performance would be poor.

-Eric

> 
> [157134.549229] ------------[ cut here ]------------
> [157134.549448] WARNING: CPU: 0 PID: 27273 at net/core/dev.c:2238 skb_warn_bad_offload+0xc3/0xd0()
> [157134.549882] bna: caps=(0x0000000100114bb3, 0x0000000007db78e9) len=2665 data_len=1139 gso_size=1448 gso_type=1 ip_summed=3
> [157134.550335] Modules linked in: vhost_net macvtap macvlan vhost tun dm_thin_pool dm_bio_prison dm_persistent_data dm_bufio mpt3sas mpt2sas scsi_transport_sas raid_class mptctl mptbase zram ipt_MASQUERADE iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack ipt_REJECT xt_CHECKSUM iptable_mangle ebtable_nat ebtables drbd(OF) libcrc32c xt_comment iptable_filter ip_tables bnx2fc cnic uio fcoe libfcoe libfc 8021q bridge stp llc dummy ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 xt_state nf_conntrack ip6table_filter ip6_tables ipv6 ext3 jbd kvm_intel kvm bcache iTCO_wdt iTCO_vendor_support serio_raw pcspkr i2c_i801 i2c_core acpi_cpufreq lpc_ich mfd_core e1000e ptp pps_core bna sg ext4(F) jbd2(F) mbcache(F) sd_mod(F) crc_t10dif(F) crct10dif_common(F) ahci(F) libahci(F) video(F) bfa(F) scsi_transport_fc(F) scsi_tgt(F) megaraid_sas(F) dm_mirror(F) dm_region_hash(F) dm_log(F) dm_mod(F)
> [157134.559980] CPU: 0 PID: 27273 Comm: curl Tainted: GF       W  O 3.14.6 #1
> [157134.560220] Hardware name: Supermicro X9SCL/X9SCM/X9SCL/X9SCM, BIOS 2.0c 10/17/2013
> [157134.560649]  00000000000008be ffff8807a3c4f2f8 ffffffff8159c163 00000000000008be
> [157134.561087]  ffff8807a3c4f348 ffff8807a3c4f338 ffffffff8105598c ffff8807a3c4f388
> [157134.561537]  ffff8807e4c998c0 ffff88080ebf0000 ffff88080ebf0000 0000000000000000
> [157134.561993] Call Trace:
> [157134.562205]  [<ffffffff8159c163>] dump_stack+0x49/0x5e
> [157134.562422]  [<ffffffff8105598c>] warn_slowpath_common+0x8c/0xc0
> [157134.562642]  [<ffffffff81055a76>] warn_slowpath_fmt+0x46/0x50
> [157134.562864]  [<ffffffff814d2db3>] skb_warn_bad_offload+0xc3/0xd0
> [157134.563085]  [<ffffffff814d6b5e>] skb_checksum_help+0x19e/0x1c0
> [157134.563305]  [<ffffffff814d94fa>] dev_hard_start_xmit+0x4ba/0x5e0
> [157134.563526]  [<ffffffff814fa70e>] sch_direct_xmit+0xfe/0x1d0
> [157134.563747]  [<ffffffff814d97f5>] __dev_queue_xmit+0x1d5/0x4b0
> [157134.563971]  [<ffffffffa0530550>] ? deliver_clone+0x60/0x60 [bridge]
> [157134.564192]  [<ffffffff814d9af0>] dev_queue_xmit+0x10/0x20
> [157134.564413]  [<ffffffffa05305e2>] br_dev_queue_push_xmit+0x92/0xd0 [bridge]
> [157134.564638]  [<ffffffffa0530af2>] br_forward_finish+0x22/0x60 [bridge]
> [157134.564863]  [<ffffffffa0530b83>] __br_deliver+0x53/0x120 [bridge]
> [157134.565085]  [<ffffffffa0530cc1>] br_deliver+0x71/0x80 [bridge]
> [157134.565305]  [<ffffffffa052ea74>] br_dev_xmit+0x1c4/0x280 [bridge]
> [157134.565529]  [<ffffffffa0530550>] ? deliver_clone+0x60/0x60 [bridge]
> [157134.565770]  [<ffffffff814d93a5>] dev_hard_start_xmit+0x365/0x5e0
> [157134.565991]  [<ffffffff814d9927>] __dev_queue_xmit+0x307/0x4b0
> [157134.566212]  [<ffffffff814d9af0>] dev_queue_xmit+0x10/0x20
> [157134.566431]  [<ffffffffa05539dd>] vlan_dev_hard_start_xmit+0x9d/0x130 [8021q]
> [157134.566852]  [<ffffffff814d93a5>] dev_hard_start_xmit+0x365/0x5e0
> [157134.567064]  [<ffffffff814d9927>] __dev_queue_xmit+0x307/0x4b0
> [157134.567277]  [<ffffffffa0530550>] ? deliver_clone+0x60/0x60 [bridge]
> [157134.567489]  [<ffffffff814d9af0>] dev_queue_xmit+0x10/0x20
> [157134.567701]  [<ffffffffa05305e2>] br_dev_queue_push_xmit+0x92/0xd0 [bridge]
> [157134.567916]  [<ffffffffa0530af2>] br_forward_finish+0x22/0x60 [bridge]
> [157134.568131]  [<ffffffffa0530b83>] __br_deliver+0x53/0x120 [bridge]
> [157134.568344]  [<ffffffffa0530cc1>] br_deliver+0x71/0x80 [bridge]
> [157134.568565]  [<ffffffffa052ea74>] br_dev_xmit+0x1c4/0x280 [bridge]
> [157134.568785]  [<ffffffff814d93a5>] dev_hard_start_xmit+0x365/0x5e0
> [157134.569005]  [<ffffffffa06796f1>] ? ipv4_confirm+0x71/0x100 [nf_conntrack_ipv4]
> [157134.569434]  [<ffffffff814d9927>] __dev_queue_xmit+0x307/0x4b0
> [157134.569653]  [<ffffffff8151a1d0>] ? ip_build_and_send_pkt+0x220/0x220
> [157134.569889]  [<ffffffff814d9af0>] dev_queue_xmit+0x10/0x20
> [157134.570107]  [<ffffffff8151a41e>] ip_finish_output+0x24e/0x490
> [157134.570331]  [<ffffffff8151a6b8>] ip_output+0x58/0x90
> [157134.570553]  [<ffffffff815197d2>] ? __ip_local_out+0xa2/0xb0
> [157134.570774]  [<ffffffff81519809>] ip_local_out+0x29/0x30
> [157134.570993]  [<ffffffff81519d24>] ip_queue_xmit+0x144/0x3d0
> [157134.571209]  [<ffffffff81531305>] tcp_transmit_skb+0x415/0x7d0
> [157134.571430]  [<ffffffff810bed6e>] ? getnstimeofday+0xe/0x30
> [157134.571646]  [<ffffffff81532d83>] tcp_write_xmit+0x1e3/0x600
> [157134.571868]  [<ffffffff815339e2>] __tcp_push_pending_frames+0x32/0xd0
> [157134.572088]  [<ffffffff81527f6c>] tcp_data_snd_check+0x3c/0xe0
> [157134.572308]  [<ffffffff8152df39>] tcp_rcv_established+0x339/0x610
> [157134.572527]  [<ffffffff8152f3cb>] ? tcp_init_tso_segs+0x3b/0x60
> [157134.572745]  [<ffffffff81538769>] tcp_v4_do_rcv+0x169/0x270
> [157134.572962]  [<ffffffff814c3a69>] __release_sock+0x79/0xf0
> [157134.573179]  [<ffffffff814c3b0e>] release_sock+0x2e/0x90
> [157134.573396]  [<ffffffff81524f95>] tcp_sendmsg+0x115/0xd80
> [157134.573615]  [<ffffffff8154d3c5>] inet_sendmsg+0x45/0xb0
> [157134.573851]  [<ffffffff81236c03>] ? selinux_socket_sendmsg+0x23/0x30
> [157134.574075]  [<ffffffff814bc8dd>] sock_aio_write+0xfd/0x110
> [157134.574292]  [<ffffffff81199cef>] do_sync_write+0x5f/0xa0
> [157134.574511]  [<ffffffff81199f5d>] ? rw_verify_area+0x5d/0xe0
> [157134.574726]  [<ffffffff8119bc51>] vfs_write+0x1d1/0x1e0
> [157134.574940]  [<ffffffff8119bd4a>] SyS_write+0x5a/0xd0
> [157134.575156]  [<ffffffff815abf79>] ia32_do_call+0x13/0x13
> [157134.575372] ---[ end trace 8fe338efa5e49e56 ]---
> 
> Server hardware is a Xeon E3-1230v2 with 10gbe bna card.
> 
> We are using the following configuration:
> 
> brtrunk0: a linux Ethernet bridge, mtu=9000
>    members: eth0 (10gbe bna adapter) attached to tagged vlans 10,102
>             vnet0 (linux kvm vm router interface) 
>   ip: none.  This bridge only carries tagged vlan frames.
> 
> vlanbr102: a linux Ethernet bridge, mtu=1500
>   members: brtrunk0.102 (vlan 102 from above)
>   ip: 192.168.102.201
>   Other VMs will ultimately join this bridge for the 102 network.
>   Currently this bridge is empty except for the hypervisor's IP.
> 
> Note that we had to "ethtool -K brtrunk0 txvlan off" because tagging of
> the linux bridge was not working with the default txvlan=on setting.
> (Is this a bug?)
> 	
> 
> The KVM virtual machine on vnet0 router with the interfaces: 
> 	eth0.10:  192.168.100.33/24  mtu=1500
> 	eth0.102: 192.168.102.110/24 mtu=1500
> 
> Performance between hosts on the 192.168.102.0/24 and 192.168.100.0/24
> networks is limited to about 150mbit (using iperf) for some reason,
> which is why we began investigating.  Ultimately this is what we wish to
> resolve.
> 
> Note that the iperf between VM and switched network on both sides
> approaches gigabit as expected, but routing across the vlan interfaces
> within the KVM instance is very slow.
> 
> Hypervisor to guest is very fast: 
> 192.168.102.201 to 192.168.102.110: [  3]  0.0-10.0 sec  36.5 GBytes  31.4 Gbits/sec
> 
> Tests from my workstation working outward:
> 1. 192.168.102.61 to 192.168.102.201: [  3]  0.0-10.0 sec  1.09 GBytes   940 Mbits/sec
> 2. 192.168.102.61 to 192.168.102.110: [  3]  0.0-10.0 sec  1.09 GBytes   939 Mbits/sec
> 3. 192.168.102.61 to 192.168.100.33 : [  3]  0.0-10.0 sec  1.09 GBytes   939 Mbits/sec
> 4. 192.168.102.61 to 192.168.100.22 : [  3]  0.0-10.0 sec   184 MBytes   154 Mbits/sec ?!?!
> 5. 192.168.100.33 to 192.168.100.22 : [  3]  0.0-10.0 sec  1.06 GBytes   913 Mbits/sec
> 
> 
> The 4th test is from an external host, through the KVM router VM, and to
> an other external host past the linux router vm (iptables is empty).
> This is the performance problem that we wish to resolve. 
> 
> Please let me know if you need any additional information or any other
> tests that you would like preformed.  This server is not yet in
> production.
> 
> -Eric
> 
> 
> 
> Other possibly useful information:
> 
> $ tracepath -n 192.168.100.22
>  1:  192.168.102.61    0.054ms pmtu 1500
>  1:  192.168.102.110   0.367ms 
>  1:  192.168.102.110   0.326ms 
>  2:  192.168.100.22    0.921ms reached
> 
> [root@hv2 ~]# ethtool -i eth0
> driver: bna
> version: 3.2.23.0
> firmware-version: 3.2.3.0
> bus-info: 0000:02:00.2
> supports-statistics: yes
> supports-test: no
> supports-eeprom-access: yes
> supports-register-dump: no
> supports-priv-flags: no
> 
> [root@hv2 ~]# ethtool -c eth0
> Coalesce parameters for eth0:
> Adaptive RX: on  TX: off
> stats-block-usecs: 0
> sample-interval: 0
> pkt-rate-low: 0
> pkt-rate-high: 0
> 
> rx-usecs: 60
> rx-frames: 0
> rx-usecs-irq: 0
> rx-frames-irq: 0
> 
> tx-usecs: 100
> tx-frames: 12
> tx-usecs-irq: 0
> tx-frames-irq: 0
> 
> rx-usecs-low: 0
> rx-frame-low: 0
> tx-usecs-low: 0
> tx-frame-low: 0
> 
> rx-usecs-high: 0
> rx-frame-high: 0
> tx-usecs-high: 0
> tx-frame-high: 0
> 
> [root@hv2 ~]# ethtool -k eth0
> Features for eth0:
> rx-checksumming: on
> tx-checksumming: on
> 	tx-checksum-ipv4: on
> 	tx-checksum-ip-generic: off [fixed]
> 	tx-checksum-ipv6: on
> 	tx-checksum-fcoe-crc: off [fixed]
> 	tx-checksum-sctp: off [fixed]
> scatter-gather: on
> 	tx-scatter-gather: on
> 	tx-scatter-gather-fraglist: off [fixed]
> tcp-segmentation-offload: on
> 	tx-tcp-segmentation: on
> 	tx-tcp-ecn-segmentation: off [fixed]
> 	tx-tcp6-segmentation: on
> udp-fragmentation-offload: off [fixed]
> generic-segmentation-offload: on
> generic-receive-offload: on
> large-receive-offload: off [fixed]
> rx-vlan-offload: on
> tx-vlan-offload: on
> ntuple-filters: off [fixed]
> receive-hashing: off [fixed]
> highdma: on [fixed]
> rx-vlan-filter: on [fixed]
> vlan-challenged: off [fixed]
> tx-lockless: off [fixed]
> netns-local: off [fixed]
> tx-gso-robust: off [fixed]
> tx-fcoe-segmentation: off [fixed]
> tx-gre-segmentation: off [fixed]
> tx-ipip-segmentation: off [fixed]
> tx-sit-segmentation: off [fixed]
> tx-udp_tnl-segmentation: off [fixed]
> tx-mpls-segmentation: off [fixed]
> fcoe-mtu: off [fixed]
> tx-nocache-copy: off
> loopback: off [fixed]
> rx-fcs: off [fixed]
> rx-all: off [fixed]
> tx-vlan-stag-hw-insert: off [fixed]
> rx-vlan-stag-hw-parse: off [fixed]
> rx-vlan-stag-filter: off [fixed]
> l2-fwd-offload: off [fixed]
> 
> [root@hv2 ~]# ethtool -k brtrunk0
> Features for brtrunk0:
> rx-checksumming: off [fixed]
> tx-checksumming: on
> 	tx-checksum-ipv4: off [fixed]
> 	tx-checksum-ip-generic: on
> 	tx-checksum-ipv6: off [fixed]
> 	tx-checksum-fcoe-crc: off [fixed]
> 	tx-checksum-sctp: off [fixed]
> scatter-gather: on
> 	tx-scatter-gather: on
> 	tx-scatter-gather-fraglist: on
> tcp-segmentation-offload: on
> 	tx-tcp-segmentation: on
> 	tx-tcp-ecn-segmentation: on
> 	tx-tcp6-segmentation: on
> udp-fragmentation-offload: on
> generic-segmentation-offload: on
> generic-receive-offload: on
> large-receive-offload: off [fixed]
> rx-vlan-offload: off [fixed]
> tx-vlan-offload: off
> ntuple-filters: off [fixed]
> receive-hashing: off [fixed]
> highdma: on
> rx-vlan-filter: off [fixed]
> vlan-challenged: off [fixed]
> tx-lockless: on [fixed]
> netns-local: on [fixed]
> tx-gso-robust: off [requested on]
> tx-fcoe-segmentation: off [requested on]
> tx-gre-segmentation: on
> tx-ipip-segmentation: on
> tx-sit-segmentation: on
> tx-udp_tnl-segmentation: on
> tx-mpls-segmentation: on
> fcoe-mtu: off [fixed]
> tx-nocache-copy: off
> loopback: off [fixed]
> rx-fcs: off [fixed]
> rx-all: off [fixed]
> tx-vlan-stag-hw-insert: off [fixed]
> rx-vlan-stag-hw-parse: off [fixed]
> rx-vlan-stag-filter: off [fixed]
> l2-fwd-offload: off [fixed]
> 
> --
> eWheeler Inc.
> 
> 
> 
> 
> 
> 
> --
> To unsubscribe from this list: send the line "unsubscribe netdev" in
> the body of a message to majordomo@...r.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Powered by blists - more mailing lists