Date:	Fri, 19 Feb 2016 18:18:59 -0800
From:	Jesse Gross <jesse@...nel.org>
To:	Tom Herbert <tom@...bertland.com>
Cc:	Alex Duyck <aduyck@...antis.com>,
	Linux Kernel Network Developers <netdev@...r.kernel.org>,
	David Miller <davem@...emloft.net>,
	Alexander Duyck <alexander.duyck@...il.com>
Subject: Re: [net-next PATCH 0/2] GENEVE/VXLAN: Enable outer Tx checksum by default

On Fri, Feb 19, 2016 at 4:14 PM, Tom Herbert <tom@...bertland.com> wrote:
> On Fri, Feb 19, 2016 at 4:08 PM, Jesse Gross <jesse@...nel.org> wrote:
>> On Fri, Feb 19, 2016 at 3:10 PM, Alex Duyck <aduyck@...antis.com> wrote:
>>> On Fri, Feb 19, 2016 at 1:53 PM, Jesse Gross <jesse@...nel.org> wrote:
>>>> On Fri, Feb 19, 2016 at 11:26 AM, Alexander Duyck <aduyck@...antis.com> wrote:
>>>>> This patch series makes it so that we enable the outer Tx checksum for IPv4
>>>>> tunnels by default.  This makes the behavior consistent with how we were
>>>>> handling this for IPv6.  In addition I have updated the internal flags for
>>>>> these tunnels so that we use a ZERO_CSUM_TX flag for IPv4 which should
>>>>> match up will with the ZERO_CSUM6_TX flag which was already in use for
>>>>> IPv6.
>>>>>
>>>>> For most network devices this should be a net gain in terms of performance
>>>>> as having the outer header checksum present allows for devices to report
>>>>> CHECKSUM_UNNECESSARY which we can then convert to CHECKSUM_COMPLETE in order
>>>>> to determine if the inner header checksum is valid.
>>>>>
>>>>> Below is some data I collected with ixgbe with an X540 that demonstrates
>>>>> this.  I located two PFs connected back to back in two different name
>>>>> spaces and then setup a pair of tunnels on each, one with checksum enabled
>>>>> and one without.
>>>>>
>>>>> Recv   Send    Send                          Utilization
>>>>> Socket Socket  Message  Elapsed              Send
>>>>> Size   Size    Size     Time     Throughput  local
>>>>> bytes  bytes   bytes    secs.    10^6bits/s  % S
>>>>>
>>>>> noudpcsum:
>>>>>  87380  16384  16384    30.00      8898.67   12.80
>>>>> udpcsum:
>>>>>  87380  16384  16384    30.00      9088.47   5.69
>>>>>
>>>>> The one spot where this may cause a performance regression is if the
>>>>> environment contains devices that can parse the inner headers and a device
>>>>> supports NETIF_F_GSO_UDP_TUNNEL but not NETIF_F_GSO_UDP_TUNNEL_CSUM.  In
>>>>> the case of such a device we have to fall back to using GSO to segment the
>>>>> tunnel instead of TSO and as a result we may take a performance hit as seen
>>>>> below with i40e.
>>>>
>>>> Do you have any numbers from 40G links? Obviously, at 10G the links
>>>> are basically saturated and while I can see a difference in the
>>>> utilization rate, I suspect that the change will be much more apparent
>>>> at higher speeds.
>>>
>>> Unfortunately I don't have any true 40G links to test with.  The
>>> closest I can get is to run PF to VF on an i40e.  Running that I have
>>> seen the numbers go from about 20Gb/s to 15Gb/s with almost all the
>>> difference being related to the fact that we are having to
>>> allocate/free more skbs and make more trips through the
>>> i40e_lan_xmit_frame function resulting in more descriptors.
>>
>> OK, I guess that is more or less in line with what I would expect off
>> the top my head. There is a reasonably significant drop in the worst
>> case.
>>
>>>> I'm concerned about the drop in performance for devices that currently
>>>> support offloads (almost none of which expose
>>>> NETIF_F_GSO_UDP_TUNNEL_CSUM as a feature). Presumably the people that
>>>> care most about tunnel performance are the ones that already have
>>>> these NICs and will be the most impacted by the drop.
>>>
>>> The problem is being able to transmit fast is kind of pointless if the
>>> receiving end cannot handle it.  We hadn't gotten around to really
>>> getting the Rx checksum bits working until the 3.18 kernel which I
>>> don't suspect many people are running so at this point messing with
>>> the TSO bits isn't really making much of a difference.  Then on top of
>>> that most devices have certain limitations on how many ports they can
>>> handle and such.  I know the i40e is supposed to support something
>>> like 10 port numbers, but the fm10k and ixgbe are limited to one port
>>> as I recall.  So this whole thing is already really brittle as it is.
>>> My goal with this change is to make the behavior more consistent
>>> across the board.
>>
>> That's true to some degree but there are certainly plenty of cases
>> where TSO makes a difference - lower CPU usage, transmitting to
>> multiple receivers, people will upgrade their kernels, etc. It's
>> clearly good to make things more consistent but hopefully not by
>> reducing existing performance. :)
>>
>>>> My hope is that we can continue to use TSO on devices that only
>>>> support NETIF_F_GSO_UDP_TUNNEL. The main problem is that the UDP
>>>> length field may vary across segments. However, in practice this is
>>>> the only on the final segment and only in cases where the total length
>>>> is not a multiple of the MSS. If we could detect cases where those
>>>> conditions are met, we could continue to use TSO with the UDP checksum
>>>> field pre-populated. A possible step even further would be to break
>>>> off the final segment into a separate packet to make things conform if
>>>> necessary. This would avoid a performance regression and I think make
>>>> this more palatable to a lot of people.
>>>
>>> I think Tom and I had discussed this possibility a bit at netconf.
>>> The GSO logic is something I planned on looking at over the next
>>> several weeks as I suspect there is probably room for improvement
>>> there.
>>
>> That sounds great.
>>
>>>>> I also haven't investigated the effect this will have on OVS.  However I
>>>>> suspect the impact should be minimal as the worst case scenario should be
>>>>> that Tx checksumming will become enabled by default which should be
>>>>> consistent with the existing behavior for IPv6.
>>>>
>>>> I don't think that it should cause any problems.
>>>
>>> Good to hear.
>>>
>>> Do you know if OVS has some way to control the VXLAN configuration so
>>> that it could disable Tx checksums?  If so that would probably be a
>>> good way to address the 40G issues assuming someone is running an
>>> environment hat had nothing but NICs that can support the TSO and Rx
>>> checksum on inner headers.
>>
>> Yes - OVS can control tx checksums on a per-endpoint basis (actually,
>> rx checksum present requirements as well though it's not exposed to
>> the user at the moment). If you had the information then you could
>> optimize what to use in an environment of, say, hypervisors and
>> hardware switches.
>>
>> However, it's certainly possible that you have a mixed set of NICs
>> such as an encap aware NIC on the transmit side and non-aware on the
>> receive side. In that case, both possible checksum settings penalize
>> somebody: off (lose GRO on receiver), on (lose TSO on sender assuming
>> no support for NETIF_F_GSO_UDP_TUNNEL_CSUM). That's why I think it's
>> important to be able to use encap TSO with local checksum to avoid
>> these bad tradeoffs, not to mention being cleaner.
>
> By "local checksum" do you mean LCO?

Yes, that's what I meant.

> Seems like we should be able to
> get that to work with NETIF_F_GSO_TUNNEL_CSUM.

I assume you mean NETIF_F_GSO_TUNNEL (no _CSUM)?

Powered by blists - more mailing lists