netdev - Re: TCP window auto-tuning sub-optimal in GRE tunnel

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <1432573139.4060.167.camel@edumazet-glaptop2.roam.corp.google.com>
Date:	Mon, 25 May 2015 09:58:59 -0700
From:	Eric Dumazet <eric.dumazet@...il.com>
To:	"John A. Sullivan III" <jsullivan@...nsourcedevel.com>
Cc:	netdev@...r.kernel.org
Subject: Re: TCP window auto-tuning sub-optimal in GRE tunnel

On Mon, 2015-05-25 at 11:42 -0400, John A. Sullivan III wrote:
> Hello, all.  I hope this is the correct list for this question.  We are
> having serious problems on high BDP networks using GRE tunnels.  Our
> traces show it to be a TCP Window problem.  When we test without GRE,
> throughput is wire speed and traces show the window size to be 16MB
> which is what we configured for r/wmem_max and tcp_r/wmem.  When we
> switch to GRE, we see over a 90% drop in throughput and the TCP window
> size seems to peak at around 500K.
> 
> What causes this and how can we get the GRE tunnels to use the max
> window size? Thanks - John

Hi John

Is it for a single flow or multiple ones ? Which kernel versions on
sender and receiver ? What is the nominal speed of non GRE traffic ?

What is the brand/model of receiving NIC  ? Is GRO enabled ?

It is possible receiver window is impacted because of GRE encapsulation
making skb->len/skb->truesize ratio a bit smaller, but not by 90%.

I suspect some more trivial issues, like receiver overwhelmed by the
extra load of GRE encapsulation.

1) Non GRE session

lpaa23:~# DUMP_TCP_INFO=1 ./netperf -H lpaa24 -Cc -t OMNI
OMNI Send TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to lpaa24.prod.google.com () port 0 AF_INET
tcpi_rto 201000 tcpi_ato 0 tcpi_pmtu 1500 tcpi_rcv_ssthresh 29200
tcpi_rtt 70 tcpi_rttvar 7 tcpi_snd_ssthresh 221 tpci_snd_cwnd 258
tcpi_reordering 3 tcpi_total_retrans 711
Local       Remote      Local  Elapsed Throughput Throughput  Local Local  Remote Remote Local   Remote  Service  
Send Socket Recv Socket Send   Time               Units       CPU   CPU    CPU    CPU    Service Service Demand   
Size        Size        Size   (sec)                          Util  Util   Util   Util   Demand  Demand  Units    
Final       Final                                             %     Method %      Method                          
1912320     6291456     16384  10.00   22386.89   10^6bits/s  1.20  S      2.60   S      0.211   0.456   usec/KB  

2) GRE session

lpaa23:~# DUMP_TCP_INFO=1 ./netperf -H 7.7.7.24 -Cc -t OMNI
OMNI Send TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 7.7.7.24 () port 0 AF_INET
tcpi_rto 201000 tcpi_ato 0 tcpi_pmtu 1500 tcpi_rcv_ssthresh 29200
tcpi_rtt 76 tcpi_rttvar 7 tcpi_snd_ssthresh 176 tpci_snd_cwnd 249
tcpi_reordering 3 tcpi_total_retrans 819
Local       Remote      Local  Elapsed Throughput Throughput  Local Local  Remote Remote Local   Remote  Service  
Send Socket Recv Socket Send   Time               Units       CPU   CPU    CPU    CPU    Service Service Demand   
Size        Size        Size   (sec)                          Util  Util   Util   Util   Demand  Demand  Units    
Final       Final                                             %     Method %      Method                          
1815552     6291456     16384  10.00   22420.88   10^6bits/s  1.01  S      3.44   S      0.177   0.603   usec/KB  


--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html