netdev - Re: Spurious TCP retransmissions on ack vs kfree

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <20210225191552.19b36496@kicinski-fedora-pc1c0hjn.dhcp.thefacebook.com>
Date:   Thu, 25 Feb 2021 19:15:52 -0800
From:   Jakub Kicinski <kuba@...nel.org>
To:     "netdev@...r.kernel.org" <netdev@...r.kernel.org>
Cc:     Eric Dumazet <edumazet@...gle.com>,
        Soheil Hassas Yeganeh <soheil@...gle.com>,
        Neal Cardwell <ncardwell@...gle.com>,
        Yuchung Cheng <ycheng@...gle.com>
Subject: Re: Spurious TCP retransmissions on ack vs kfree_skb reordering

On Thu, 25 Feb 2021 15:25:15 -0800 Jakub Kicinski wrote:
> Hi!
> 
> We see large (4-8x) increase of what looks like TCP RTOs after rising 
> the Tx coalescing above Rx coalescing timeout.
> 
> Quick tracing of the events seems to indicate that the data has already
> been acked when we enter tcp:tcp_retransmit_skb:

Seems like I'm pretty lost here and the tcp:tcp_retransmit_skb events
are less spurious than I thought. Looking at some tcpdump traces we see:

0.045277 IP6 A > B: Flags [SEW], seq 2248382925:2248383296, win 61920, options [mss 1440,sackOK,TS val 658870494 ecr 0,nop,wscale 11], length 371

0.045348 IP6 B > A: Flags [S.E], seq 961169456, ack 2248382926, win 65535, options [mss 1440,sackOK,TS val 883864022 ecr 658870494,nop,wscale 9], length 0

0.045369 IP6 A > B: Flags [P.], seq 1:372, ack 1, win 31, options [nop,nop,TS val 658870494 ecr 883864022], length 371


So looks potentially TFO related?

To try to count timeouts I run:

bpftrace --btf -e 'tracepoint:tcp:tcp_retransmit_skb { 
  $icsk = (struct inet_connection_sock *)args->skaddr; 
  if ($icsk->icsk_ca_state != 4) { return; } 
  if ($icsk->icsk_pending)       { return; }

  printf(...);
}'

At tx-usecs coalescing of 25us I see 0 of those events.
At 100us there is a few.
At 200us there is a lot.