netdev - Re: Spurious TCP retransmissions on ack vs kfree

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <20210226080628.3e40e037@kicinski-fedora-pc1c0hjn.dhcp.thefacebook.com>
Date:   Fri, 26 Feb 2021 08:06:28 -0800
From:   Jakub Kicinski <kuba@...nel.org>
To:     Eric Dumazet <edumazet@...gle.com>
Cc:     "netdev@...r.kernel.org" <netdev@...r.kernel.org>,
        Soheil Hassas Yeganeh <soheil@...gle.com>,
        Neal Cardwell <ncardwell@...gle.com>,
        Yuchung Cheng <ycheng@...gle.com>
Subject: Re: Spurious TCP retransmissions on ack vs kfree_skb reordering

On Fri, 26 Feb 2021 11:41:22 +0100 Eric Dumazet wrote:
> On Fri, Feb 26, 2021 at 11:05 AM Eric Dumazet <edumazet@...gle.com> wrote:
> >
> > On Fri, Feb 26, 2021 at 4:15 AM Jakub Kicinski <kuba@...nel.org> wrote:  
> > >
> > > On Thu, 25 Feb 2021 15:25:15 -0800 Jakub Kicinski wrote:  
> > > > Hi!
> > > >
> > > > We see large (4-8x) increase of what looks like TCP RTOs after rising
> > > > the Tx coalescing above Rx coalescing timeout.
> > > >
> > > > Quick tracing of the events seems to indicate that the data has already
> > > > been acked when we enter tcp:tcp_retransmit_skb:  
> > >
> > > Seems like I'm pretty lost here and the tcp:tcp_retransmit_skb events
> > > are less spurious than I thought. Looking at some tcpdump traces we see:
> > >
> > > 0.045277 IP6 A > B: Flags [SEW], seq 2248382925:2248383296, win 61920, options [mss 1440,sackOK,TS val 658870494 ecr 0,nop,wscale 11], length 371
> > >
> > > 0.045348 IP6 B > A: Flags [S.E], seq 961169456, ack 2248382926, win 65535, options [mss 1440,sackOK,TS val 883864022 ecr 658870494,nop,wscale 9], length 0  
> >
> > The SYNACK does not include the prior payload.
> >  
> > > 0.045369 IP6 A > B: Flags [P.], seq 1:372, ack 1, win 31, options [nop,nop,TS val 658870494 ecr 883864022], length 371  
> >
> > So this rtx is not spurious.
> >
> > However in your prior email you wrote :
> >
> > bytes_in:      0
> > bytes_out:   742
> > bytes_acked: 742
> >
> > Are you sure that at the time of the retransmit, bytes_acked was 742 ?
> > I do not see how this could happen.  
> 
> Yes, this packetdrill test confirms TCP INFO stuff seems correct .

Looks like it's TcpExtTCPSpuriousRtxHostQueues - the TFO fails as it
might, but at the time the syn is still not kfree_skb()d because of 
the IRQ coalescing settings, so __tcp_retransmit_skb() returns -EBUSY
and we have to wait for a timeout.

Credit to Neil Spring @FB for figuring it out.