netdev - Re: [PATCH net] ipv6: gro: flush instead of assuming different flows on hop

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite for Android: free password hash cracker in your pocket

[<prev] [next>] [<thread-prev] [day] [month] [year] [list]

Message-ID: <20220124160240.02a451bd@kicinski-fedora-PC1C0HJN.hsd1.ca.comcast.net>
Date:   Mon, 24 Jan 2022 16:02:40 -0800
From:   Jakub Kicinski <kuba@...nel.org>
To:     Eric Dumazet <edumazet@...gle.com>
Cc:     David Miller <davem@...emloft.net>,
        David Ahern <dsahern@...il.com>,
        Paolo Abeni <pabeni@...hat.com>,
        Herbert Xu <herbert@...dor.apana.org.au>,
        netdev <netdev@...r.kernel.org>,
        Yuchung Cheng <ycheng@...gle.com>,
        Neal Cardwell <ncardwell@...gle.com>
Subject: Re: [PATCH net] ipv6: gro: flush instead of assuming different
 flows on hop_limit mismatch

Sorry for the delay I had to do some homework and more tests.

On Fri, 21 Jan 2022 08:37:12 -0800 Eric Dumazet wrote:
> On Fri, Jan 21, 2022 at 7:15 AM Jakub Kicinski <kuba@...nel.org> wrote:
> > > We implemented SACK compress in TCP stack to avoid extra SACK being
> > > sent by the receiver
> > >
> > > We have an extension of this SACK compression for TCP flows terminated
> > > by Google servers,
> > > since modern TCP stacks do not need the old rule of TCP_FASTRETRANS_THRESH
> > > DUPACK to start retransmits.
> > >
> > > Something like this pseudo code:
> > >
> > > diff --git a/net/ipv4/tcp_input.c b/net/ipv4/tcp_input.c
> > > index dc49a3d551eb919baf5ad812ef21698c5c7b9679..d72554ab70fd2e16ed60dc78a905f4aa1414f8c9
> > > 100644
> > > --- a/net/ipv4/tcp_input.c
> > > +++ b/net/ipv4/tcp_input.c
> > > @@ -5494,7 +5494,8 @@ static void __tcp_ack_snd_check(struct sock *sk,
> > > int ofo_possible)
> > >         }
> > >         if (tp->dup_ack_counter < TCP_FASTRETRANS_THRESH) {
> > >                 tp->dup_ack_counter++;
> > > -               goto send_now;
> > > +               if (peer_is_using_old_rule_about_fastretrans(tp))
> > > +                       goto send_now;
> > >         }
> > >         tp->compressed_ack++;
> > >         if (hrtimer_is_queued(&tp->compressed_ack_timer))
> > >  
> >
> > Is this something we could upstream / test? peer_is_using.. does not
> > exist upstream.  
> 
> Sure, because we do not have a standardized way (at SYN SYNACK time)
> to advertise
> that the stack is not 10 years old.
> 
> This could be a per net-ns sysctl, or a per socket flag, or a per cgroup flag.
> 
> In our case, we do negotiate special TCP options, and allow these options
> only from internal communications.
> 
> (So we store this private bit in the socket itself)

This does not fix the problem, unfortunately. I still see TCP detecting
reordering based on SACK if re-transmits have higher TTL.

> > Coincidentally, speaking of sending SACKs, my initial testing was on
> > 5.12 kernels and there I saw what appeared to a lay person (me) like
> > missing ACKs. Receiver would receive segments:
> >
> > _AB_C_D_E
> >
> > where _ indicates loss. It'd SACK A, then generate the next SACK after E
> > (SACKing C D E), sender would rexmit A which makes receiver ACK all
> > the way to the end of B. Now sender thinks B arrived after CDE because
> > it was never sacked.
> >
> > Perhaps it was fixed by commit a29cb6914681 ("net: tcp better handling
> > of reordering then loss cases").. or it's a result of some out-of-tree
> > hack. I thought I'd mention it tho in case it immediately rings a bell
> > for anyone.  
> 
> Could all the missing SACK have been lost ?

I had tcpdump on both ends, but I can't repro any more with the GRO fix
applied. Maybe it was also related to that. Somehow.

> Writing a packetdrill test for this scenario should not be too hard.