Message-ID: <CANn89iKRXpKkCzRw_7+VyG6jD2Tm5VUPQ-0bhQKUwh2sgzJZuA@mail.gmail.com>
Date: Thu, 23 Oct 2025 22:29:01 -0700
From: Eric Dumazet <edumazet@...gle.com>
To: Christoph Schwarz <cschwarz@...sta.com>
Cc: Neal Cardwell <ncardwell@...gle.com>, netdev@...r.kernel.org
Subject: Re: TCP sender stuck despite receiving ACKs from the peer

On Thu, Oct 23, 2025 at 3:52 PM Christoph Schwarz <cschwarz@...sta.com> wrote:
>
> On 10/3/25 18:24, Neal Cardwell wrote:
> [...]
> > Thanks for the report!
> >
> > A few thoughts:
> >
> [...]
> >
> > (2) After that, would it be possible to try this test with a newer
> > kernel? You mentioned this is with kernel version 5.10.165, but that's
> > more than 2.5 years old at this point, and it's possible the bug has
> > been fixed since then.  Could you please try this test with the newest
> > kernel that is available in your distribution? (If you are forced to
> > use 5.10.x on your distribution, note that even with 5.10.x there is
> > v5.10.245, which was released yesterday.)
> >
> > (3) If this bug is still reproducible with a recent kernel, would it
> > be possible to gather .pcap traces from both client and server,
> > including SYN and SYN/ACK? Sometimes it can be helpful to see the
> > perspective of both ends, especially if there are middleboxes
> > manipulating the packets in some way.
> >
> > Thanks!
> >
> > Best regards,
> > neal
>
> Hi,
>
> I want to give an update as we made some progress.
>
> We tried with the 6.12.40 kernel, but it was much harder to reproduce
> and we were not able to do a successful packet capture and reproduction
> at the same time. So we went back to 5.10.165, added more tracing and
> eventually figured out how the TCP connection got into the bad state.
>
> This is a backtrace from the TCP stack calling down to the device driver:
>   => fdev_tx    // ndo_start_xmit hook of a proprietary device driver
>   => dev_hard_start_xmit
>   => sch_direct_xmit
>   => __qdisc_run
>   => __dev_queue_xmit
>   => vlan_dev_hard_start_xmit
>   => dev_hard_start_xmit
>   => __dev_queue_xmit
>   => ip_finish_output2
>   => __ip_queue_xmit
>   => __tcp_transmit_skb
>   => tcp_write_xmit
>
> tcp_write_xmit sends segments of 65160 bytes. Due to an MSS of 1448,
> they get broken down into 45 packets of 1448 bytes each.

So the driver does not support TSO ? Quite odd in 2025...

One thing you want is to make sure your vlan device (the one without a
Qdisc on it)
advertizes tso support.

ethtool -k vlan0


> These 45
> packets eventually reach dev_hard_start_xmit, which is a simple loop
> forwarding packets one by one. When the problem occurs, we see that
> dev_hard_start_xmit transmits the initial N packets successfully, but
> the remaining 45-N ones fail with error code 1. The loop runs to
> completion and does not break.
>
> The error code 1 from dev_hard_start_xmit gets returned through the call
> stack up to tcp_write_xmit, which treats this as error and breaks its
> own loop without advancing snd_nxt:
>
>                 if (unlikely(tcp_transmit_skb(sk, skb, 1, gfp)))
>                         break; // <<< breaks here
>
> repair:
>                 /* Advance the send_head.  This one is sent out.
>                  * This call will increment packets_out.
>                  */
>                 tcp_event_new_data_sent(sk, skb);
>
>  From packet captures we can prove that the 45 packets show up on the
> kernel device on the sender. In addition, the first N of those 45
> packets show up on the kernel device on the peer. The connection is now
> in the problem state where the peer is N packets ahead of the sender and
> the sender thinks that it never those packets, leading to the problem as
> described in my initial mail.
>
> Furthermore, we noticed that the N-45 missing packets show up as drops
> on the sender's kernel device:
>
> vlan0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
>          inet 127.2.0.1  netmask 255.255.255.0  broadcast 0.0.0.0
>          [...]
>          TX errors 0  dropped 36 overruns 0  carrier 0  collisions 0
>
> This device is a vlan device stacked on another device like this:
>
> 49: vlan0@...ent: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> noqueue state UP mode DEFAULT group default qlen 1000
>      link/ether 02:1c:a7:00:00:01 brd ff:ff:ff:ff:ff:ff
> 3: parent: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 10000 qdisc prio state
> UNKNOWN mode DEFAULT group default qlen 1000
>      link/ether xx:xx:xx:xx:xx:xx brd ff:ff:ff:ff:ff:ff
>
> Eventually packets need to go through the device driver, which has only
> a limited number of TX buffers. The driver implements flow control: when
> it is about to exhaust its buffers, it stops TX by calling
> netif_stop_queue. Once more buffers become available again, it resumes
> TX by calling netif_wake_queue. From packet counters we can tell that
> this is happening frequently.
>
> At this point we suspected "qdisc noqueue" to be a factor, and indeed,
> after adding a queue to vlan0 the problem no longer happened, although
> there are still TX drops on the vlan0 device.
>
> Missing queue or not, we think there is a disconnect between the device
> driver API and the TCP stack. The device driver API only allows
> transmitting packets one by one (ndo_start_xmit). The TCP stack operates
> on larger segments that is breaks down into smaller pieces
> (tcp_write_xmit / __tcp_transmit_skb). This can lead to a classic "short
> write" condition which the network stack doesn't seem to handle well in
> all cases.
>
> Appreciate you comments,

Very nice analysis, very much appreciated.

I think the issue here is that __tcp_transmit_skb() trusts the return
of icsk->icsk_af_ops->queue_xmit()

An error means : the packet was _not_ sent at all.

Here, it seems that the GSO layer returns an error, even if some
segments were sent.
This needs to be confirmed and fixed, but in the meantime, make sure
vlan0 has TSO support.
It will also be more efficient to segment (if you ethernet device has
no TSO capability) at the last moment,
because all the segments will be sent in  the described scenario
thanks to qdisc requeues.

Powered by blists - more mailing lists