Message-ID: <CAGRyCJGRqJuh=d8AomhYx08bGuf4dcOJ8o0JrsqqUeVyK8SAcw@mail.gmail.com>
Date: Fri, 7 Nov 2025 14:34:15 +0100
From: Daniele Palmas <dnlplm@...il.com>
To: Simon Schippers <simon.schippers@...dortmund.de>
Cc: Eric Dumazet <edumazet@...gle.com>, oneukum@...e.com, andrew+netdev@...n.ch, 
	davem@...emloft.net, kuba@...nel.org, pabeni@...hat.com, 
	netdev@...r.kernel.org, linux-usb@...r.kernel.org, 
	linux-kernel@...r.kernel.org
Subject: Re: [PATCH net-next v1 0/1] usbnet: Add support for Byte Queue Limits (BQL)

Hi Simon,

Il giorno gio 6 nov 2025 alle ore 11:00 Simon Schippers
<simon.schippers@...dortmund.de> ha scritto:
>
> On 11/6/25 09:38, Daniele Palmas wrote:
> > Hi Simon,
> >
> > Il giorno mer 5 nov 2025 alle ore 12:05 Simon Schippers
> > <simon.schippers@...dortmund.de> ha scritto:
> >>
> >> On 11/5/25 11:35, Daniele Palmas wrote:
> >>> Hello Simon,
> >>>
> >>> Il giorno mer 5 nov 2025 alle ore 11:40 Simon Schippers
> >>> <simon.schippers@...dortmund.de> ha scritto:
> >>>>
> >>>> On 11/4/25 18:02, Eric Dumazet wrote:
> >>>>> On Tue, Nov 4, 2025 at 8:14 AM Simon Schippers
> >>>>> <simon.schippers@...dortmund.de> wrote:
> >>>>>>
> >>>>>> During recent testing, I observed significant latency spikes when using
> >>>>>> Quectel 5G modems under load. Investigation revealed that the issue was
> >>>>>> caused by bufferbloat in the usbnet driver.
> >>>>>>
> >>>>>> In the current implementation, usbnet uses a fixed tx_qlen of:
> >>>>>>
> >>>>>> USB2: 60 * 1518 bytes = 91.08 KB
> >>>>>> USB3: 60 * 5 * 1518 bytes = 454.80 KB
> >>>>>>
> >>>>>> Such large transmit queues can be problematic, especially for cellular
> >>>>>> modems. For example, with a typical celluar link speed of 10 Mbit/s, a
> >>>>>> fully occupied USB3 transmit queue results in:
> >>>>>>
> >>>>>> 454.80 KB / (10 Mbit/s / 8 bit/byte) = 363.84 ms
> >>>>>>
> >>>>>> of additional latency.
> >>>>>
> >>>>> Doesn't 5G need to push more packets to the driver to get good aggregation ?
> >>>>>
> >>>>
> >>>> Yes, but not 455 KB for low speeds. 5G requires a queue of a few ms to
> >>>> aggregate enough packets for a frame but not of several hundred ms as
> >>>> calculated in my example. And yes, there are situations where 5G,
> >>>> especially FR2 mmWave, reaches Gbit/s speeds where a big queue is
> >>>> required. But the dynamic queue limit approach of BQL should be well
> >>>> suited for these varying speeds.
> >>>>
> >>>
> >>> out of curiosity, related to the test with 5G Quectel, did you test
> >>> enabling aggregation through QMAP (kernel module rmnet) or simply
> >>> qmi_wwan raw_ip ?
> >>>
> >>> Regards,
> >>> Daniele
> >>>
> >>
> >> Hi Daniele,
> >>
> >> I simply used qmi_wwan. I actually never touched rmnet before.
> >> Is the aggregation through QMAP what you and Eric mean with aggregation?
> >> Because then I misunderstood it, because I was thinking about aggregating
> >> enough (and not too many) packets in the usbnet queue.
> >>
> >
> > I can't speak for Eric, but, yes, that is what I meant for
> > aggregation, this is the common way those high-cat modems are used:
>
> Hi Daniele,
>
> I think I *really* have to take a look at rmnet and aggregation through
> QMAP for future projects :)
>
> > it's not clear to me if the change you are proposing could have any
> > impact when rmnet is used, that's why I was asking the test
> > conditions.
> >
> > Thanks,
> > Daniele
> >
>
> This patch has an impact on the underlying USB physical transport of
> rmnet. From my understanding, the call stack is as follows:
>
> rmnet_map_tx_aggregate or rmnet_send_skb
>
> |
> | Calling dev_queue_xmit(skb)
> V
>
> qmi_wwan used for USB modem
>
> |
> |  ndo_start_xmit(skb, net) is called
> V
>
> usbnet_start_xmit is executed where the size of the internal queue is
> dynamically changed using the Byte Queue Limits algorithm by this patch.
>
> Correct me if I am wrong, but I think in the end usbnet is used.
>

Exactly, I was just wondering if this patch had any effect on the
overall throughput performance once the aggregation is enabled.

Hopefully I'll be able to perform some tests once the patch is merged.

Thanks,
Daniele

> Thanks,
> Simon
>
> >> Thanks
> >>
> >>>>>>
> >>>>>> To address this issue, this patch introduces support for
> >>>>>> Byte Queue Limits (BQL) [1][2] in the usbnet driver. BQL dynamically
> >>>>>> limits the amount of data queued in the driver, effectively reducing
> >>>>>> latency without impacting throughput.
> >>>>>> This implementation was successfully tested on several devices as
> >>>>>> described in the commit.
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> Future work
> >>>>>>
> >>>>>> Due to offloading, TCP often produces SKBs up to 64 KB in size.
> >>>>>
> >>>>> Only for rates > 500 Mbit. After BQL, we had many more improvements in
> >>>>> the stack.
> >>>>> https://lwn.net/Articles/564978/
> >>>>>
> >>>>>
> >>>>
> >>>> I also saw these large SKBs, for example, for my USB2 Android tethering,
> >>>> which advertises a network speed of < 500 Mbit/s.
> >>>> I saw these large SKBs by looking at the file:
> >>>>
> >>>> cat /sys/class/net/INTERFACE/queues/tx-0/byte_queue_limits/inflight
> >>>>
> >>>> For UDP-only traffic, inflight always maxed out at MTU size.
> >>>>
> >>>> Thank you for your replies!
> >>>>
> >>>>>> To
> >>>>>> further decrease buffer bloat, I tried to disable TSO, GSO and LRO but it
> >>>>>> did not have the intended effect in my tests. The only dirty workaround I
> >>>>>> found so far was to call netif_stop_queue() whenever BQL sets
> >>>>>> __QUEUE_STATE_STACK_XOFF. However, a proper solution to this issue would
> >>>>>> be desirable.
> >>>>>>
> >>>>>> I also plan to publish a scientific paper on this topic in the near
> >>>>>> future.
> >>>>>>
> >>>>>> Thanks,
> >>>>>> Simon
> >>>>>>
> >>>>>> [1] https://medium.com/@tom_84912/byte-queue-limits-the-unauthorized-biography-61adc5730b83
> >>>>>> [2] https://lwn.net/Articles/469652/
> >>>>>>
> >>>>>> Simon Schippers (1):
> >>>>>>   usbnet: Add support for Byte Queue Limits (BQL)
> >>>>>>
> >>>>>>  drivers/net/usb/usbnet.c | 8 ++++++++
> >>>>>>  1 file changed, 8 insertions(+)
> >>>>>>
> >>>>>> --
> >>>>>> 2.43.0
> >>>>>>
> >>>>

Powered by blists - more mailing lists