netdev - Re: TCP and BBR: reproducibly low cwnd and bandwidth

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <CANn89iL+ykS1d10wPLHJiYsgZj0u1q8EqEOkbBuFtBkONT8GSg@mail.gmail.com>
Date:   Fri, 16 Feb 2018 14:59:52 -0800
From:   Eric Dumazet <edumazet@...gle.com>
To:     Oleksandr Natalenko <oleksandr@...alenko.name>
Cc:     Neal Cardwell <ncardwell@...gle.com>,
        Eric Dumazet <eric.dumazet@...il.com>,
        "David S. Miller" <davem@...emloft.net>,
        Netdev <netdev@...r.kernel.org>,
        Yuchung Cheng <ycheng@...gle.com>,
        Soheil Hassas Yeganeh <soheil@...gle.com>,
        Jerry Chu <hkchu@...gle.com>, Dave Taht <dave.taht@...il.com>
Subject: Re: TCP and BBR: reproducibly low cwnd and bandwidth

On Fri, Feb 16, 2018 at 2:50 PM, Oleksandr Natalenko
<oleksandr@...alenko.name> wrote:
> Hi.
>
> On pátek 16. února 2018 21:54:05 CET Eric Dumazet wrote:
>> /* snip */
>> Something fishy really :
>> /* snip */
>> Not only the receiver suddenly adds a 25 ms delay, but also note that
>> it acknowledges all prior segments (ack 112949), but with a wrong ecr
>> value ( 2327043753 )
>> instead of 2327043759
>> /* snip */
>
> Eric has encouraged me to look closer at what's there in the ethtool, and I've
> just had a free time to play with it. I've found out that enabling scatter-
> gather (ethtool -K enp3s0 sg on, it is disabled by default on both hosts)
> brings the throughput back to normal even with BBR+fq_codel.
>
> Whyyyy? What's the deal BBR has with sg?

Well, no effect  here on e1000e (1 Gbit) at least

# ethtool -K eth3 sg off
Actual changes:
scatter-gather: off
tx-scatter-gather: off
tcp-segmentation-offload: off
tx-tcp-segmentation: off [requested on]
tx-tcp6-segmentation: off [requested on]
generic-segmentation-offload: off [requested on]

# tc qd replace dev eth3 root pfifo_fast
# ./super_netperf 1 -H 7.7.7.84 -- -K cubic
    941
# ./super_netperf 1 -H 7.7.7.84 -- -K bbr
    941
# tc qd replace dev eth3 root fq
# ./super_netperf 1 -H 7.7.7.84 -- -K cubic
    941
# ./super_netperf 1 -H 7.7.7.84 -- -K bbr
    941
# tc qd replace dev eth3 root fq_codel
# ./super_netperf 1 -H 7.7.7.84 -- -K cubic
    941
# ./super_netperf 1 -H 7.7.7.84 -- -K bbr
    941
#