netdev - xfrm regression: TCP MSS calculation broken by commit b515d263, results in TCP stall

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite for Android: free password hash cracker in your pocket

[<prev] [next>] [thread-next>] [day] [month] [year] [list]

Message-ID: <20220114173133.tzmdm2hy4flhblo3@dwarf.suse.cz>
Date:   Fri, 14 Jan 2022 18:31:33 +0100
From:   Jiri Bohac <jbohac@...e.cz>
To:     Sabrina Dubroca <sd@...asysnail.net>,
        Steffen Klassert <steffen.klassert@...unet.com>,
        Herbert Xu <herbert@...dor.apana.org.au>,
        "David S. Miller" <davem@...emloft.net>
Cc:     Mike Maloney <maloneykernel@...il.com>,
        Eric Dumazet <eric.dumazet@...il.com>, netdev@...r.kernel.org
Subject: xfrm regression: TCP MSS calculation broken by commit b515d263,
 results in TCP stall

Hello,

our customer found that commit
b515d2637276a3810d6595e10ab02c13bfd0b63a ("xfrm: xfrm_state_mtu
should return at least 1280 for ipv6") in v5.14 breaks the TCP
MSS calculation in ipsec transport mode, resulting complete
stalls of TCP connections. This happens when the (P)MTU is 1280
or slighly larger.

The desired formula for the MSS is:
	MSS = (MTU - ESP_overhead) - IP header - TCP header

However, the above patch clamps the (MTU - ESP_overhead) to a
minimum of 1280, turning the formula into
	MSS = max(MTU - ESP overhead, 1280) -  IP header - TCP header

With the (P)MTU near 1280, the calculated MSS is too large and
the resulting TCP packets never make it to the destination
because they are over the actual PMTU.

Trying to fix the exact same problem as the broken patch, which I
was unaware of, I sent an alternative patch in this thread of
April 2021:
https://lore.kernel.org/netdev/20210429170254.5grfgsz2hgy2qjhk@dwarf.suse.cz/
(note the v1 is broken and followed by v2!)

In that thread I also found other problems with
b515d2637276a3810d6595e10ab02c13bfd0b63a - in tunnel mode it
causes suboptimal double fragmentation:
https://lore.kernel.org/netdev/20210429202529.codhwpc7w6kbudug@dwarf.suse.cz/

I therefore propose to revert
b515d2637276a3810d6595e10ab02c13bfd0b63a and
apply the v2 version of my patch, which I'll re-send in reply to
this e-mail.

Thanks,

-- 
Jiri Bohac <jbohac@...e.cz>
SUSE Labs, Prague, Czechia