| lists.openwall.net | lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC | |
|
Open Source and information security mailing list archives
| ||
|
Message-ID: <26568720.1r3eYUQgxm@benoit.monin>
Date: Fri, 20 Sep 2024 14:10:17 +0200
From: Benoît Monin <benoit.monin@....fr>
To: "David S. Miller" <davem@...emloft.net>, David Ahern <dsahern@...nel.org>,
Eric Dumazet <edumazet@...gle.com>, Jakub Kicinski <kuba@...nel.org>,
Paolo Abeni <pabeni@...hat.com>
Cc: netdev@...r.kernel.org, linux-kernel@...r.kernel.org
Subject: Re: PROBLEM: invalid udp checksum with ip6gre-in-udp
Hi,
12/09/2024 Benoît Monin :
> Hi again,
>
> 05/09/2024 Benoît Monin :
> > Hi all,
> >
> > I am having issue with GRE-in-UDP (GRE with fou encapsulation) over
> > IPv6: the outer UDP checksum is only valid if an inner checksum is
> > present and valid. This problem is only present over IPv6, not IPv4,
> > and reproducible with different kernel versions and cpu architecture.
> >
> > Here is the test setup I used:
> >
> > +----------+ +------------+
> > | | fd00::2/64 fd00::1/64 | |
> > | tester | 10.0.0.33/24 10.0.0.11/24 | DUT |
> > | |------------------------------------| |
> > +----------+ +------------+
> >
> > Two machines are connected with an ethernet cable, and each side is
> > setup with an ipv4 and an ipv6 address.
> >
> > On the device under test, two GRE-in-UDP tunnels are setup, one over
> > ipv6 and one over ipv4, aimed at the tester. Each tunnel is configured
> > with an ipv4 address:
> >
> > modprobe -a fou fou6
> >
> > ip link add gre6 type ip6gre local fd00::1 remote fd00::2 encap fou encap-sport 1234 encap-dport 4567
> > ip link set up gre6
> > ip address add 172.20.6.1/24 dev gre6
> >
> > ip link add gre4 type gre local 10.0.0.11 remote 10.0.0.33 encap fou encap-sport 3456 encap-dport 6789 encap-csum
> > ip link set up gre4
> > ip address add 172.20.4.1/24 dev gre4
> >
> > On the tester, no setup is done, it is only running tcpdump to capture
> > the traffic emitted by the DUT.
> >
> > The following commands are used on the device under test to send
> > packets via the tunnels:
> >
> > ping -c1 -W0.1 172.20.6.2
> > ./send_udp 172.20.6.2 5555 "ip6gre-in-udp with inner udp csum"
> > SO_NO_CHECK=1 ./send_udp 172.20.6.2 5555 "ip6gre-in-udp without inner udp csum"
> > ping -c1 -W0.1 172.20.4.2
> > ./send_udp 172.20.4.2 5555 "gre-in-udp with inner udp csum"
> > SO_NO_CHECK=1 ./send_udp 172.20.4.2 5555 "gre-in-udp without inner udp csum"
> >
> > Three packets are sent in each GRE tunnels :
> > * One ICMP echo request
> > * One UDP packet with a valid checksum
> > * One UDP packet with the checksum set to 0
> >
> > Here is a link to the pcap containing the six packets generated by the
> > previous commands:
> > https://onaip.mooo.com/pub/tmp/bug_ip6gre-in-udp.pcap
> >
> > Some details about the captured packets:
> > IP6 fd00::1 > fd00::2: 1234 > 4567: [bad udp cksum 0xfa75 -> 0xe680!]
> > IP6 fd00::1 > fd00::2: 1234 > 4567: [udp sum ok]
> > IP6 fd00::1 > fd00::2: 1234 > 4567: [bad udp cksum 0xfa62 -> 0xb57c!]
> > IP 10.0.0.11.3456 > 10.0.0.33.6789: [udp sum ok]
> > IP 10.0.0.11.3456 > 10.0.0.33.6789: [udp sum ok]
> > IP 10.0.0.11.3456 > 10.0.0.33.6789: [udp sum ok]
> >
> > For the tunnel over ipv6, only the UDP packet sent with a valid
> > checksum get encapsulated with a valid checksum. For the ping and the
> > UDP packet with a zero checksum, the outer UDP checksum matches the
> > partial checksum of the pseudo-header.
> >
> > For ipv4, the UDP checksum of the encapsulation are all valid.
> >
> > The device under test used for the capture is an x86-64 machine with a
> > realtek ethernet adapter (r8169 driver) running a 6.10.7 kernel.
> >
> > The problem was also seen on an arm64 board (freescale ls1046) with
> > dpaa ethernet driver running a 4.14 kernel. on this hardware, the
> > packets that would have an invalid checksum are not emitted and the tx
> > error counter of the ethernet interface increases.
> >
> > In all cases, disabling hardware checksumming for ipv6 with ethtool can
> > be used as a work-around:
> >
> > ethtool -K eth0 tx-checksum-ipv6 off
> >
> > This and the partial checksum value seems to point to an error in the
> > handling of hardware checksumming in the particular case of fou6
> > encapsulation, but I have not been able to figure out what could be
> > causing it.
> >
> > Did I miss a configuration parameter for ip6gre-in-udp? Any advise on
> > how to debug that would be appreciated.
> >
> I did some more digging with 6.11-rc7, and it is not a problem in the
> common code or udp encapsulation, I just got "lucky" with my tests...
>
> First in fou6.c, fou6_build_udp constructs the upd header and calls
> udp6_set_csum. If the inner packet has a valid checksum, the outer udp
> get computed by reusing it, otherwise the partial checksum is set.
>
> Next the outer ipv6 is built in ip6tunnel.c:ip6_tnl_xmit. With the
> default value of encaplimit (4), an destination options header is
> inserted to pass that value. This means that ipv6_hdr(skb)->nexthdr is
> set to 60 (NEXTHDR_DEST), not 17 (IPPROTO_UDP).
>
> So when sending a packet without a valid checksum in a ip6gre-in-udp,
> we pass a skb with a partial udp checksum and an ipv6 header with an
> extension to ndo_start_xmit.
>
> Finally, what are the odds of testing two different hardware and
> finding two similar bugs?
>
> For the LS1046 platform, the Tx checksum is done in
> dpaa_enable_tx_csum. For ipv6, the layer 4 protocol is extracted from
> the skb with l4_proto = ipv6h->nexthdr. Since the value does not match
> IPPROTO_UDP nor IPPROTO_TCP, the function errors out and the packet is
> discarded.
>
> For the PC with a realtek 8169 card, the code is similar in
> rtl8169_tso_csum_v2, with ip_protocol = ipv6_hdr(skb)->nexthdr. The
> error case then triggers a WARN_ON_ONCE(1) and the packet is still sent
> on the wire with a partial udp checksum.
>
> There seems to be quite a few places in drivers/net/ethernet that use
> ipv6_hdr(skb)->nexthdr as the IP protocol, with only some of them
> calling ipv6_skip_exthdr afterward to take care of the extension
> headers. So my proposal is to add a small helper (ipv6_protocol?) and
> fix the drivers where needed.
>
> I'll try to come up with a patch set, unless someone has a better idea.
>
Looking more into this, I don't think the problem is in the ethernet
drivers. Both dpaa and r8169 declare NETIF_F_IPV6_CSUM in their
hw_features. The documentation for this flag says "IPv6 extension
headers are not supported with this feature". Yet in the call-path
described in my previous email, the driver gets handed an skb with both
an IPv6 extension header and a UDP header with a partial checksum.
Below is the ftrace of a ping sent through an ip6gre-in-udp tunnel,
when reaching the validate_xmit_skb call of the ethernet driver:
3) | validate_xmit_skb_list() {
3) | validate_xmit_skb() {
3) | netif_skb_features() {
3) | rtl8169_features_check [r8169]() {
3) 0.576 us | rtl_quirk_packet_padto [r8169]();
3) 1.812 us | }
3) 0.492 us | skb_network_protocol();
3) 4.152 us | }
3) 0.504 us | skb_csum_hwoffload_help();
3) 0.480 us | validate_xmit_xfrm();
3) 7.128 us | }
3) 8.076 us | }
The call to skb_csum_hwoffload_help does not trigger a call to
skb_checksum_help. I guess the bug can be reproduced with any ethernet
card that has tx-checksum-ipv6 set to on in its offload features.
It looks like skb_csum_hwoffload_help does not catch that particular
case. Again, any help would be appreciated.
--
Benoît
Powered by blists - more mailing lists