Message-ID: <ecf16770431c8b30782e3443085641eb685aa5f9.camel@alliedtelesis.co.nz>
Date: Wed, 10 Apr 2024 22:59:47 +0000
From: Paul Davey <Paul.Davey@...iedtelesis.co.nz>
To: "netdev@...r.kernel.org" <netdev@...r.kernel.org>
CC: "steffen.klassert@...unet.com" <steffen.klassert@...unet.com>
Subject: combination of dot1q tagging and gro esp offload appears to corrupt
 esp transport mode traffic

Hi,

I have observed a what I believe is a bug in IPSec esp4 transport mode
with gro enabled in the presence of 802.1Q tagged packets.
However I am not sure what the best course of action is to resolve the
problem.

I believe the issue occurs with the following chain of key events:

1) An ESP transport mode packet containing an 802.1Q tag is received
   on a network interface with GRO enabled.
2) dev_gro_receive will find the ETH_P_8021Q packet offload and call
   vlan_gro_receive which determines the inner ether type and
   continues the gro handler chain
3) the gro handlers eventually reach esp4_gro_receive which looks up
   the xfrm state from the spi and submits the skb to xfrm_input after
   setting the XFRM_GRO flag in xfrm_offload flags.
4) xfrm4_transport_input removes the encapsulation header by moving
   the network header forward to just before the decrypted payload
   content.
5) xfrm4_tranport_finish adjusts new network header protocol, pushes
   skb to include new network header and updates total length and
   updates checksum
6) xfrm4_tranport_finish checks if XFRM_GRO was set in the
   xfrm_offload flags and if so calls skb_mac_header_rebuild.
7) skb_mac_header_rebuild moves the mac header from it's original
   position to just before the new network header.  
   However mac_len is still the original ethernet mac length and the
   mac header still contains ETH_P_8021Q as the ethertype, resulting
   in a packet claiming to be 802.1Q but starting with an IP header
   with no TCI or inner ethertype.
8) skb is queued on the xfrm napi instance for reception
9) skb is processed by skb_vlan_untag in __netif_receive_skb_core as
   skb->protocol is a VLAN ethertype.
10) skb_vlan_untag removes the "vlan tag" leaving an skb with
    skb->protocol set to the total length from the IP header and
    missing the start of the IP header and any ethertype.
11) This skb is obviously not recognised as IP and not sent the
    expected destination, instead usually received as ETH_P_802_2 if
    the IP datagram was short enough.

It appears the load bearing kernel configs for this to happen would be
CONFIG_VLAN_8021Q as this would gate the presence of vlan_gro_receive
and CONFIG_INET_ESP_OFFLOAD or CONFIG_INET6_ESP_OFFLOAD as this will
gate the esp4_gro_receive or esp6_gro_receive paths and setting of
XFRM_GRO flag.

generic-receive-offload needs to be enabled on the underlying
receiving interface.

I have reproduced this on 6.8.2 using an ipsec protected l2tp tunnel
over an 802.1Q tagging interface between a vm and the host machine.

The tunnel mode encapsulation removal code seems to avoid this issue
by forcing the skb->protocol to ETH_P_IP or ETH_P_IPV6 and also
setting this in the mac header if present.

I would appreciate any advice on how to go about fixing this issue.

Reproduction setup below the line, substitute interface names as
necessary.

Thanks,
Paul Davey

---

DUT:

ip link set eth0 up
ip link add link eth0 eth0.10 type vlan id 10
ip addr add 10.0.1.1/24 dev eth0.10
ip link set up dev eth0.10

ip l2tp add tunnel remote 10.0.1.2 local 10.0.1.1 tunnel_id 1 \
peer_tunnel_id 2 encap udp udp_sport 5000 udp_dport 5001
ip l2tp add session name l2tp1 tunnel_id 1 session_id 1 \
peer_session_id 2 

ip link show l2tp1
ip addr add 10.1.0.1/24 dev l2tp1
ip link set l2tp1 up

ip xfrm policy add src 10.0.1.1 dst 10.0.1.2 proto udp sport 5000 \
dport 5001 dir out tmpl src 0.0.0.0 dst 0.0.0.0 proto esp spi 0x100 \
mode transport reqid 1
ip xfrm policy add src 10.0.1.2 dst 10.0.1.1 proto udp sport 5001 \
dport 5000 dir in tmpl src 0.0.0.0 dst 0.0.0.0 proto esp \
mode transport reqid 1

ip xfrm state add src 10.0.1.1 dst 10.0.1.2 proto esp spi 0x100 \
reqid 1 mode transport replay-window 1 flag esn \
aead "rfc4106(gcm(aes))" 0x8516687c549c3921da3e57b2503851095000f090 128
ip xfrm state add src 10.0.1.2 dst 10.0.1.1 proto esp spi 0x101 \
reqid 1 mode transport replay-window 1 flag esn \
aead "rfc4106(gcm(aes))" 0x8516687c549c3921da3e57b2503851095000f090 128

# Corrupted packets can be seen here as strange 802.3LLC frames 
# on vlan 1280
tcpdump -e -i eth0

Link-Partner:

ip link add link tap0 tap0.10 type vlan id 10
ip addr add 10.0.1.2/24 dev tap0.10
ip link set up dev tap0.10

ip l2tp add tunnel remote 10.0.1.1 local 10.0.1.2 tunnel_id 2 \
peer_tunnel_id 1 encap udp udp_sport 5001 udp_dport 5000
ip l2tp add session name l2tp2 tunnel_id 2 session_id 2 \
peer_session_id 1

ip addr add dev l2tp2 10.1.0.2/24
ip link set l2tp2 up

ip xfrm policy add src 10.0.1.1 dst 10.0.1.2 proto udp sport 5000 \
dport 5001 dir in tmpl src 0.0.0.0 dst 0.0.0.0 proto esp \
mode transport reqid 1
ip xfrm policy add src 10.0.1.2 dst 10.0.1.1 proto udp sport 5001 \
dport 5000 dir out tmpl src 0.0.0.0 dst 0.0.0.0 proto esp spi 0x101 \
mode transport reqid 1

ip xfrm state add src 10.0.1.1 dst 10.0.1.2 proto esp spi 0x100 \
reqid 1 mode transport replay-window 1 flag esn \
aead "rfc4106(gcm(aes))" 0x8516687c549c3921da3e57b2503851095000f090 128
ip xfrm state add src 10.0.1.2 dst 10.0.1.1 proto esp spi 0x101 \
reqid 1 mode transport replay-window 1 flag esn \
aead "rfc4106(gcm(aes))" 0x8516687c549c3921da3e57b2503851095000f090 128

# This will show Destination Host Unreachable as ARP in l2tp is never
# received on the DUT
ping 10.1.0.1

Powered by blists - more mailing lists