netdev - IPv4 saddr do not match with selected output device in double default gateways scene

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [thread-next>] [day] [month] [year] [list]

Date:   Tue, 8 Mar 2022 14:41:27 +0800
From:   "Ziyang Xuan (William)" <william.xuanziyang@...wei.com>
To:     David Miller <davem@...emloft.net>, <yoshfuji@...ux-ipv6.org>,
        David Ahern <dsahern@...nel.org>,
        Jakub Kicinski <kuba@...nel.org>,
        netdev <netdev@...r.kernel.org>, <ja@....bg>
CC:     Linux Kernel Mailing List <linux-kernel@...r.kernel.org>
Subject: IPv4 saddr do not match with selected output device in double default
 gateways scene

Create VLAN devices and add default gateways with following commands:

# ip link add link eth2 dev eth2.71 type vlan id 71
# ip link add link eth2 dev eth2.72 type vlan id 72
# ip addr add 192.168.71.41/24 dev eth2.71
# ip addr add 192.168.72.41/24 dev eth2.72
# ip link set eth2.71 up
# ip link set eth2.72 up
# route add -net default gw 192.168.71.1 dev eth2.71
# route add -net default gw 192.168.72.1 dev eth2.72

Add a nameserver configuration in the following file:
# cat /etc/resolv.conf
nameserver 8.8.8.8

Use the following command trigger DNS packet:
# ping www.baidu.com

Assume the above test machine is client.

Of course, we should also create VLAN devices in peer server as following:

# ip link add link eth2 dev eth2.71 type vlan id 71
# ip link add link eth2 dev eth2.72 type vlan id 72
# ip addr add 192.168.71.1/24 dev eth2.71
# ip addr add 192.168.72.1/24 dev eth2.72
# ip link set eth2.71 up
# ip link set eth2.72 up

We capture packets with tcpdump in client machine when ping:
# tcpdump -i eth2 -vne
...
20:30:22.996044 52:54:00:20:23:a9 > 52:54:00:d2:4f:e3, ethertype 802.1Q (0x8100), length 77: vlan 71, p 0, ethertype IPv4, (tos 0x0, ttl 64, id 25407, offset 0, flags [DF], proto UDP (17), length 59)
    192.168.72.41.42666 > 8.8.8.8.domain: 58562+ A? www.baidu.com. (31)
20:30:22.996125 52:54:00:20:23:a9 > 52:54:00:d2:4f:e3, ethertype 802.1Q (0x8100), length 77: vlan 71, p 0, ethertype IPv4, (tos 0x0, ttl 64, id 25408, offset 0, flags [DF], proto UDP (17), length 59)
    192.168.72.41.42666 > 8.8.8.8.domain: 25803+ AAAA? www.baidu.com. (31)
...

We can find that IPv4 saddr "192.168.72.41" do not match with selected VLAN device "eth2.71".

I tracked the related processes, and found that user space program uses connect() firstly, then sends UDP packet.

The problem happens in the connect() process. Analysis as following with codes:

static inline struct rtable *ip_route_connect(struct flowi4 *fl4,
					      __be32 dst, __be32 src, u32 tos,
					      int oif, u8 protocol,
					      __be16 sport, __be16 dport,
					      struct sock *sk)
{
	struct net *net = sock_net(sk);
	struct rtable *rt;

	ip_route_connect_init(fl4, dst, src, tos, oif, protocol,
			      sport, dport, sk);

	if (!dst || !src) {

		/* rtable and fl4 are matched after the first __ip_route_output_key().
		 * rtable->dst.dev->name == "eth2.72" && rtable->rt_gw4 == 0x148a8c0
		 * fl4->saddr == 0x2948a8c0
		 */
		rt = __ip_route_output_key(net, fl4);
		if (IS_ERR(rt))
			return rt;
		ip_rt_put(rt);
		flowi4_update_output(fl4, oif, tos, fl4->daddr, fl4->saddr);
	}
	security_sk_classify_flow(sk, flowi4_to_flowi_common(fl4));

	/* rtable and fl4 do not match after the second __ip_route_output_key().
	 * rtable->dst.dev->name == "eth2.71" && rtable->rt_gw4 == 0x147a8c0
	 * fl4->saddr == 0x2948a8c0
	 */
	return ip_route_output_flow(net, fl4, sk);
}

Deep tracking, it because fa->fa_default has changed in fib_select_default() after first __ip_route_output_key() process,
and a new fib_nh is selected in fib_select_default() within the second __ip_route_output_key() process but not update flowi4.
So the phenomenon described at the beginning happens.

Does it a kernel bug or a user problem? If it is a kernel bug, is there any good solution?