Date:   Sun, 28 Mar 2021 16:05:29 +0200
From:   Michal Soltys <msoltyspl@...dex.pl>
To:     Linux Netdev List <netdev@...r.kernel.org>,
        David Miller <davem@...emloft.net>
Subject: [BUG / question] in routing rules, some options (e.g. ipproto, sport)
 cause rules to be ignored in presence of packet marks

Hi,

I'm not sure how it behaved in earlier kernels (can check later), but it 
is / looks bugged in at least recent 5.x+ ones (tests were done with 
5.11.8 and 5.10.25).

Consider following setup:

# ip -o ad sh
1: lo    inet 127.0.0.1/8 scope host lo
2: right1    inet 10.0.10.2/24 scope global
3: right2    inet 10.0.20.2/24 scope global

# ip ro sh tab main
default via 10.0.10.1 dev right1
10.0.10.0/24 dev right1 proto kernel scope link src 10.0.10.2
10.0.20.0/24 dev right2 proto kernel scope link src 10.0.20.2

# ip ro sh tab 123
default via 10.0.20.1 dev right2 src 10.0.20.2

And routing rules:

0:      from all lookup local
9:      from all fwmark 0x1 ipproto udp sport 1194 lookup 123
10:     from all ipproto udp sport 1194 lookup 123
32766:  from all lookup main
32767:  from all lookup default

This - without any mangling via ipt/nft or by other means - works 
correctly, for example:

nc -u -p 1194 1.2.3.4 12345

will be routed out correctly via 'right2' using 10.0.20.2

But if we add mark to locally outgoing packets:

iptables -t mangle -A OUTPUT -j MARK --set-mark 1

Then *both* rule 9 and rule 10 will be ignored during reroute check. 
tcpdump on interface 'right1' will show:

# tcpdump -nvi right1 udp
tcpdump: listening on right1, link-type EN10MB (Ethernet), snapshot 
length 262144 bytes
13:21:59.684928 IP (tos 0x0, ttl 64, id 8801, offset 0, flags [DF], 
proto UDP (17), length 33)
     10.0.20.2.1194 > 1.2.3.4.12345: UDP, length 5

Initial routing decision in rule 10 will set the address correctly, but 
the packet goes out via interface right1, ignoring both 9 and 10.

If I add another routing roule:

8:      from all fwmark 0x1 lookup 123

Then the packects will flow correctly - but I *cannot* use (from the 
ones I tested): sport, dport, ipproto, uidrange - as they will cause the 
rule to be ignored. For example, this setup of routing rules will fail, 
if there is any mark set on a packet (nc had uid 1120):

# ip ru sh
0:      from all lookup local
10:     from all ipproto udp lookup 123
10:     from all sport 1194 lookup 123
10:     from all dport 12345 lookup 123
10:     from all uidrange 1120-1120 lookup 123
32766:  from all lookup main
32767:  from all lookup default

Adding correct fwmark to the above rules will have *no* effect either. 
Only fwmark *alone* will work (or in combination with: iif, from, to - 
from the ones I tested).

I peeked at fib_rule_match() in net/core/fib_rules.c - but it doesn't 
look like there is anything wrong there. I initially suspected lack of 
'rule->mark &&' in mark related line - but considering that rules such 
as 'from all fwmark 1 sport 1194 lookup main' also fail, it doesn't look 
like it's the culprit (and mark_mask covers that test either way).

OTOH, perhaps nf_ip_reroute() / ip_route_me_harder() are somehow the 
culprit here - but I haven't analyzed them yet. Perhaps it's just an 
issue of changing output interface incorrectly after ip_route_me_harder() ?

Is this a bug ? Or am I misinterpreting how 'reroute check' works after 
initial routing decision ? One would expect routing rules during 
post-mangle check to not be ignored out of the blue, only because packet 
mark changed on the packet. Not mentioning both marks and routing rules 
can be used for separate purposes (e.g. marks for shaping).

Powered by blists - more mailing lists