netdev - Re: [BUG / question] in routing rules, some options (e.g. ipproto, sport) cause rules to be ignored in presence of packet marks

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <YGI99fyA6MYKixuB@shredder.lan>
Date:   Mon, 29 Mar 2021 23:52:05 +0300
From:   Ido Schimmel <idosch@...sch.org>
To:     Michal Soltys <msoltyspl@...dex.pl>
Cc:     Linux Netdev List <netdev@...r.kernel.org>,
        David Miller <davem@...emloft.net>
Subject: Re: [BUG / question] in routing rules, some options (e.g. ipproto,
 sport) cause rules to be ignored in presence of packet marks

On Sun, Mar 28, 2021 at 04:05:29PM +0200, Michal Soltys wrote:
> Hi,
> 
> I'm not sure how it behaved in earlier kernels (can check later), but it is
> / looks bugged in at least recent 5.x+ ones (tests were done with 5.11.8 and
> 5.10.25).
> 
> Consider following setup:
> 
> # ip -o ad sh
> 1: lo    inet 127.0.0.1/8 scope host lo
> 2: right1    inet 10.0.10.2/24 scope global
> 3: right2    inet 10.0.20.2/24 scope global
> 
> # ip ro sh tab main
> default via 10.0.10.1 dev right1
> 10.0.10.0/24 dev right1 proto kernel scope link src 10.0.10.2
> 10.0.20.0/24 dev right2 proto kernel scope link src 10.0.20.2
> 
> # ip ro sh tab 123
> default via 10.0.20.1 dev right2 src 10.0.20.2
> 
> And routing rules:
> 
> 0:      from all lookup local
> 9:      from all fwmark 0x1 ipproto udp sport 1194 lookup 123
> 10:     from all ipproto udp sport 1194 lookup 123
> 32766:  from all lookup main
> 32767:  from all lookup default
> 
> This - without any mangling via ipt/nft or by other means - works correctly,
> for example:
> 
> nc -u -p 1194 1.2.3.4 12345
> 
> will be routed out correctly via 'right2' using 10.0.20.2
> 
> But if we add mark to locally outgoing packets:
> 
> iptables -t mangle -A OUTPUT -j MARK --set-mark 1
> 
> Then *both* rule 9 and rule 10 will be ignored during reroute check. tcpdump
> on interface 'right1' will show:
> 
> # tcpdump -nvi right1 udp
> tcpdump: listening on right1, link-type EN10MB (Ethernet), snapshot length
> 262144 bytes
> 13:21:59.684928 IP (tos 0x0, ttl 64, id 8801, offset 0, flags [DF], proto
> UDP (17), length 33)
>     10.0.20.2.1194 > 1.2.3.4.12345: UDP, length 5
> 
> Initial routing decision in rule 10 will set the address correctly, but the
> packet goes out via interface right1, ignoring both 9 and 10.
> 
> If I add another routing roule:
> 
> 8:      from all fwmark 0x1 lookup 123
> 
> Then the packects will flow correctly - but I *cannot* use (from the ones I
> tested): sport, dport, ipproto, uidrange - as they will cause the rule to be
> ignored. For example, this setup of routing rules will fail, if there is any
> mark set on a packet (nc had uid 1120):
> 
> # ip ru sh
> 0:      from all lookup local
> 10:     from all ipproto udp lookup 123
> 10:     from all sport 1194 lookup 123
> 10:     from all dport 12345 lookup 123
> 10:     from all uidrange 1120-1120 lookup 123
> 32766:  from all lookup main
> 32767:  from all lookup default
> 
> Adding correct fwmark to the above rules will have *no* effect either. Only
> fwmark *alone* will work (or in combination with: iif, from, to - from the
> ones I tested).
> 
> I peeked at fib_rule_match() in net/core/fib_rules.c - but it doesn't look
> like there is anything wrong there. I initially suspected lack of
> 'rule->mark &&' in mark related line - but considering that rules such as
> 'from all fwmark 1 sport 1194 lookup main' also fail, it doesn't look like
> it's the culprit (and mark_mask covers that test either way).
> 
> OTOH, perhaps nf_ip_reroute() / ip_route_me_harder() are somehow the culprit
> here - but I haven't analyzed them yet. Perhaps it's just an issue of
> changing output interface incorrectly after ip_route_me_harder() ?

ip_route_me_harder() does not set source / destination port in the
flow key, so it explains why fib rules that use them are not hit after
mangling the packet. These keys were added in 4.17, but I
don't think this use case every worked. You have a different experience?

> 
> Is this a bug ? Or am I misinterpreting how 'reroute check' works after
> initial routing decision ? One would expect routing rules during post-mangle
> check to not be ignored out of the blue, only because packet mark changed on
> the packet. Not mentioning both marks and routing rules can be used for
> separate purposes (e.g. marks for shaping).
>