netdev - Re: 6.1: possible bug with netfilter conntrack?

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <Y8CaaCoOAx6XzWq/@shell.armlinux.org.uk>
Date:   Thu, 12 Jan 2023 23:40:24 +0000
From:   "Russell King (Oracle)" <linux@...linux.org.uk>
To:     netdev@...r.kernel.org, netfilter-devel@...r.kernel.org,
        coreteam@...filter.org
Subject: Re: 6.1: possible bug with netfilter conntrack?

On Thu, Jan 12, 2023 at 11:03:56PM +0000, Russell King (Oracle) wrote:
> Hi,
> 
> I've noticed that my network at home is rather struggling, and having
> done some investigation, I find that the router VM is dropping packets
> due to lots of:
> 
> nf_conntrack: nf_conntrack: table full, dropping packet
> 
> I find that there are about 2380 established and assured connections
> with a destination of my incoming mail server with destination port 25,
> and 2 packets. In the reverse direction, apparently only one packet was
> sent according to conntrack. E.g.:
> 
> tcp      6 340593 ESTABLISHED src=180.173.2.183 dst=78.32.30.218
> sport=49694 dport=25 packets=2 bytes=92 src=78.32.30.218
> dst=180.173.2.183 sport=25 dport=49694 packets=1 bytes=44 [ASSURED]
> use=1
> 
> However, if I look at the incoming mail server, its kernel believes
> there are no incoming port 25 connetions, which matches exim.
> 
> I hadn't noticed any issues prior to upgrading from 5.16 to 6.1 on the
> router VM, and the firewall rules have been the same for much of
> 2021/2022.
> 
> Is this is known issue? Something changed between 5.16 and 6.1 in the
> way conntrack works?
> 
> I'm going to be manually clearing the conntrack table so stuff works
> again without lots of packet loss on my home network...
> 
> Thanks.

Having cleared out all the dport=25 and dport=587 entries, and
observation there after, it looks like conntrack generally works
as one would expect - I see connections become established, and
then disappear as one expects.

All traffic between the 'net and the mail server goes through the
router VM in both directions - there is no asymetry there (the
mail server is on a DMZ network which is only routed within the
router VM to the PPP interfaces also in the router VM for the
public network.) So, conntrack will be aware of every packet in
both directions.

I do see conntrack entries entering TIME_WAIT, LAST_ACK, CLOSE
etc.

Given the packet counts as per my example above, it looks like
conntrack only saw:

src=180.173.2.183 dst=78.32.30.218	SYN
src=78.32.30.218 dst=180.173.2.183	SYN+ACK
src=180.173.2.183 dst=78.32.30.218	ACK

and I suspect at that point, the connection went silent - until
Exim timed out and closed the connection, as does seem to be the
case:

2023-01-11 21:32:04 no host name found for IP address 180.173.2.183
2023-01-11 21:33:05 SMTP command timeout on connection from [180.173.2.183]:64332 I=[78.32.30.218]:25

but if Exim closed the connection, why didn't conntrack pick it up?

-- 
RMK's Patch system: https://www.armlinux.org.uk/developer/patches/
FTTP is here! 40Mbps down 10Mbps up. Decent connectivity at last!