netdev - RE: [Question]: TCP resets when using ECMP for load-balancing between multiple servers.

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [day] [month] [year] [list]

Message-ID:
 <DBBP189MB14338FF989660CDBABE27F67951BA@DBBP189MB1433.EURP189.PROD.OUTLOOK.COM>
Date: Fri, 18 Aug 2023 13:17:08 +0000
From: Sriram Yagnaraman <sriram.yagnaraman@....tech>
To: Ido Schimmel <idosch@...sch.org>, "dsahern@...il.com" <dsahern@...il.com>,
	"pabeni@...hat.com" <pabeni@...hat.com>
CC: "netdev@...r.kernel.org" <netdev@...r.kernel.org>
Subject: RE: [Question]: TCP resets when using ECMP for load-balancing between
 multiple servers.

Hi Ido,

> -----Original Message-----
> From: Ido Schimmel <idosch@...sch.org>
> Sent: Thursday, 17 August 2023 18:41
> To: Sriram Yagnaraman <sriram.yagnaraman@....tech>; dsahern@...il.com;
> pabeni@...hat.com
> Cc: netdev@...r.kernel.org
> Subject: Re: [Question]: TCP resets when using ECMP for load-balancing
> between multiple servers.
> 
> + David, Paolo
> 
> On Tue, Aug 15, 2023 at 10:10:48PM +0200, Sriram Yagnaraman wrote:
> > All packets in the same flow (L3/L4 depending on multipath hash
> > policy) should be directed to the same target, but after [0] we see
> > stray packets directed towards other targets. This, for instance,
> > causes RST to be sent on TCP connections. This happens on a static
> > setup, with no changes to the nexthops, so there is no hash space
> reassignment.
> 
> Which multipath hash policy are you using? I guess the issue is more visible
> with L4 as ip_can_use_hint() at least makes sure the destination IP is the same
> before using the hint.

Yes, I am using L4 multipath hash policy. But I think the problem will be seen even on L3, if the source addresses are different.
 
> 
> >
> > IIUC, route hints when the next hop is part of a multipath group
> > causes packets in the same receive batch to be sent to the same next
> > hop irrespective of which nexthop the multipath hash points to. I am
> > no expert in this area, so please let me know if there is a simple
> > explanation on how to fix this problem?
> >
> > Below is a patch which has a selftest that describes the problem setup
> > and a hack to solve the problem in ipv4. For ipv6, I have just
> > commented out the part the returns the route hint, just for testing.
> 
> Did you consider marking the skb instead of the route? Something like [1].
> Compile tested only.
> 

Thank you so much for the idea/code. 
No, I didn't think of that. I will try your patch and get back with the results. 

> Also, are you positive that your selftest fails before the patch and passes after?
> It is using VRFs, which use FIB rules, which should in turn disable the use of
> hints. If this is indeed the case, then try using namespaces instead. There are
> various examples outside of the forwarding directory.

Ah, my mistake. I wrote the selftest to explain the problem and didn't test it thoroughly, sorry about that.
I will write a test with namespaces and send a proper patchset for review. Thanks once again.