netdev - RE: : getsockopt(fd, SOL_IP, SO_ORIGINAL_DST, sa, &salen) is in fact sometimes returning the source IP instead the destination IP

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <MWHPR16MB150280EC34A9D4A8ECF809D4ED990@MWHPR16MB1502.namprd16.prod.outlook.com>
Date:   Wed, 23 Jan 2019 08:07:48 +0000
From:   "Mohandass, Roobesh" <Roobesh_Mohandass@...fee.com>
To:     Willy Tarreau <w@....eu>, Lukas Tribus <lists@...i.eu>
CC:     Florian Westphal <fw@...len.de>,
        "netdev@...r.kernel.org" <netdev@...r.kernel.org>
Subject: RE: : getsockopt(fd, SOL_IP, SO_ORIGINAL_DST, sa, &salen) is
 in fact sometimes returning the source IP instead the destination IP

Hi Willy/Florian/Lukas,

 disabling nf_conntrack_tcp_loose (solved the problem) and for last 24 hours, we are not evening seeing a single wrong connection data by send-proxy.

Worth a note in your documentation related to this, as users might be aware off.

Thanks very much for your help and support, I will report back if I see anything again related to this.

-Roobesh G M

-----Original Message-----
From: Mohandass, Roobesh 
Sent: Thursday, January 17, 2019 10:54 AM
To: Willy Tarreau <w@....eu>; Lukas Tribus <lists@...i.eu>
Cc: Florian Westphal <fw@...len.de>; netdev@...r.kernel.org
Subject: RE: [NETDEV]: getsockopt(fd, SOL_IP, SO_ORIGINAL_DST, sa, &salen) is in fact sometimes returning the source IP instead the destination IP

Hi Willy/Florian/Lukas,

Thanks for your help around this concern, sorry for the delayed response. I will test this out and get back to you.

-Roobesh G M

-----Original Message-----
From: Willy Tarreau <w@....eu>
Sent: Sunday, January 13, 2019 12:04 AM
To: Lukas Tribus <lists@...i.eu>
Cc: Mohandass, Roobesh <Roobesh_Mohandass@...fee.com>; Florian Westphal <fw@...len.de>; netdev@...r.kernel.org
Subject: Re: [NETDEV]: getsockopt(fd, SOL_IP, SO_ORIGINAL_DST, sa, &salen) is in fact sometimes returning the source IP instead the destination IP

This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe.

Hi Lukas,

On Sat, Jan 12, 2019 at 07:01:34PM +0100, Lukas Tribus wrote:
> > Roobesh, do you use the destination address only for logging or 
> > anywhere else in the request path ? And could you check if you have 
> > nf_conntrack_tcp_loose set as Florian suggests ? I really think he 
> > figured it right.
> 
> It's about what we send with the PROXY protocol to the backend server, 
> Roobesh reported things like that (src and dst is the same):
> 
> PROXY TCP4 192.220.26.39 192.220.26.39 45066 45066 PROXY TCP4
> 192.220.26.39 192.220.26.39 45075 45075
> 
> So the call would actually happen at the beginning of the TCP connection.

That sounds quite shocking to me then. Maybe we're facing such a sequence:

 1) first session comes from this port, and client closes first (FIN)
 2) haproxy then closes with (FIN)
 3) client doesn't respond with the final ACK (e.g. blocked by another
    firewall in between or the client's own conntrack)
 4) the socket on the haproxy side remains in LAST_ACK state and ACKs
    are periodically resent
 5) local conntrack is in TIME_WAIT and expires faster than the largest
    interval between two such ACKs
 6) one of these retransmitted ACKs reopens the connection in reverse
    direction due to nf_conntrack_tcp_loose. The connection is then
    seen in ESTABLISHED state and might be kept a bit longer.
 8) the connection finally expires in the local TCP stack but not yet
    in conntrack.
 7) later the client reuses the same source port while the connection
    is still present in the conntrack table.
 8) assuming tcp_be_liberal is also set, the packets can pass through
    conntrack and establish a new connection to haproxy.
 9) haproxy calls getsockopt(SO_ORIGINAL_DST) and gets the other end
    point since the connection was created at step 6 above in the
    other direction.

I could be wrong on certain specific points above but it looks plausible. 

> Initial report is here:
> https://discourse.haproxy.org/t/send-proxy-not-modifying-some-traffic-
> with-proxy-ip-port-details/3336

Ah cool, I didn't have all this, thank you!

> Let's see if disabling nf_conntrack_tcp_loose changes things.

Yes this really is the only thing I can think of, and in this case noone is wrong in this chain (neither kernel nor haproxy). We'll need to document it in my opinion.

Thanks,
Willy