netdev - Re: tcp_tw_recycle broken?

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <20081115055748.GY24654@1wt.eu>
Date:	Sat, 15 Nov 2008 06:57:48 +0100
From:	Willy Tarreau <w@....eu>
To:	Karl Pickett <karl.pickett@...il.com>
Cc:	linux-kernel@...r.kernel.org, netdev@...r.kernel.org
Subject: Re: tcp_tw_recycle broken?

On Fri, Nov 14, 2008 at 11:37:06PM -0500, Karl Pickett wrote:
> Hey.  Developing a http proxy on fedora 9 (2.6.25) and running into a
> strange issue.
> 
> Having the proxy set up and tear down 6000 tcp connections a second to
> the same test server ip and port,
> it quickly blows up (5 seconds) due to all 30000 ephemeral ports going
> to TIME_WAIT.
> setting tw_recycle=1 fixed the problem, and there are never more than
> a couple hundred ports in TIME_WAIT.
> 
> BUT...
> 
> Changing the load test to alternate between two test server ips, it
> blows up.  Connect: can't assign requested address. (note I am not
> binding before hand, I tried
> and binding first to port 0 made no difference - it just blows up then
> during the bind).
> 
> And there are ~28K ports in TIME_WAIT.  For example:
> 
> proxy_ip:30000 load_test_1:8080 TIME_WAIT
> proxy_ip:30000 load_test_2:8080 TIME_WAIT
> ...
> but most are not duplicates of the same local port.
> 
> 
> What. The.  Heck.
> 
> So short of rebuilding the kernel with time_wait as 1 second, is there
> any other way not to brick my proxy?

two things :
  - set tcp_tw_reuse to 1 too.
  - do a setsockopt(SO_REUSEADDR) before connect()

Using this, my proxy has no problem at 35K sess/s on 2.6.25. I'm not sure
if disabling either option above still works.

Hoping this helps,
Willy

--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html